This question already has answers here:
Expand ranges defined by "from" and "to" columns
(10 answers)
Closed 3 years ago.
I am working on incorporating a variable that is recorded once per unit to a yearly dataset. While it is quite straightforward to repeat the observations n times, I have trouble assigning years to the observations.
The structure of my data is as follows:
id startyear endyear dummy
1 1946 2005 1
2 1957 2005 1
3 1982 2005 1
4 1973 2005 1
What I want to do is to create a new row, called years, which repeats unit 1 n = 2005 - 1946 = 59 times; unit 2 2005-1957 times, and so forth as well as assigning the year, generating the following output:
id startyear endyear dummy year
1 1946 2005 1 1946
1 1946 2005 1 1947
1 1946 2005 1 1948
1 1946 2005 1 1949
[…]
I have attempted to use slice and mutate in dplyr, in combination with rep and seq but neither gives me the result I want. Any help would be greatly appreciated.
We can use map2 to create a sequence from 'startyear' to 'endyear' for each element into a list and then unnest
library(tidyverse)
df1 %>%
mutate(year = map2(startyear, endyear, `:`)) %>%
unnest
# id startyear endyear dummy year
#1 1 1946 2005 1 1946
#2 1 1946 2005 1 1947
#3 1 1946 2005 1 1948
#4 1 1946 2005 1 1949
#5 1 1946 2005 1 1950
#6 1 1946 2005 1 1951
#7 1 1946 2005 1 1952
#...
Or do a group by 'id', mutate into a list and unnest
df1 %>%
group_by(id) %>%
mutate(year = list(startyear:endyear)) %>%
unnest
Less elegant alternative, almost as simple:
library(tidyverse)
df1 %>%
uncount(endyear - startyear + 1, .id = "row") %>%
mutate(year = startyear + row - 1)
Related
This question already has answers here:
Extend an irregular sequence and add zeros to missing values
(9 answers)
Closed 1 year ago.
I have some data in R that looks like this.
year freq
<int> <int>
1902 2
1903 2
1905 1
1906 4
1907 1
1908 1
1909 1
1912 1
1914 1
1915 1
The data was read in using the following code.
data = read.csv("earthquakes.csv")
my_var <- c('year')
new_data <- data[my_var]
counts <- count(data, 'year')
This is 1 page of a 7 page table. I need to fill in the missing years with a count of 0 from 1900-1999. How would I go about this? I haven't been able to find an example online where year is the primary column.
We may use complete on the 'counts' data
library(tidyr)
complete(counts, year = 1990:1999, fill = list(freq = 0))
1) Convert the input, shown in the Note, to zoo class and then to ts class. The latter will fill iln the missing years with NA. Replace the NA's with 0, convert back to data frame and set the names to the original names.
If a ts series is ok as output then omit the last two lines. If in addition it is ok to use NA rather than 0 then omit the last three lines.
library(zoo)
DF |>
read.zoo() |>
as.ts() |>
na.fill(0) |>
fortify.zoo() |>
setNames(names(DF))
giving:
year freq
1 1902 2
2 1903 2
3 1904 0
4 1905 1
5 1906 4
6 1907 1
7 1908 1
8 1909 1
9 1910 0
10 1911 0
11 1912 1
12 1913 0
13 1914 1
14 1915 1
2) for a base solution use merge. Omit the last line if NA is ok instead of 0.
m <- merge(DF, data.frame(year = min(DF$year):max(DF$year)), all = TRUE)
transform(m, freq = replace(freq, is.na(freq), 0))
Note
Lines <- "year freq
1902 2
1903 2
1905 1
1906 4
1907 1
1908 1
1909 1
1912 1
1914 1
1915 1"
DF <- read.table(text = Lines, header = TRUE)
I have a data frame with Date and Velocity as they are seen below. My issue is that some years are missing like 1945 and 1951.
I would like to add 1945 to Date only once and on the position that it should be on between 1944 and 1946. I know some years are repeated. The day and month are not very important as they are more of a position holder. I plan to make the velocity equal to 0 for all the added years (e.g. mm-dd-1945)
What I have
Date Velocity
2/23/1944 1
12/26/1944 2
1/7/1946 5
3/25/1947 8
4/14/1948 10
6/18/1949 12
1/31/1950 13
12/7/1950 14
1/27/1952 15
I tried doing the following
NewYear <- complete(Data,Date = seq.Date(min(Data$Date),
max(Data$Date), by="year"))
but all of the existing dates get overwritten and I end up with this
Date Velocity
2/23/1944 NA
2/23/1945 NA
2/23/1946 NA
2/23/1947 NA
2/23/1948 NA
2/23/1949 NA
2/23/1950 NA
2/23/1951 NA
2/23/1952 NA
Desired Output
Date Velocity
2/23/1944 1
12/26/1944 2
1/01/1945 0
1/7/1946 5
3/25/1947 8
4/14/1948 10
6/18/1949 12
1/31/1950 13
12/7/1950 14
1/1/1951 0
1/27/1952 15
We first need to extract the year from the date then use complete to get missing years and replace the missing Date with first day of the Year.
library(dplyr)
df %>%
mutate(Date = as.Date(Date, "%m/%d/%Y"),
Year = as.integer(format(Date, "%Y"))) %>%
tidyr::complete(Year = seq(min(Year), max(Year)), fill = list(Velocity = 0)) %>%
mutate(Date = if_else(is.na(Date), as.Date(paste0(Year, "-01-01")), Date))
# Year Date Velocity
# <int> <date> <dbl>
# 1 1944 1944-02-23 1
# 2 1944 1944-12-26 2
# 3 1945 1945-01-01 0
# 4 1946 1946-01-07 5
# 5 1947 1947-03-25 8
# 6 1948 1948-04-14 10
# 7 1949 1949-06-18 12
# 8 1950 1950-01-31 13
# 9 1950 1950-12-07 14
#10 1951 1951-01-01 0
#11 1952 1952-01-27 15
Add select(-Year) if you don't want Year column in your final output.
I have tried to find a solution via similar topics, but haven't found anything suitable. This may be due to the search terms I have used. If I have missed something, please accept my apologies.
Here is a excerpt of my data UN_ (the provided sample should be sufficient):
country year sector UN
AT 1990 1 1.407555
AT 1990 2 1.037137
AT 1990 3 4.769618
AT 1990 4 2.455139
AT 1990 5 2.238618
AT 1990 Total 7.869005
AT 1991 1 1.484667
AT 1991 2 1.001578
AT 1991 3 4.625927
AT 1991 4 2.515453
AT 1991 5 2.702081
AT 1991 Total 8.249567
....
BE 1994 1 3.008115
BE 1994 2 1.550344
BE 1994 3 1.080667
BE 1994 4 1.768645
BE 1994 5 7.208295
BE 1994 Total 1.526016
BE 1995 1 2.958820
BE 1995 2 1.571759
BE 1995 3 1.116049
BE 1995 4 1.888952
BE 1995 5 7.654881
BE 1995 Total 1.547446
....
What I want to do is, to add another row with UN_$sector = Residual. The value of residual will be (UN_$sector = Total) - (the sum of column UN for the sectors c("1", "2", "3", "4", "5")) for a given year AND country.
This is how it should look like:
country year sector UN
AT 1990 1 1.407555
AT 1990 2 1.037137
AT 1990 3 4.769618
AT 1990 4 2.455139
AT 1990 5 2.238618
----> AT 1990 Residual TO BE CALCULATED
AT 1990 Total 7.869005
As I don't want to write many, many lines of code I'm looking for a way to automate this. I was told about loops, but can't really follow the concept at the moment.
Thank you very much for any type of help!!
Best,
Constantin
PS: (for Parfait)
country year sector UN ETS
UK 2012 1 190336512 NA
UK 2012 2 18107910 NA
UK 2012 3 8333564 NA
UK 2012 4 11269017 NA
UK 2012 5 2504751 NA
UK 2012 Total 580957306 NA
UK 2013 1 177882200 NA
UK 2013 2 20353347 NA
UK 2013 3 8838575 NA
UK 2013 4 11051398 NA
UK 2013 5 2684909 NA
UK 2013 Total 566322778 NA
Consider calculating residual first and then stack it with other pieces of data:
# CALCULATE RESIDUALS BY MERGED COLUMNS
agg <- within(merge(aggregate(UN ~ country + year, data = subset(df, sector!='Total'), sum),
aggregate(UN ~ country + year, data = subset(df, sector=='Total'), sum),
by=c("country", "year")),
{UN <- UN.y - UN.x
sector = 'Residual'})
# ROW BIND DIFFERENT PIECES
final_df <- rbind(subset(df, sector!='Total'),
agg[c("country", "year", "sector", "UN")],
subset(df, sector=='Total'))
# ORDER ROWS AND RESET ROWNAMES
final_df <- with(final_df, final_df[order(country, year, as.character(sector)),])
row.names(final_df) <- NULL
Rextester demo
final_df
# country year sector UN
# 1 AT 1990 1 1.407555
# 2 AT 1990 2 1.037137
# 3 AT 1990 3 4.769618
# 4 AT 1990 4 2.455139
# 5 AT 1990 5 2.238618
# 6 AT 1990 Residual -4.039062
# 7 AT 1990 Total 7.869005
# 8 AT 1991 1 1.484667
# 9 AT 1991 2 1.001578
# 10 AT 1991 3 4.625927
# 11 AT 1991 4 2.515453
# 12 AT 1991 5 2.702081
# 13 AT 1991 Residual -4.080139
# 14 AT 1991 Total 8.249567
# 15 BE 1994 1 3.008115
# 16 BE 1994 2 1.550344
# 17 BE 1994 3 1.080667
# 18 BE 1994 4 1.768645
# 19 BE 1994 5 7.208295
# 20 BE 1994 Residual -13.090050
# 21 BE 1994 Total 1.526016
# 22 BE 1995 1 2.958820
# 23 BE 1995 2 1.571759
# 24 BE 1995 3 1.116049
# 25 BE 1995 4 1.888952
# 26 BE 1995 5 7.654881
# 27 BE 1995 Residual -13.643015
# 28 BE 1995 Total 1.547446
I think there are multiple ways you can do this. What I may recommend is to take advantage of the tidyverse suite of packages which includes dplyr.
Without getting too far into what dplyr and tidyverse can achieve, we can talk about the power of dplyr's inline commands group_by(...), summarise(...), arrange(...) and bind_rows(...) functions. Also, there are tons of great tutorials, cheat sheets, and documentation on all tidyverse packages.
Although it is less and less relevant these days, we generally want to avoid for loops in R. Therefore, we will create a new data frame which contains all of the Residual values then bring it back into your original data frame.
Step 1: Calculating all residual values
We want to calculate the sum of UN values, grouped by country and year. We can achieve this by this value
res_UN = UN_ %>% group_by(country, year) %>% summarise(UN = sum(UN, na.rm = T))
Step 2: Add sector column to res_UN with value 'residual'
This should yield a data frame which contains country, year, and UN, we now need to add a column sector which the value 'Residual' to satisfy your specifications.
res_UN$sector = 'Residual'
Step 3 : Add res_UN back to UN_ and order accordingly
res_UN and UN_ now have the same columns and they can now be added back together.
UN_ = bind_rows(UN_, res_UN) %>% arrange(country, year, sector)
Piecing this all together, should answer your question and can be achieved in a couple lines!
TLDR:
res_UN = UN_ %>% group_by(country, year) %>% summarise(UN = sum(UN, na.rm = T))`
res_UN$sector = 'Residual'
UN_ = bind_rows(UN_, res_UN) %>% arrange(country, year, sector)
I am looking to calculate relative age of animals. I need to subtract sequentially each year from the next for each animal in my dataset. Because an animal can have multiple reproductive events in a year, I need the age for the remaining events in that year (i.e. all events after the first) to be the same as the initial calculation.
Update:
The dataset more resembles this:
Year ID Age
1 1975 6 -1
2 1975 6 -1
3 1976 6 -1
4 1977 6 -1
6 1975 9 -1
8 1978 9 -1
And I need it to look like this
Year ID Age
1 1975 6 0
2 1975 6 0
3 1976 6 1
4 1977 6 2
6 1975 9 0
8 1978 9 3
Apologies for the initial confusion, if I wasn't clear on what I needed to accomplish.
Any help would be greatly appreciated.
Things done "by group" are usually easiest to do using dplyr or data.table
library(dplyr)
your_data %>%
group_by(ID) %>% # group by ID
mutate(Age = Year - min(Year)) # add new column
or
library(data.table)
setDT(your_data) # convert to data table
# add new column by group
your_data[, Age := Year - min(Year), by = ID]
In base R, ave is probably easiest for adding a groupwise columns to existing data:
your_data$Age = with(your_data, ave(Year, ID, function(x) x - min(x)))
but the syntax isn't as nice as the options above.
You can test on this data:
your_data = read.table(text = " Year ID Age
1 1975 6 -1
2 1975 6 -1
3 1976 6 -1
4 1977 6 -1
6 1975 9 -1
8 1978 9 -1 ", header = T)
if you're trying to figure out the relative age based on one intial birth year, 1975 (which it seems like you are), then you can just make a new column called "RelativeAge" and set it equal to the year - 1975
data$RelativeAge = (Year-1975)
then just get rid of the original "Age" column, or rename as necessary
I have a table that looks like the following:
Year Country Variable 1 Variable 2
1970 UK 1 3
1970 USA 1 3
1971 UK 2 5
1971 UK 2 3
1971 UK 1 5
1971 USA 2 2
1972 USA 1 1
1972 USA 2 5
I'd be grateful if someone could tell me how I can aggregate the data to group it first by year, then country with the sum of variable 1 and variable 2 coming afterwards so the output would be:
Year Country Sum Variable 1 Sum Variable 2
1970 UK 1 3
1970 USA 1 3
1971 UK 5 13
1971 USA 2 2
1972 USA 3 6
This is the code I've tried to no avail (the real dataframe is 125,000 rows by 30+ columns hence the subset. Please be kind, I'm new to R!)
#making subset from data
GT2 <- subset(GT1, select = c("iyear", "country_txt", "V1", "V2"))
#making sure data types are correct
GT2[,2]=as.character(GT2[,2])
GT2[,3] <- as.numeric(as.character( GT2[,3] ))
GT2[,4] <- as.numeric(as.character( GT2[,4] ))
#removing NA values
GT2Omit <- na.omit(GT2)
#trying to aggregate - i.e. group by year, then country with the sum of Variable 1 and Variable 2 being shown
aggGT2 <-aggregate(GT2Omit, by=list(GT2Omit$iyear, GT2Omit$country_txt), FUN=sum, na.rm=TRUE)
Your aggregate is almost correct:
> aggGT2 <-aggregate(GT2Omit[3:4], by=GT2Omit[c("country_txt", "iyear")], FUN=sum, na.rm=TRUE)
> aggGT2
country_txt iyear V1 V2
1 UK 1970 1 3
2 USA 1970 1 3
3 UK 1971 5 13
4 USA 1971 2 2
5 USA 1972 3 6
dplyr is almost always the answer nowadays.
library(dplyr)
aggGT1 <- GT1 %>% group_by(iyear, country_txt) %>% summarize(sv1=sum(V1), sv2=sum(V2))
Having said that, it is good to learn basic R functions like aggregate and by.