I'm trying to create a new variable which equals the latest month's value minus the previous month's (or 3 months prior, etc.).
A quick df:
country <- c("XYZ", "XYZ", "XYZ")
my_dates <- c("2021-10-01", "2021-09-01", "2021-08-01")
var1 <- c(1, 2, 3)
df1 <- country %>% cbind(my_dates) %>% cbind(var1) %>% as.data.frame()
df1$my_dates <- as.Date(df1$my_dates)
df1$var1 <- as.numeric(df1$var1)
For example, I've tried (partially from: How to subtract months from a date in R?)
library(tidyverse)
df2 <- df1 %>%
mutate(dif_1month = var1[my_dates==max(my_dates)] -var1[my_dates==max(my_dates) %m-% months(1)]
I've also tried different variations of using lag():
df2 <- df1 %>%
mutate(dif_1month = var1[my_dates==max(my_dates)] - var1[my_dates==max(my_dates)-lag(max(my_dates), n=1L)])
Any suggestions on how to grab the value of a variable when dates equal the second latest observation?
Thanks for help, and apologies for not including any data. Can edit if necessary.
Edited with a few potential answers:
#this gives me the value of var1 of the latest date
df2 <- df1 %>%
mutate(value_1month = var1[my_dates==max(my_dates)])
#this gives me the date of the second latest date
df2 <- df1 %>%
mutate(month1 = max(my_dates) %m-%months(1))
#This gives me the second to latest value
df2 <- df1 %>%
mutate(var1_1month = var1[my_dates==max(my_dates) %m-%months(1)])
#This gives me the difference of the latest value and the second to last of var1
df2 <- df1 %>%
mutate(diff_1month = var1[my_dates==max(my_dates)] - var1[my_dates==max(my_dates) %m-%months(1)])
mutate requires the output to be of the same length as the number of rows of the original data. When we do the subsetting, the length is different. We may need ifelse or case_when
library(dplyr)
library(lubridate)
df1 %>%
mutate(diff_1month = case_when(my_dates==max(my_dates) ~
my_dates %m-% months(1)))
NOTE: Without a reproducible example, it is not clear about the column types and values
Based on the OP's update, we may do an arrange first, grab the last two 'val' and get the difference
df1 %>%
arrange(my_dates) %>%
mutate(dif_1month = diff(tail(var1, 2)))
. my_dates var1 dif_1month
1 XYZ 2021-08-01 3 -1
2 XYZ 2021-09-01 2 -1
3 XYZ 2021-10-01 1 -1
Related
I have a large database with a date column that has date numbers coming from Excel, incomplete dates that are missing the year (but year is in another column), and some cells with missing date. I found out how to change format of the dates, but the problem is how to filter the three types of cells I have in the date variable (that is date numbers from excel, incomplete dates, and empty cell). I managed to do it by filtering a by a created column (value) that I DON'T have in the real database.
This is my original database:
This is what I required end result:
What I managed to do was to filter the dataset with the fictitious value column and convert the date to the required format. This is what I did:
library(dplyr)
data_a <- read.csv(text = "
year,date,value
2018,43238,1
2017,43267,2
2020,7/25,3
2018,,4
2013,,5
2000,8/23,6
2000,9/21,7")
data_b <- data_a %>%
filter(value %in% c(1,2)) %>%
mutate(data_formatted = as.Date(as.numeric(date), origin = "1899-12-30"))
data_c <- data_a %>%
filter(value %in% c(3, 6, 7)) %>%
mutate(data_formatted = as.Date(paste0(year, "/", date)))
data_d <- data_a %>%
filter(value %in% c(4, 5)) %>%
mutate(data_formatted = NA)
data_final <- rbind(data_b, data_c, data_d)
I need to do the same all at once WITHOUT using the value column.
You can use do conditional for the scenarios and apply different functions to convert to date.
Code
library(dplyr)
library(stringr)
library(lubridate)
data_a %>%
mutate(
data_formatted = case_when(
!str_detect(date,"/") ~ as.Date(as.numeric(date), origin = "1899-12-30"),
TRUE ~ ymd(paste0(year, "/", date))
)
)
Output
year date value data_formatted
1 2018 43238 1 2018-05-18
2 2017 43267 2 2018-06-16
3 2020 7/25 3 2020-07-25
4 2018 4 <NA>
5 2013 5 <NA>
6 2000 8/23 6 2000-08-23
7 2000 9/21 7 2000-09-21
Please try
data_a2 <- data_a %>% mutate(date2=as.numeric(ifelse(str_detect(date,'\\/'), '',date)),
date2_=as.numeric(as.Date(ifelse(str_detect(date,'\\/'), paste0(year,'/',date),''), format='%Y/%m/%d')),
date_formatted=as.Date(coalesce(date2,date2_), origin = "1970-01-01")) %>%
dplyr::select(-date2,-date2_)
I am an aspiring data scientist, and this will be my first ever question on StackOF.
I have this line of code to help wrangle me data. My date filter is static. I would prefer not to have to go in an change this hardcoded value every year. What is the best alternative for my date filter to make it more dynamic? The date column is also difficult to work with because it is not a
"date", it is a "dbl"
library(dplyr)
library(lubridate)
# create a sample dataframe
df <- data.frame(
DATE = c(20191230, 20191231, 20200122)
)
Tried so far:
df %>%
filter(DATE >= 20191231)
# load packages (lubridate for dates)
library(dplyr)
library(lubridate)
# create a sample dataframe
df <- data.frame(
DATE = c(20191230, 20191231, 20200122)
)
This looks like this:
DATE
1 20191230
2 20191231
3 20200122
# and now...
df %>% # take the dataframe
mutate(DATE = ymd(DATE)) %>% # turn the DATE column actually into a date
filter(DATE >= floor_date(Sys.Date(), "year") - days(1))
...and filter rows where DATE is >= to one day before the first day of this year (floor_date(Sys.Date(), "year"))
DATE
1 2019-12-31
2 2020-01-22
I am trying to accomplish the following:
group data by id
remove any rows after '3' occurs.
find the closest '1','2' or NA that precedes '3' and only keep that row.
My data:
data <- data.frame(
id=c(1,1,1,1,1, 2,2,2,2, 3,3,3),
a=c(NA,1,2,3,3, NA,3,2,3, 1,5,3))
Desired output:
desired <- data.frame(
id=c(1,2,3), a=c(2,NA,1))
For steps 1-2, I have tried:
data %>% group_by(id) %>% slice(if(first(a) == 3))
but that seems quite off.
Thank you.
This breaks the problem into separate steps
data %>%
group_by(id) %>%
filter(row_number()<first(which(a==3))) %>% # drop things past a 3
filter(a %in% c(1,2,NA)) %>% # only keep 1,2 or NA
filter(row_number()==n()) # choose the last row in each group
I would like some help regarding a data frame transformation required for an analysis. My data consists of a large amount of individuals with all their historic employment. "EX" is a code representing the reason for ending employment. Something like this:
id Date_start Date_end EX
13 "2001-02-01" "2001-05-30" A
13 "2002-03-01" "2010-06-02" B
14 ... ...
...
So what I would like to do is to "fill in the gaps". This may not be easy but its even more difficult because I want it aggregated by id and each new row should have the EX value of the row before, like this:
id Date_start Date_end EX
13 "2001-02-01" "2001-05-30" A
13 "2001-05-31" "2002-02-28" A
13 "2002-03-01" "2010-06-02" B
14 ... ...
...
I believe the trick would be some kind of lag and aggregate but I'm totally lost.
This is a little bit tricky, and you can mainly utilize the dplyr package to do the manipulation and lubridate packages to convert the date format(you can use as.Date() for sure, but lubridate makes it easier).
library(dplyr)
library(lubridate)
1.Creating the sample data you provided.
names <- c("id", "Date_start", "Date_end", "EX")
row1 <- c(13 , "2001-02-01" , "2001-05-30" , "A")
row2 <- c(13 , "2002-03-01" , "2010-06-02" , "B")
testdata <- rbind(row1,row2) %>% data.frame(stringsAsFactors = F)
row.names(testdata) <- NULL
names(testdata) <- names
testdata$Date_start <- testdata$Date_start %>% as_date()
testdata$Date_end <- testdata$Date_end %>% as_date()
testdata
2.Creating a new data set that has the data you want to add.
id: we are using the same id value since it is grouping by id.
Date_start: we are creating the Date_start with a value if there is gap, otherwise "" (empty column, and we are filtering them out).
Date_end: Same logic for Date_end.
EX: we are using the second last EX value as you stated.
new_data <- test_data %>%
group_by(id) %>%
mutate(Date_start1 = ifelse(Date_start-lag(Date_end) == 1,0,lag(Date_end)+1),
Date_end1 = ifelse(Date_start-lag(Date_end) == 1,0,Date_start-1),
EX=first(EX)) %>%
filter(!Date_start1 ==0) %>%
select(id, Date_start=Date_start1,Date_end=Date_end1,EX) %>%
distinct() %>%
ungroup()
3.Since we want to fill the gap days, mutate made it into numeric value, and we are using as_date() from lubriate to convert it into date format.
new_data$Date_start <- as_date(new_data$Date_start)
new_data$Date_end <- as_date(new_data$Date_end)
4.Combine it with your sample data and arrange it by Date_state.
final <- rbind(testdata,new_data) %>% data.frame() %>% arrange(Date_start)
final
Your final result is as below.
I've got a data frame (df1) with an ID variable and two date variables (dat1 and dat2).
I'd like to subset the data frame so that I get the observations for which the difference between dat2 and dat1 is less than or equal to 30 days.
I'm trying to use dplyr() but I can't get it to work.
Any help would be much appreciated.
Starting point (df):
df1 <- data.frame(ID=c("a","b","c","d","e","f"),dat1=c("01/05/2017","01/05/2017","01/05/2017","01/05/2017","01/05/2017","01/05/2017"),dat2=c("14/05/2017","05/06/2017","23/05/2017","15/10/2017","15/11/2017","15/12/2017"), stringsAsFactors = FALSE)
Desired outcome (df):
dfgoal <- data.frame(ID=c("a","c"),dat1=c("01/05/2017","01/05/2017"),dat2=c("14/05/2017","23/05/2017"),newvar=c(13,22))
Current code:
library(dplyr)
df2 <- df1 %>% mutate(newvar = as.Date(dat2) - as.Date(dat1)) %>%
filter(newvar <= 30)
We need to convert to Date class before doing the subtraction
library(dplyr)
library(lubridate)
df1 %>%
mutate_at(2:3, dmy) %>%
mutate(newvar = as.numeric(dat2- dat1)) %>%
filter(newvar <=30)
The as.Date also needs to include the format argument, otherwise, it will think that the format is in the accepted %Y-%m-%d. Here, it is in %d/%m/%Y
df1 %>%
mutate(newvar = as.numeric(as.Date(dat2, "%d/%m/%Y") - as.Date(dat1, "%d/%m/%Y"))) %>%
filter(newvar <= 30)
# ID dat1 dat2 newvar
#1 a 01/05/2017 14/05/2017 13
#2 c 01/05/2017 23/05/2017 22