The following file names were used in a camera trap study. The S number represents the site, P is the plot within a site, C is the camera number within the plot, the first string of numbers is the YearMonthDay and the second string of numbers is the HourMinuteSecond.
file.names <- c( 'S123.P2.C10_20120621_213422.jpg',
'S10.P1.C1_20120622_050148.jpg',
'S187.P2.C2_20120702_023501.jpg')
file.names
Use a combination of str_sub() and str_split() to produce a data frame with columns corresponding to the site, plot, camera, year, month, days, hour, minute, and second for these three file names. So we want to produce code that will create the data frame:
Site
Plot
Camera
Year
Month
Day
Hour
Minute
Second
S123
P2
C10
2012
06
21
21
34
22
S10
P1
C1
2012
06
22
05
01
48
S187
P2
C2
2012
07
02
02
35
01
My codes are below:
file.names %>%
str_sub(start = 1, end = -5) %>%
str_replace_all("_", ".") %>%
str_split(pattern = fixed("."), n = 5)
I have no idea how to split date and time
nms <- c("Site", "Plot", "Camera", "Year", "Month", "Day", "Hour", "Minute", "Second")
library(tidyverse)
data.frame(file.names) %>%
extract(file.names, nms,
'(\\w+)\\.(\\w+)\\.(\\w+)_(\\d{4})(\\d{2})(\\d{2})_(\\d{2})(\\d{2})(\\d{2})')
Site Plot Camera Year Month Day Hour Minute Second
1 S123 P2 C10 2012 06 21 21 34 22
2 S10 P1 C1 2012 06 22 05 01 48
3 S187 P2 C2 2012 07 02 02 35 01
in Base R:
type.convert(strcapture('(\\w+)\\.(\\w+)\\.(\\w+)_(\\d{4})(\\d{2})(\\d{2})_(\\d{2})(\\d{2})(\\d{2})',
file.names, as.list(setNames(character(length(nms)), nms))), as.is = TRUE)
Site Plot Camera Year Month Day Hour Minute Second
1 S123 P2 C10 2012 6 21 21 34 22
2 S10 P1 C1 2012 6 22 5 1 48
3 S187 P2 C2 2012 7 2 2 35 1
This is a specific case where your data is pretty neatly formatted with fields separated by either _ or ., and where the date and time fields have uniform character length. That means you can skip doing regex and instead just split by those delimeters, drop the substrings into a data frame, then separate the date components and the time components by their positions. As is often the case, as a tidyverse solution you're trading writing extra code for it being pretty easy to follow and scale.
library(magrittr)
strsplit(file.names, split = "[._]") %>%
purrr::map_dfr(setNames, c("site", "plot", "camera", "date", "time", "ext")) %>%
tidyr::separate(date, into = c("year", "month", "day"), sep = c(4, 6)) %>%
tidyr::separate(time, into = c("hour", "minute", "second"), sep = c(2, 4)) %>%
dplyr::select(-ext)
#> # A tibble: 3 × 9
#> site plot camera year month day hour minute second
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 S123 P2 C10 2012 06 21 21 34 22
#> 2 S10 P1 C1 2012 06 22 05 01 48
#> 3 S187 P2 C2 2012 07 02 02 35 01
The ext column was leftover from the initial string splitting, so you can drop it.
I don't know anything about str_sub or str_split other than the fact that they may be efforts to adapt the sub and strsplit functions to an alternate universe. I just learned base R and have not really seen the need to learn a new syntax. Here's a base solution:
as.POSIXct( sub( "([^_]+[_])(\\d{8})[_](\\d{6})", "\\2 \\4", file.names) , format="%Y%m%d %H%M%S")
[1] "2012-06-21" "2012-06-22" "2012-07-02"
You can real the sub pattern as
1) beginning with the start of the string collect all the non-underscore characters into the first capture group
2) Then get the next 8 digits (if they exist) in a second capture group
3) and everything that follows will be in a third capture group
The substitution is to just return the contents of the second capture group. The conversion to Date values is straightforward. I'm assuming that should be clear from the code, but if not then see ?as.Date.
Here's the rest;
as.POSIXct( sub( "([^_]+[_])(\\d{8})[_](\\d{6})(.+$)", "\\2 \\3", file.names) ,
format="%Y%m%d %H%M%S")
[1] "2012-06-21 21:34:22 PDT" "2012-06-22 05:01:48 PDT" "2012-07-02 02:35:01 PDT"
If you want the break out then convert to POSIXlt and extract the resulting list.
Related
I have a dataframe with monthly data, one column containing the year and one column containing the month. I'd like to combine them into one column with Date format, going from this:
Year Month Data
2020 1 54
2020 2 58
2020 3 78
2020 4 59
To this:
Date Data
2020-01 54
2020-02 58
2020-03 78
2020-04 59
I think you can't represent a Date format in R without showing the day. If you want a character column, like in your example, you can do:
> x <- data.frame(Year = c(2020,2020,2020,2020), Month = c(1,2,3,4), Data = c(54,58,78,59))
> x$Month <- ifelse(nchar(x$Month == 1), paste0(0, x$Month), x$Month) # add 0 behind.
> x$Date <- paste(x$Year, x$Month, sep = '-')
> x
Year Month Data Date
1 2020 01 54 2020-01
2 2020 02 58 2020-02
3 2020 03 78 2020-03
4 2020 04 59 2020-04
> class(x$Date)
[1] "character"
If you want a Date type column you will have to add:
x$Date <- paste0(x$Date, '-01')
x$Date <- as.Date(x$Date, format = '%Y-%m-%d')
x
class(x$Date)
Maybe the simplest way would be to arbitrarily set a day (e.g. 01) to all your dates ? Therefore date intervals would be preserved.
data<-data.frame(Year=c(2020,2020,2020,2020), Month=c(1,2,3,4), Data=c(54,58,78,59))
data$Date<-gsub(" ","",paste(data$Year,"-",data$Month,"-","01"))
data$Date<-as.Date(data$Date,format="%Y-%m-%d")
You can use sprintf -
sprintf('%d-%02d', data$Year, data$Month)
#[1] "2020-01" "2020-02" "2020-03" "2020-04"
I am trying to convert a column in my dataset that contains week numbers into weekly Dates. I was trying to use the lubridate package but could not find a solution. The dataset looks like the one below:
df <- tibble(week = c("202009", "202010", "202011","202012", "202013", "202014"),
Revenue = c(4543, 6764, 2324, 5674, 2232, 2323))
So I would like to create a Date column with in a weekly format e.g. (2020-03-07, 2020-03-14).
Would anyone know how to convert these week numbers into weekly dates?
Maybe there is a more automated way, but try something like this. I think this gets the right days, I looked at a 2020 calendar and counted. But if something is off, its a matter of playing with the (week - 1) * 7 - 1 component to return what you want.
This just grabs the first day of the year, adds x weeks worth of days, and then uses ceiling_date() to find the next Sunday.
library(dplyr)
library(tidyr)
library(lubridate)
df %>%
separate(week, c("year", "week"), sep = 4, convert = TRUE) %>%
mutate(date = ceiling_date(ymd(paste(year, "01", "01", sep = "-")) +
(week - 1) * 7 - 1, "week", week_start = 7))
# # A tibble: 6 x 4
# year week Revenue date
# <int> <int> <dbl> <date>
# 1 2020 9 4543 2020-03-01
# 2 2020 10 6764 2020-03-08
# 3 2020 11 2324 2020-03-15
# 4 2020 12 5674 2020-03-22
# 5 2020 13 2232 2020-03-29
# 6 2020 14 2323 2020-04-05
I have a data frame (dates) that looks like this:
year month start end
2000 06 01 10
2000 06 11 20
2000 06 21 30
I want to create a vector of character strings (one for each row in the data frame) so that each date follows this format:
year month start-end (first row would be 2000 06 01-10).
I've tried using a for loop with the paste function:
titles <- character()
for (i in 1:nrow(dates)){
titles[i] <- paste(dates[i, 1], dates[i,2], dates[i,3], dates[i,4])
}
> titles
[1] "2000 06 01 10" "2000 06 11 20" "2000 06 21 30"
but I can't figure out how to replace the last space with a dash. Is there a way to coerce the paste function into doing this or is there another function I can use?
Thanks for the help
Following your solution, if you just replace
paste(dates[i, 1], dates[i,2], dates[i,3], dates[i,4])
with
paste(dates[i, 1], dates[i,2], paste(dates[i,3], dates[i,4], sep = "-"))
that should work already. This just nests the "-" separating paste within the " " separating paste (default of paste is " ").
A more elegant one-liner would be to use apply:
apply(dates, 1, function(row)paste(row[1], row[2], paste(row[3], row[4], sep = "-")))
[1] "2000 06 01-10" "2000 06 11-20" "2000 06 21-30"
Instead of a loop, you may want to consider:
df$titles <- with(df, paste(year, month, start, end, sep = "-"))
df
# year month start end titles
# 1 2000 06 01 10 2000-06-01-10
# 2 2000 06 11 20 2000-06-11-20
# 3 2000 06 21 30 2000-06-21-30
We can use unite from tidyr:
library(tidyverse)
df %>%
unite("new_date", year:end, sep = " ") %>%
mutate(new_date = sub("\\s(\\d+)$", "-\\1", new_date))
or with two unite's:
df %>%
unite("temp_date", year:start, sep = " ") %>%
unite("new_date", temp_date, end, sep = "-")
Output:
new_date
1 2000 6 1-10
2 2000 6 11-20
3 2000 6 21-30
I have a column with date formatted as MM-DD-YYYY, in the Date format.
I want to add 2 columns one which only contains YYYY and the other only contains MM.
How do I do this?
Once again base R gives you all you need, and you should not do this with sub-strings.
Here we first create a data.frame with a proper Date column. If your date is in text format, parse it first with as.Date() or my anytime::anydate() (which does not need formats).
Then given the date creating year and month is simple:
R> df <- data.frame(date=Sys.Date()+seq(1,by=30,len=10))
R> df[, "year"] <- format(df[,"date"], "%Y")
R> df[, "month"] <- format(df[,"date"], "%m")
R> df
date year month
1 2017-12-29 2017 12
2 2018-01-28 2018 01
3 2018-02-27 2018 02
4 2018-03-29 2018 03
5 2018-04-28 2018 04
6 2018-05-28 2018 05
7 2018-06-27 2018 06
8 2018-07-27 2018 07
9 2018-08-26 2018 08
10 2018-09-25 2018 09
R>
If you want year or month as integers, you can wrap as as.integer() around the format.
A base R option would be to remove the substring with sub and then read with read.table
df1[c('month', 'year')] <- read.table(text=sub("-\\d{2}-", ",", df1$date), sep=",")
Or using tidyverse
library(tidyverse)
separate(df1, date, into = c('month', 'day', 'year') %>%
select(-day)
Note: it may be better to convert to datetime class instead of using the string formatting.
df1 %>%
mutate(date =mdy(date), month = month(date), year = year(date))
data
df1 <- data.frame(date = c("05-21-2017", "06-25-2015"))
I have a large data set (as a csv) and wish to calculate the time between dates. What is the most efficient way to do this?
eg, data:
ID start end
01 01-04-2017 05-04-2017
01 04-04-2017 06-04-2017
01 11-04-2017 21-04-2017
02 19-05-2017 22-05-2017
02 22-05-2017 24-05-2017
02 02-06-2017 05-06-2017
02 09-06-2017 12-06-2017
...
It's not so simple because there may be overlaps - as shown above.
What I'd like as output is:
ID time
01 15
02 11
...
I've thought about splitting the data into a list based on the ID (split(dataframe(df$start, df$end), df$ID)) but this is slow for a large dataframe. I've also considered looping through the df and comparing the differences but this is also slow.
Is there an efficient way to do this in R?
You can use findInterval to check which interval of start dates each value of end falls into. If they overlap, two will have the same interval, which can be used for grouping and aggregation to eliminate overlaps:
library(dplyr)
df <- read.table(text = 'ID start end
01 01-04-2017 05-04-2017
01 04-04-2017 06-04-2017
01 11-04-2017 21-04-2017
02 19-05-2017 22-05-2017
02 22-05-2017 24-05-2017
02 02-06-2017 05-06-2017
02 09-06-2017 12-06-2017', header = TRUE, colClasses = 'character') %>%
mutate_at(-1, as.Date, format = '%d-%m-%Y') # parse dates
df_aggregated <- df %>%
group_by(ID) %>%
group_by(ID, overlap = findInterval(end, start)) %>%
summarise(start = min(start), end = max(end)) %>%
select(-overlap) %>% ungroup() # clean up
df_aggregated
#> # A tibble: 5 × 3
#> ID start end
#> <chr> <date> <date>
#> 1 01 2017-04-01 2017-04-06
#> 2 01 2017-04-11 2017-04-21
#> 3 02 2017-05-19 2017-05-24
#> 4 02 2017-06-02 2017-06-05
#> 5 02 2017-06-09 2017-06-12
Once the data is tidied, summarizing is easy:
df_aggregated %>% group_by(ID) %>% summarise(span = sum(end - start))
#> # A tibble: 2 × 2
#> ID span
#> <chr> <time>
#> 1 01 15 days
#> 2 02 11 days
This approach assumes each group is ordered by start; if not, add arrange(start).