I have a vector of strings that include a repeating pattern of start and end dates for variables collected at a site. Here is the first entry:
"1942-10-06,1996-03-31Snow Depth (in/mm)1942-11-01,1996-03-31Snowfall (in/mm)1942-10-01,1997-12-27Growing Degree DaysHeating Degree DaysAverage Temperature (F/C)Maximum Temperature (F/C)1950-08-01,1970-03-31Observation Time Temperature (F/C)1942-10-01,1997-12-27Minimum Temperature (F/C)1942-10-01,1996-03-31Precipitation (in/mm)"
Can someone help me reformat each string into a table that includes the start date, end date, and variable name?
The below code should work following some assumptions about the way your data are formatted:
Your start dates are in "yyyy-mm-dd" or "yyyy-dd-mm" format and are
followed by a comma,
Your end dates are in the same format as your start dates and follow
a comma, and
Your variable names follow an end date and contain no
numbers.
As alluded to by Oriol Mirosa these assumptions may not hold.
# Your string
string = "1942-10-06,1996-03-31Snow Depth (in/mm)1942-11-01,1996-03-31Snowfall (in/mm)1942-10-01,1997-12-27Growing Degree DaysHeating Degree DaysAverage Temperature (F/C)Maximum Temperature (F/C)1950-08-01,1970-03-31Observation Time Temperature (F/C)1942-10-01,1997-12-27Minimum Temperature (F/C)1942-10-01,1996-03-31Precipitation (in/mm)"
# Extract text matching Assumptions 1-3, respectively, above
library(stringr)
start_dates = str_extract_all(string, "[0-9]{4}-[0-9]{2}-[0-9]{2},")
end_dates = str_extract_all(string, ",[0-9]{4}-[0-9]{2}-[0-9]{2}")
var_names = str_extract_all(string,
",[0-9]{4}-[0-9]{2}-[0-9]{2}([^[0-9]])+")
# Remove the irrelevant bits (e.g., leading/trailing commas)
start_dates = as.Date(gsub(",", "", unlist(start_dates))) #remove ","
end_dates = as.Date(gsub(",", "", unlist(end_dates))) #remove ","
var_names = gsub(",[0-9]{4}-[0-9]{2}-[0-9]{2}", "", unlist(var_names))
# Put into table
X = data.frame("Start_date" = start_dates,
"End_date" = end_dates,
"Var_name" = var_names)
Related
Hello everyone I hope you guys are having a good one,
I have the following date frame:
ID TX
GROUP
HUDJDUDOOD--BANNK2--OLDODOLD985555545UIJF
1
UJDID YUH23498 IDX09
2
854 UIJSAZXC
3
I would like to be able to extract the longest string for each value under the column ID TX knowing that each cell may have different strings or maybe just one but in some instances they may be separated by punctuation such as "," "--", "," "--" ect or even a space " ".
I have thought of the following I need to first replace punctuation by a white space " " then.. separate or split each cell by " " after that I will calculate the length of each string perhaps with nchart() or str_length() and select the index of the string the the longest value, but I have not been able yet to do so as I cant mannage to select the index (word) that I need after splitting the values since I dont know in what index the longest string may be.. my desired output would be:
OUTPUT
OLDODOLD985555545UIJF
YUH23498
UIJSAZXC
sidenote: no worries there will not be ties.
Thank you so much guys for your help I will be very alert to award you for your response!
# Your data
dat <- structure(list(ID_TX = c("HUDJDUDOOD--BANNK2--OLDODOLD985555545UIJF",
"UJDID YUH23498 IDX09", "854 UIJSAZXC"), GROUP = 1:3), class = "data.frame", row.names = c(NA,
-3L))
# Splitting strings in the data
spl <- strsplit(dat$ID_TX, "--|\\s")
# Identify the position of the longest string in each row
idx <- spl|> lapply(nchar) |> lapply(which.max) |> unlist()
# Select the longest string and bind them to a data.frame
mapply(function(x,y) spl[[x]][y], seq_along(idx),idx) |>
as.data.frame() |>
setNames("OUTPUT")
# The result
# OUTPUT
#1 OLDODOLD985555545UIJF
#2 YUH23498
#3 UIJSAZXC
Trying to add a new column in my data table denoting the month (either as a numeric value or character) using an already available column of "SetDate", which is in the format mdy.
I'm new to R and having trouble. Thank you
base solution:
f = "%m/%d/%y" # note the lowercase y; it's because the year is 92, not 1992
dataset$SetDateMonth <- format(as.POSIXct(dataset$SetDate, format = f), "%m")
Basically, what it does is it converts the column from character (presumed class) to POSIXct, which allows for an easy extraction of month information.
Quick test:
format(as.POSIXct('1/1/92', format = "%m/%d/%y"), "%m")
[1] "01"
Try this (created a small example):
library(lubridate)
date_example <- "1/1/92"
lubridate::mdy(date_example)
[1] "1992-01-01"
lubridate::mdy(date_example) %>% lubridate::month()
[1] 1
If you want full month as character string, use:
lubridate::mdy(date_example) %>% lubridate::month(label = TRUE, abbr = FALSE)
I have a large dataframe of 22641 obs. and 12 variables.
The first column "year" includes extracted values from satellite images in the format below.
1_1_1_1_LT05_127024_19870517_00005ff8aac6b6bf60bc
From this format, I only want to keep the date which in this case is 19870517 and format it as date (so two different things). Usually, I use the regex to extract the words that I want, but here the date is different for each cell and I have no idea how to replace the above text with only the date. Maybe the way to do this is to search by position within the sentence but I do not know how.
Any ideas?
Thanks.
It's not clear what the "date is different in each cell" means but if it means that the value of the date is different and it is always the 7th field then either of (1) or (2) will work. If it either means that it consists of 8 consecutive digits anywhere in the text or 8 consecutive digits surrounded by _ anywhere in the text then see (3).
1) Assuming the input DF shown in reproducible form in the Note at the end use read.table to read year, pick out the 7th field and then convert it to Date class. No packages are used.
transform(read.table(text = DF$year, sep = "_")[7],
year = as.Date(as.character(V7), "%Y%m%d"), V7 = NULL)
## year
## 1 1987-05-17
2) Another alternative is separate in tidyr. 0.8.2 or later is needed.
library(dplyr)
library(tidyr)
DF %>%
separate(year, c(rep(NA, 6), "year"), extra = "drop") %>%
mutate(year = as.Date(as.character(year), "%Y%m%d"))
## year
## 1 1987-05-17
3) This assumes that the date is the only sequence of 8 digits in the year field use this or if we know it is surrounded by _ delimiters then the regular expression "_(\\d{8})_" can be used instead.
library(gsubfn)
transform(DF,
year = do.call("c", strapply(DF$year, "\\d{8}", ~ as.Date(x, "%Y%m%d"))))
## year
## 1 1987-05-17
Note
DF <- data.frame(year = "1_1_1_1_LT05_127024_19870517_00005ff8aac6b6bf60bc",
stringsAsFactors = FALSE)
Not sure if this will generalize to your whole data but maybe:
gsub(
'(^(?:.*?[^0-9])?)(\\d{8})((?:[^0-9].*)?$)',
'\\2',
'1_1_1_1_LT05_127024_19870517_00005ff8aac6b6bf60bc',
perl = TRUE
)
## [1] "19870517"
This uses group capturing and throws away anything but bounded 8 digit strings.
You can use sub to extract the data string and as.Date to convert it into R's date format:
as.Date(sub(".+?([0-9]+)_[^_]+$", "\\1", txt), "%Y%m%d")
# [1] "1987-05-17"
where txt <- "1_1_1_1_LT05_127024_19870517_00005ff8aac6b6bf60bc"
I have a vector of birth dates as character strings formatted "10-Feb-85".
When I use the as.Date() function in R it assumes the two digit year is after 2000 (none of these birth dates are after the year 2000).
example:
as.Date(x = "10-Feb-52", format = "%d-%b-%y")
returns: 2052-02-10
I'm not proficient in regular expressions but
I think that this is an occasion for a regular expression to insert a "19" after the second "-" or before the last two digits.
I've found a regex that counts forward three characters and inserts a letter:
gsub(pattern = "^(.{3})(.*)$", replacement = "\\1d\\2", x = "abcefg")
But I'm not sure how to count two from the end.
Any help is appreciated.
insert a "19" after the second "-" or before the last two digits.
Before the last two digits:
gsub(pattern = "-(\\d{2})$", replacement = "-19\\1", x = "10-Feb-52")
See the R demo. Here, - is matched first, then 2 digits ((\\d{2})) - that are at the end of string ($) - are matched and captured into Group 1.
After the second -:
gsub(pattern = "^((?:[^-]*-){2})", replacement = "\\119", x = "10-Feb-52")
See another demo. Here, 2 sequences ({2}) of 0+ chars other than - ([^-]*) are matched from the start of the string (^) and captured into group 1. The replacement contains a backreference that restores the captured text in the replacement result.
Slight problem where my as.Date function gives a different result when I put it in a for loop. I'm looking in a folder with subfolders (per date) that contain images. I build date_list to organize all the dates (for plotting options in a later stage). The Julian Day starts from the first of January of the year, so because I have 4 years of date, the year must be flexible.
# Set up list with 4 columns and counter Q. jan is used to set all dates to the first of january
date_list <- outer(1:52, 1:4)
q = 1
jan <- "-01-01"
for (scene in folders){
year <- as.numeric(substr(scene, start=10, stop=13))
day <- as.numeric(substr(scene, start=14, stop=16))
datum <- paste(year, day, sep='_')
date_list[q, 1] <- datum
date_list[q, 2] <- year
date_list[q, 3] <- day
date_list[q, 4] <- as.Date(day, origin = as.Date(paste(year,jan, sep="")))
q = q+1
}
Output final row:
[52,] "2016_267" "2016" "267" "17068"
What am i missing in date_list[q, 4] that doesn't transfer my integer to a date?
running the following code does work, but due to the large amount of scenes and folders I like to automate this:
as.Date(day, origin = as.Date(paste(year,jan, sep="")))
Thank you for your time!
Well, I assume this would answer your first question:
date_list[q, 4] <- as.character(as.Date(datum,format="%Y_%j"))
as.Date accept a format argument, (the %Y and %j are documented in strptime), the %jis the julian day, this is a little easier to read than using origin and multiple paste calls.
Your problem is actually linked to what a Date object is:
> dput(as.Date("2016-01-10"))
structure(16810, class = "Date")
When entered into a matrix (your date_list) it is coerced to character w
without special treatment before like this:
> d<-as.Date("2016-01-10")
> class(d)<-"character"
> d
[1] "16810"
Hence you get only the number of days since 1970-01-01. When you ask for the date as character representation with as.character, it gives the correct value because the Date class as a as.character method which first compute the date in human format before returning a character value.
Now if I understood well your problem I would go this way:
First create a function to work on one string:
name_to_list <- function(name) {
dpart <- substr(name, start=10, stop=16)
date <- as.POSIXlt(dpart, format="%Y%j")
c("datum"=paste(date$year+1900,date$yday,sep="_"), "year"=date$year+1900, "julian_day"=date$yday, "date"=as.character(date) )
}
this function just get your substring, and then convert it to POSIXlt class, which give us julian day, year and date in one pass. as the year is stored as integer since 1900 (could be negative), we have to add 1900 when storing the year in the fields.
Then if your folders variable is a vector of string:
lapply(folders,name_to_list)
wich for folders=c("LC81730382016267LGN00","LC81730382016287LGN00","LC81730382016167LGN00") gives:
[[1]]
datum year julian_day date
"2016_266" "2016" "266" "2016-09-23"
[[2]]
datum year julian_day date
"2016_286" "2016" "286" "2016-10-13"
[[3]]
datum year julian_day date
"2016_166" "2016" "166" "2016-06-15"
Do you mean to output your day as 3 numbers? Should it not be 2 numbers?
day <- as.numeric(substr(scene, start=15, stop=16))
or
day <- as.numeric(substr(scene, start=14, stop=15))
That could at least be part of the issue. Providing an example of what typical values of "scene" are would be helpful here.