I am importing some weather data, but the timestamp is split across different columns. I want to join these columns and create posix objects out of them.
datenum <- c()
for (i in 1:dim(weather)[1]){
date_string <- paste0(weather$Year.UTC[i],'-',weather$Month.UTC[i],'-',weather$Day.UTC[i],'-',weather$Hour.UTC[i]) # different columns of data
# for i = 1, date_string = "2012-12-31-23"
datenum[i] <- as.POSIXct(date_string, format="%Y-%m-%d-%H",tz="GMT", origin = "1960-01-01")
# for i = 1, datenum[1] = 1356994800 (numeric)
}
as.Date(datenum[1], origin = "1960-01-01")
# Gives character = "7285-07-27"
To visually confirm that I am doing it right, I would like to see a string in the form "yyyy-mm-dd HH:MM:SS", which is what I try to obtain with as.Date. The origin is the same when converting to a serial number and back to a character, but the date is completely wrong. What I am doing wrong?
Why so complicated?
weather <- data.frame(Year.UTC=c(2012, 2013),
Month.UTC=c(1,2),
Day.UTC=c(1,2),
Hour.UTC=c(22,23))
weather <- within(weather, datetime <-
as.POSIXct(paste(Year.UTC, Month.UTC, Day.UTC, Hour.UTC, sep="-"),
format="%Y-%m-%d-%H", tz="UTC"))
# Year.UTC Month.UTC Day.UTC Hour.UTC datetime
#1 2012 1 1 22 2012-01-01 22:00:00
#2 2013 2 2 23 2013-02-02 23:00:00
As you see, you don't need a loop at all.
Related
I have a column of data with two different formats: yyyyqq and yyyyy. I want to reformat the column to mmddyyyyy.
Whenever I use the following command as.Date(as.character(x), format = "%y") the output is yyyy-12-03. I cannot get any other combination of as.Date to work.
I'm sure this is a simple fix, but how do I do this?
Using the following assumptions:
2021 <- 2021-01-01
2021Q1 <- 2021-01-01
2021Q2 <- 2021-04-01
2021Q3 <- 2021-07-01
2021Q4 <- 2021-10-01
You can use the following:
as.Date(paste(substr(x, 1, 4), 3*as.numeric(max(substr(x, 6, 6),1))-2, "1", sep = "-"))
Edit: You can wrap this in a format(..., "%m%d%Y) but as already said in the comments I would not recommend it.
Here is a function which translates to the first (if frac=0) or last (if frac=1) date of the period. First append a 01 (first of the period) or 04 (last of the period) to the end of the input. That puts them all in yyyyqq format possibly with junk at the end. Then yearqtr will convert to a yearqtr object ignoring any junk. Then convert that to a Date object. as.Date.yearqtr uses the same meaning for frac. Finally format it as a character string in mm/dd/yyyy format.
(One alternative is to replace the format(...) line with chron::as.chron() in which case it will render in the same manner, since the format specified is the default for chron, but be a chron dates object which can be manipulated more conveniently, e.g. it sorts chronologically, than a character string.)
library(zoo)
to_date <- function(x, frac = 1) x |>
paste0(if (frac == 1) "04" else "01") |>
as.yearqtr("%Y%q") |>
as.Date(frac = frac) |>
format("%m/%d/%Y")
# test data
dd <- data.frame(x = c(2001, 2003, 200202, 200503))
transform(dd, first = to_date(x, frac = 0), last = to_date(x, frac = 1))
giving:
x first last
1 2001 01/01/2001 12/31/2001
2 2003 01/01/2003 12/31/2003
3 200202 04/01/2002 06/30/2002
4 200503 07/01/2005 09/30/2005
For a time series analysis of over 1000 raster in a raster stack I need the date. The data is almost weekly in the structure of the files
"... 1981036 .... tif"
The zero separates year and week
I need something like: "1981-36"
but always get the error
Error in charToDate (x): character string is not in a standard unambiguous format
library(sp)
library(lubridate)
library(raster)
library(Zoo)
raster_path <- ".../AVHRR_All"
all_raster <- list.files(raster_path,full.names = TRUE,pattern = ".tif$")
all_raster
brings me:
all_raster
".../VHP.G04.C07.NC.P1981036.SM.SMN.Andes.tif"
".../VHP.G04.C07.NC.P1981037.SM.SMN.Andes.tif"
".../VHP.G04.C07.NC.P1981038.SM.SMN.Andes.tif"
…
To get the year and the associated week, I have used the following code:
timeline <- data.frame(
year= as.numeric(substr(basename(all_raster), start = 17, stop = 17+3)),
week= as.numeric(substr(basename(all_raster), 21, 21+2))
)
timeline
brings me:
timeline
year week
1 1981 35
2 1981 36
3 1981 37
4 1981 38
…
But I need something like = "1981-35" to be able to plot my time series later
I tried that:
timeline$week <- as.Date(paste0(timeline$year, "%Y")) + week(timeline$week -1, "%U")
and get the error:Error in charToDate(x) : character string is not in a standard unambiguous format
or I tried that
fileDates <- as.POSIXct(substr((all_raster),17,23), format="%y0%U")
and get the same error
until someone will post a better way to do this, you could try:
x <- c(".../VHP.G04.C07.NC.P1981036.SM.SMN.Andes.tif", ".../VHP.G04.C07.NC.P1981037.SM.SMN.Andes.tif",
".../VHP.G04.C07.NC.P1981038.SM.SMN.Andes.tif")
xx <- substr(x, 21, 27)
library(lubridate)
dates <- strsplit(xx,"0")
dates <- sapply(dates,function(x) {
year_week <- unlist(x)
year <- year_week[1]
week <- year_week[2]
start_date <- as.Date(paste0(year,'-01-01'))
date <- start_date+weeks(week)
#note here: OP asked for beginning of week.
#There's some ambiguity here, the above is end-of-week;
#uncommment here for beginning of week, just subtracted 6 days.
#I think this might yield inconsistent results, especially year-boundaries
#hence suggestion to use end of week. See below for possible solution
#date <- start_date+weeks(week)-days(6)
return (as.character(date))
})
newdates <- as.POSIXct(dates)
format(newdates, "%Y-%W")
Thanks to #Soren who posted this anwer here: Get the month from the week of the year
You can do it if you specify that Monday is a Weekday 1 with %u:
w <- c(35,36,37,38)
y <- c(1981,1981,1981,1981)
s <- c(1,1,1,1)
df <- data.frame(y,w,s)
df$d <- paste(as.character(df$y), as.character(df$w),as.character(df$s), sep=".")
df$date <- as.Date(df$d, "%Y.%U.%u")
# So here we have variable date as date if you need that for later.
class(df$date)
#[1] "Date"
# If you want it to look like Y-W, you can do the final formatting:
df$date <- format(df$date, "%Y-%U")
# y w s d date
# 1 1981 35 1 1981.35.1 1981-35
# 2 1981 36 1 1981.36.1 1981-36
# 3 1981 37 1 1981.37.1 1981-37
# 4 1981 38 1 1981.38.1 1981-38
# NB: though it looks correct, the resulting df$date is actually a character:
class(df$date)
#[1] "character"
Alternatively, you could do the same by setting the Sunday as 0 with %w.
I had a column with date datatype. in my column the dates are in 4/1/2007 format. now I want to extract month value from that column and date value from that column in different column in R. my date are from 01/01/2012 to 01/01/ 2015 plz help me.
If your variable is date type (as you say in the post) simply use following to extract month:
month_var = format(df$datecolumn, "%m") # this will give output like "09"
month_var = format(df$datecolumn, "%b") # this will give output like "Sep"
month_var = format(df$datecolumn, "%B") # this will give output like "September"
If your date variable in not in date format, then you will have to convert them into date format.
df$datecolumn<- as.Date(x, format = "%m/%d/%Y")
Assuming your initial data is character and not POSIX.
df <- data.frame(d = c("4/1/2007", "01/01/2012", "02/01/2015"),
stringsAsFactors = FALSE)
df
# d
# 1 4/1/2007
# 2 01/01/2012
# 3 02/01/2015
These are not yet "dates", just strings.
df$d2 = as.POSIXct(df$d, format = "%m/%d/%Y")
df
# d d2
# 1 4/1/2007 2007-04-01
# 2 01/01/2012 2012-01-01
# 3 02/01/2015 2015-02-01
Now they proper dates (in the R fashion). These two lines extract just a single component from each "date"; see ?strptime for details on all available formats.
df$dY = format(df$d2, "%Y")
df$dm = format(df$d2, "%m")
df
# d d2 dY dm
# 1 4/1/2007 2007-04-01 2007 04
# 2 01/01/2012 2012-01-01 2012 01
# 3 02/01/2015 2015-02-01 2015 02
An alternative method would be to extract the substrings from each string, but now you're getting into regex-pain; for that, I'd suggest sticking with somebody else's regex lessons-learned, and translate through POSIXct (or even POSIXlt if you want).
I have my data in following format.
x <- c("2012-03-01T00:05:55+00:00", "2012-03-01T00:06:23+00:00",
"2012-03-01T00:06:52+00:00")
Actual data is very long.
My objective is
convert them to hourly time-series in R
to aggregate my data to hourly data
First convert your dates into a date-time class using asPOSIXct
df = data.frame(x = c("2012-03-01T00:05:55+00:00", "2012-03-01T00:06:23+00:00",
"2012-03-01T00:06:52+00:00"))
df$times = as.POSIXct(df$x, format = "%Y-%m-%dT00:%H:%M+%S")
Then extract just the hour part using format
df$hour = format(df$times, '%H')
This give you :
x times hour
1 2012-03-01T00:05:55+00:00 2012-03-01 05:55:00 05
2 2012-03-01T00:06:23+00:00 2012-03-01 06:23:00 06
3 2012-03-01T00:06:52+00:00 2012-03-01 06:52:00 06
Or you can extract the date and the hour using:
df$date_hour = format(df$times, '%Y-%m-%d:%H')
for more infor see ?strftime it says "A conversion specification is introduced by %, usually followed by a single letter or O or E and then a single letter. Any character in the format string not part of a conversion specification is interpreted literally (and %% gives %). Widely implemented conversion specifications include:... %H
Hours as decimal number (00–23). As a special exception strings such as 24:00:00 are accepted for input, since ISO 8601 allows these."
Now you can do any aggregartion you want using something like plyr::ddply
library(plyr)
ddply(df, .(hour), nrow)
hour V1
1 05 1
2 06 2
or
ddply(df, .(date_hour), nrow)
date_hour V1
1 2012-03-01:05 1
2 2012-03-01:06 2
I need to convert date (m/d/y format) into 3 separate columns on which I hope to run an algorithm.(I'm trying to convert my dates into Julian Day Numbers). Saw this suggestion for another user for separating data out into multiple columns using Oracle. I'm using R and am throughly stuck about how to code this appropriately. Would A1,A2...represent my new column headings, and what would the format difference be with the "update set" section?
update <tablename> set A1 = substr(ORIG, 1, 4),
A2 = substr(ORIG, 5, 6),
A3 = substr(ORIG, 11, 6),
A4 = substr(ORIG, 17, 5);
I'm trying hard to improve my skills in R but cannot figure this one...any help is much appreciated. Thanks in advance... :)
I use the format() method for Date objects to pull apart dates in R. Using Dirk's datetext, here is how I would go about breaking up a date into its constituent parts:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
datetxt <- as.Date(datetxt)
df <- data.frame(date = datetxt,
year = as.numeric(format(datetxt, format = "%Y")),
month = as.numeric(format(datetxt, format = "%m")),
day = as.numeric(format(datetxt, format = "%d")))
Which gives:
> df
date year month day
1 2010-01-02 2010 1 2
2 2010-02-03 2010 2 3
3 2010-09-10 2010 9 10
Note what several others have said; you can get the Julian dates without splitting out the various date components. I added this answer to show how you could do the breaking apart if you needed it for something else.
Given a text variable x, like this:
> x
[1] "10/3/2001"
then:
> as.Date(x,"%m/%d/%Y")
[1] "2001-10-03"
converts it to a date object. Then, if you need it:
> julian(as.Date(x,"%m/%d/%Y"))
[1] 11598
attr(,"origin")
[1] "1970-01-01"
gives you a Julian date (relative to 1970-01-01).
Don't try the substring thing...
See help(as.Date) for more.
Quick ones:
Julian date converters already exist in base R, see eg help(julian).
One approach may be to parse the date as a POSIXlt and to then read off the components. Other date / time classes and packages will work too but there is something to be said for base R.
Parsing dates as string is almost always a bad approach.
Here is an example:
datetxt <- c("2010-01-02", "2010-02-03", "2010-09-10")
dates <- as.Date(datetxt) ## you could examine these as well
plt <- as.POSIXlt(dates) ## now as POSIXlt types
plt[["year"]] + 1900 ## years are with offset 1900
#[1] 2010 2010 2010
plt[["mon"]] + 1 ## and months are on the 0 .. 11 intervasl
#[1] 1 2 9
plt[["mday"]]
#[1] 2 3 10
df <- data.frame(year=plt[["year"]] + 1900,
month=plt[["mon"]] + 1, day=plt[["mday"]])
df
# year month day
#1 2010 1 2
#2 2010 2 3
#3 2010 9 10
And of course
julian(dates)
#[1] 14611 14643 14862
#attr(,"origin")
#[1] "1970-01-01"
To convert date (m/d/y format) into 3 separate columns,consider the df,
df <- data.frame(date = c("01-02-18", "02-20-18", "03-23-18"))
df
date
1 01-02-18
2 02-20-18
3 03-23-18
Convert to date format
df$date <- as.Date(df$date, format="%m-%d-%y")
df
date
1 2018-01-02
2 2018-02-20
3 2018-03-23
To get three seperate columns with year, month and date,
library(lubridate)
df$year <- year(ymd(df$date))
df$month <- month(ymd(df$date))
df$day <- day(ymd(df$date))
df
date year month day
1 2018-01-02 2018 1 2
2 2018-02-20 2018 2 20
3 2018-03-23 2018 3 23
Hope this helps.
Hi Gavin: another way [using your idea] is:
The data-frame we will use is oilstocks which contains a variety of variables related to the changes over time of the oil and gas stocks.
The variables are:
colnames(stocks)
"bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC"
"emMN" "emMN.1" "chdate" "chV" "cbO" "chC" "chMN" "chMX"
One of the first things to do is change the emdate field, which is an integer vector, into a date vector.
realdate<-as.Date(emdate,format="%m/%d/%Y")
Next we want to split emdate column into three separate columns representing month, day and year using the idea supplied by you.
> dfdate <- data.frame(date=realdate)
year=as.numeric (format(realdate,"%Y"))
month=as.numeric (format(realdate,"%m"))
day=as.numeric (format(realdate,"%d"))
ls() will include the individual vectors, day, month, year and dfdate.
Now merge the dfdate, day, month, year into the original data-frame [stocks].
ostocks<-cbind(dfdate,day,month,year,stocks)
colnames(ostocks)
"date" "day" "month" "year" "bpV" "bpO" "bpC" "bpMN" "bpMX" "emdate" "emV" "emO" "emC" "emMN" "emMX" "chdate" "chV"
"cbO" "chC" "chMN" "chMX"
Similar results and I also have date, day, month, year as separate vectors outside of the df.