I am retrieving data from an ArcGIS Rest service using a web request in an R script. One of the attribute columns does contain a date value. These dates are returned as epoch values (character strings). I try to convert these epoch strings to human readable dates, but until now to no avail...
See my reproducible script below. The column containing the date is called WVK_BEGDAT. The content of this column should be converted to human readable dates.
I have tried several suggestions found via Google...
library(httr)
library(sf)
library(lubridate)
url <- parse_url("https://services.arcgis.com/nSZVuSZjHpEZZbRo/arcgis/rest/services")
url$path <- paste(url$path, "NWB_Wegvakken/FeatureServer/0/query", sep = "/")
url$query <- list(where = "WEGBEHSRT = 'R' AND WEGNUMMER = '015'",
outFields = "*",
returnGeometry = "true",
f = "json")
request <- build_url(url)
request
NWB <- st_read(request, stringsAsFactors = FALSE)
plot(st_geometry(NWB))
NWB$WVK_BEGDAT2 <- as_date(NWB$WVK_BEGDAT, format="%d-%m-%Y", tz = "CET")
The closest I get is using the as_date() function from the lubridate package. This function at least does not return an error, bu fills the column with NAs.
Any suggestions? Thanks in advance.
as.POSIXct should do the trick:
as.POSIXct(as.numeric(NWB$WVK_BEGDAT)/1000, origin = "1970-01-01")
You have to divide by 1000, because your epoch times seem to be in milliseconds.
Related
How can I convert dates in R to a string without dashes or slashes or letters and times without colons. For example I can get 2017-12-07 in R but I need 201712071520 to use in an Weather API call. How can I do that? For reference please see the example call below for startDateTime and endDateTime. I would like to convert the dates that I have into 20171207 format and append it with a fixed time (1520) without the colon. Thanks for helping!
I have been told this question has been asked before but the other examples are doing the opposite converting character strings into R dates and times.
Here is an example of the API I am calling:
https://api.weather.com/v3/wx/hod/conditions/historical/point?pointType=nearest&geocode=39.86,-104.67&startDateTime=201712071520&endDateTime=201712071520&units=e&format=json&apiKey=yourApiKey
Moved from comments.
If x is of R's "Date" class then use the indicated format statement:
x <- as.Date("2017-12-07") # test input
format(x, "%Y%m%d1520")
## [1] "201712071520"
See ?strptime for more on percent codes.
This is a bit more generic solution. It would look like this:
library(lubridate)
input_date = "2017-1-7" #intentionally taking different date to make it more generic
fixed_text = "1520"
input_date = ymd(input_date)
output_date = paste(year(input_date), sprintf(fmt = '%02d', month(input_date)), sprintf(fmt = '%02d', day(input_date)), fixed_text, sep = "")
print(output_date)
Data: DOWNLOAD .TXT
Code:
data = read.table("DistrBdaily1yrs.txt", header = TRUE, sep = "", dec = ",")
data$DATE = as.Date(as.character(data$DATE),format="%Y%m%d")
dataXts = xts(data$QUANTITY,data$DATE, frequency = 6)
tseries = ts(dataXts, start = start(dataXts), end = end(dataXts), frequency = 6)
What I'm trying to do is to convert the xts dataXts object to a ts object with correct starting and ending date in order to use the decompose function. In this case start = start(dataXts) and end = end(dataXts) give me the right starting and ending date but tseries doesn't recognize the data column in dataXts and then think that all is data.
How can I fix this?
I am not sure I was able to "FORCE" xts to ts but i got the decompose part to function:
library("data.table")
# I was unable to read-in using read.table() for some reason.... used fread() as it is much faster
data <- fread("DistrBdaily1yrs.txt", header = TRUE, sep="\t")
# Set column names to the ones I saw on dropbox, as i was unable to read-in header for some reason!
colnames(data) <- c("DATE", "QUANTITY")
# Keep as-is
data$DATE = as.Date(as.character(data$DATE),format="%Y%m%d")
dataXts = xts(data$QUANTITY,data$DATE, frequency = 6)
# Not sure what the "QUANTITY" Column means but it must be turned into "numeric"
# You can see this post on how to do it if the following is unsatisfactory:
# http://stackoverflow.com/questions/3605807/how-to-convert-numbers-with-comma-inside-from-character-to-numeric-in-r
a<-as.numeric(gsub(",",".",dataXts))
dataXts <- reclass(a, match.to=dataXts); colnames(dataXts)<- "QUANTITY"
# Now convert it to timeSeries
timeseries <- ts(dataXts,frequency=6)
# decompose
decompose(timeseries)
Also, when I convert xts to ts I assume that it will use the first and last dates in order to construct the ts which is why i left out start = start(dataXts), end = end(dataXts) in the ts() function. Also see ?ts since you cannot pass Dates in the start or end criteria, rather:
Either a single number or a vector of two integers, which specify a natural time unit and a (1-based) number of samples into the time unit.
You can always convert back to xts using reclass:
# for example: Say you only want the trend
reclass(decompose(timeseries)$trend,match.to=dataXts)
I am working with data from csv files that will all look the same so I am hoping to come up with a code that can be easily applied to all of them.
However, sadly enough I am failing at step one :-(.
The csv files have the date and time saved in one column, so when I import them with read.csv that column gets read as a chr. How can I most easily convert this into a date that I then can use for plotting and analysis?
Here is what I tried:
load the data --> will save the date and time as chr under mydata$Date.Time (e.g. 1/1/15 0:00)
mydata<-read.csv(file.choose(), stringsAsFactors = FALSE,
strip.white = TRUE,
na.strings = c("NA",""), skip=16,
header=TRUE)
separate the Date.Time into Date and Time:
new <- do.call( rbind , strsplit( as.character( mydata$Date.Time ) , " " ) )
add these two back to the df mydata:
cbind( mydata , Date = new[,2] , Time = new[,1] )
convert Date into a date format via as.Date:
mydata$Date <- as.Date(new[,1], format="")
So this works fine for the date however I am stuck with the time, I tried this:
mydata$Time <- format(as.POSIXct(new[,2], format="%H:%M"))
this gives me the following error:
Error in as.POSIXlt.character(x, tz, ...) :
character string is not in a standard unambiguous format
I wonder if there is a smarter way of doing this? Reading in time and date seems to be one of the substantial tasks that I would like to understand. Is there a way of R directly recognizing the date and time from the csv? Or is it generally smarter to generate a time vector by its own, if so how would I do that?
Thanks so much for your help.
Sandra
If you want to use time only, consider using the chron package:
library(chron)
mytime <- times("21:19:37")
or in your case
times(new[,2])
assuming that that's a character vector.
I tried the chron approach but it wouldn't work for me :-(.
So what I ended up doing is just creating a time vector for the period that I am loading the data in for:
date <-seq(as.POSIXct("2015/1/1 00:00"), as.POSIXct("2015/1/31 23:00"), "hours")
and then adding it back to the df.
Not what I wanted but it will work until I find the ultimate solution :-)
I am still learning R, and get very confused when using various data types, classes, etc. I have run into this issue of "Dates" not being in the right format for xts countless times now, and find a solution each time after searching long and hard for (what I consider) complicated solutions.
I am looking for a way to load a CSV into R and convert the date upon loading it each time I want to load a csv into R. 99% of my files contain Date as the first column, in format 01-31-1900 (xts wants YYYY-mm-dd).
Right now I have the following:
FedYieldCurve <- read.csv("Yield Curve.csv", header = TRUE, sep = ",", stringsAsFactors = FALSE)
FedYieldCurve$Date <- format(as.Date(FedYieldCurve$Date), "%Y/%m/%d")
and i am getting: Error in charToDate(x) :
character string is not in a standard unambiguous format
The format argument must be inside as.Date. Try this (if the dates in the files are stored in the 01-31-1900 format):
as.Date(FedYieldCurve$Date,format="%m-%d-%Y")
When you try to coerce a string to a Date object you have to specify the format of the string as the format argument in the as.Date call. You have the error you reported when you try to coerce a string which has a format other than the standard YYYY-mm-dd.
Provide a few lines of the file when asking questions like this. In the absence of this we have supplied some data below in a self contained example.
Use read.zoo from the zoo package (which xts loads) specifying the format. (Replace the read.zoo line with the commented line to read from a file.)
Lines <- "Date,Value
01-31-1900,3"
library(xts)
# z <- read.zoo("myfile.csv", header = TRUE, sep = ",", format = "%m-%d-%Y")
z <- read.zoo(text = Lines, header = TRUE, sep = ",", format = "%m-%d-%Y")
x <- as.xts(z)
See ?read.zoo and Reading Data in zoo.
I'm trying to convert the following date/time string into a zoo object:
2004:071:15:23:41.87250
2004:103:15:24:15.35931
year:doy:hour:minute:second
The date/time string is stored in a dataframe without headers. What's the best way to go about this in R?
Cheers!
Edit based on answer by Gavin:
# read in time series from CSV file; each entry as described above
timeSeriesDates <- read.csv("timeseriesdates.csv", header = FALSE, sep = ",")
# convert to format that can be used as a zoo object
timeSeriesDatesZ <- as.POSIXct(timeSeriesDates$V1, format = "%Y:%j:%H:%M:%S")
Read the data in to R in the usual way. You will have something like the following:
dats <- data.frame(times = c("2004:071:15:23:41.87250", "2004:103:15:24:15.35931"))
dats
These can be converted to one of the POSIXt classes via:
dats <- transform(dats, as.POSIXct(times, format = "%Y:%j:%H:%M:%S"))
or
data$times <- as.POSIXct(dats$times, format = "%Y:%j:%H:%M:%S"))
which can then be used in a zoo object. See ?strftime for details on the placeholders used in the format argument; essentially %j is the day of the year placeholder.
To do the zoo bit, we would do, using some dummy data for the actual time series
ts <- rnorm(2) ## dummy data
require(zoo) ## load zoo
tsZoo <- zoo(ts, dats$times)
the last line gives:
> tsZoo
2004:071:15:23:41.87250 2004:103:15:24:15.35931
0.3503648 -0.2336064
One thing to note with fractional seconds is that i) the exact fraction you have may not be recordable using floating point arithmetic. Also, R may not show the full fractional seconds given the value of an option in R; digits.secs. See ?options for more on this particular option and how to change it.
Here's a commented example for the first string:
R> s <- "2004:103:15:24:15.35931"
R> # split on the ":" and convert the result to a numeric vector
R> n <- as.numeric(strsplit(s, ":")[[1]])
R> # Use the year, hour, minute, second to create a POSIXct object
R> # for the first of the year; then add the number of days (as seconds)
R> ISOdatetime(n[1], 1, 1, n[3], n[4], n[5])+n[2]*60*60*24
[1] "2004-04-13 16:24:15 CDT"