I have got CSV files which has the Date in the following format:
25-Aug-2004
I want to read it as an "xts" object so as to use the function "periodReturn" in quantmod package.
Can I use the following file for the function?
Symbol Series Date Prev.Close Open.Price High.Price Low.Price
1 XXX EQ 25-Aug-2004 850.00 1198.70 1198.70 979.00
2 XXX EQ 26-Aug-2004 987.95 992.00 997.00 975.30
Guide me with the same.
Unfortunately I can't speak for the ts part, but this is how you can convert your dates to a proper format that can be read by other functions as dates (or time).
You can import your data into a data.frame as usual (see here if you've missed it). Then, you can convert your Date column into a POSIXlt (POSIXt) class using strptime function.
nibha <- "25-Aug-2004" # this should be your imported column
lct <- Sys.getlocale("LC_TIME"); Sys.setlocale("LC_TIME", "C") #temporarily change locale to C if you happen go get NAs
strptime(nibha, format = "%d-%b-%Y")
Sys.setlocale("LC_TIME", lct) #revert back to your locale
Try this. We get rid of the nuisance columns and specify the format of the time index, then convert to xts and apply the dailyReturn function:
Lines <- "Symbol Series Date Prev.Close Open.Price High.Price Low.Price
1 XXX EQ 25-Aug-2004 850.00 1198.70 1198.70 979.00
2 XXX EQ 26-Aug-2004 987.95 992.00 997.00 975.30"
library(quantmod) # this also pulls in xts & zoo
z <- read.zoo(textConnection(Lines), format = "%d-%b-%Y",
colClasses = rep(c(NA, "NULL", NA), c(1, 2, 5)))
x <- as.xts(z)
dailyReturn(x)
Of course, textConnection(Lines) is just to keep the example self contained and in reality would be replaced with something like "myfile.dat".
Related
I would like to change all the mixed date format into one format for example d-m-y
here is the data frame
x <- data.frame("Name" = c("A","B","C","D","E"), "Birthdate" = c("36085.0","2001-sep-12","Feb-18-2005","05/27/84", "2020-6-25"))
I hv tried using this code down here, but it gives NAs
newdateformat <- as.Date(x$Birthdate,
format = "%m%d%y", origin = "2020-6-25")
newdateformat
Then I tried using parse, but it also gives NAs which means it failed to parse
require(lubridate)
parse_date_time(my_data$Birthdate, orders = c("ymd", "mdy"))
[1] NA NA "2001-09-12 UTC" NA
[5] "2005-02-18 UTC"
and I also could find what is the format for the first date in the data frame which is "36085.0"
i did found this code but still couldn't understand what the number means and what is the "origin" means
dates <- c(30829, 38540)
betterDates <- as.Date(dates,
origin = "1899-12-30")
p/s : I'm quite new to R, so i appreciate if you can use an easier explanation thank youuuuu
You should parse each format separately. For each format, select the relevant rows with a regular expression and transform only those rows, then move on the the next format. I'll give the answer with data.table instead of data.frame because I've forgotten how to use data.frame.
library(lubridate)
library(data.table)
x = data.table("Name" = c("A","B","C","D","E"),
"Birthdate" = c("36085.0","2001-sep-12","Feb-18-2005","05/27/84", "2020-6-25"))
# or use setDT(x) to convert an existing data.frame to a data.table
# handle dates like "2001-sep-12" and "2020-6-25"
# this regex matches strings beginning with four numbers and then a dash
x[grepl('^[0-9]{4}-',Birthdate),Birthdate1:=ymd(Birthdate)]
# handle dates like "36085.0": days since 1904 (or 1900)
# see https://learn.microsoft.com/en-us/office/troubleshoot/excel/1900-and-1904-date-system
# this regex matches strings that only have numeric characters and .
x[grepl('^[0-9\\.]+$',Birthdate),Birthdate1:=as.Date(as.numeric(Birthdate),origin='1904-01-01')]
# assume the rest are like "Feb-18-2005" and "05/27/84" and handle those
x[is.na(Birthdate1),Birthdate1:=mdy(Birthdate)]
# result
> x
Name Birthdate Birthdate1
1: A 36085.0 2002-10-18
2: B 2001-sep-12 2001-09-12
3: C Feb-18-2005 2005-02-18
4: D 05/27/84 1984-05-27
5: E 2020-6-25 2020-06-25
I have a dataframe of strings representing times, such as:
times <- structure(list(exp1 = c("17:19:04 \r", "17:28:53 \r", "17:38:44 \r"),
exp2 = c("17:22:04 \r", "17:31:53 \r", "17:41:45 \r")),
row.names = c(NA, 3L), class = "data.frame")
If I run strptime() on a single element of my dataframe times, it converts it into a nice POSIXt object:
strptime(times[1,1], '%H:%M:%S')
[1] "2020-02-19 17:19:04 GMT"
Great, so now I'd like to convert my whole dataframe times into this format.
I cannot seem to find the solution to do this smoothly.
A few of the things I have tried so far:
strptime(times, '%H:%M:%S') # generates NA
strftime(times, '%H:%M:%S') # Error: do not know how to convert 'x' to class “POSIXlt”
apply(times, 2, function(x) strftime(x, '%H:%M:%S')) # Error: character string is not in a standard unambiguous format
The closest I got to what I want is:
apply(times, 2, function(x) strptime(x, '%H:%M:%S'))
It generates a messy list. I can probably find a way to use it, but there must be a more staightforward way?
You could use lapply.
times[] <- lapply(times, strptime, '%H:%M:%S')
# exp1 exp2
# 1 2020-02-19 17:19:04 2020-02-19 17:22:04
# 2 2020-02-19 17:28:53 2020-02-19 17:31:53
# 3 2020-02-19 17:38:44 2020-02-19 17:41:45
Note: apply also works.
times[] <- apply(times, 2, function(x) strptime(x, '%H:%M:%S'))
The trick is to replace the columns (in contrast to overwriting the data frame with a list) with [] <-, which can be seen as abbreviated for times[1:2] <- lapply(times[1:2], ·) in this case.
This question already has answers here:
Specify custom Date format for colClasses argument in read.table/read.csv
(4 answers)
How to avoid: read.table truncates numeric values beginning with 0
(3 answers)
Closed 3 years ago.
After I imported the text file into R, R omitted "0" in the time column.
For example:
Before import time | After import time
077250 | 77250
000002 | 2
Thus, unable to convert to the correct time format. (from 77250 to 07:25:50)
How can i convert the integer time to the correct time format?
I have tried:
chron (time, "%H:%M:%S")
strptime(time, "%H:%M:%S")
time <- as.hms(time)
You can use str_pad from the stringr package to restore the zeroes:
library(stringr)
time_old <- "2"
time_new <- str_pad(time_old, width = 6, side = "left", pad = 0)
Then, you should be able to use the chron function:
chron::chron(times = time_new, format = list(times = "hms"),
out.format = "h:m:s")
[1] 00:00:02
We can use sprintf and strptime/as.POSIXct
If you have read them as numeric use %d in sprintf or use %s if they are characters.
x <- c(072550, 2)
format(strptime(sprintf("%06d", x), "%H%M%S"), "%T")
#[1] "07:25:50" "00:00:02"
x <- c("072550", "2")
format(strptime(sprintf("%06s", x), "%H%M%S"), "%T")
#[1] "07:25:50" "00:00:02"
This possibly duplicate question shows how to read the data in the format you want directly, by specifying your own formatting function through colClasses :
setAs("character","myDate", function(from) as.Date(from, format="%Y%m%d") )
setAs("character","myTime", function(from) chron(times = from, format = "hms", out.format = "h:m:s"))
tmp <- c("1\t20080815\t072550", "2\t20100523\t000002")
con <- textConnection(tmp)
tmp2 <- read.delim(con, colClasses=c('numeric','myDate','myTime'), header=FALSE)
tmp2 contains :
V1 V2 V3
1 1 2008-08-15 07:25:50
2 2 2010-05-23 00:00:02
read.delim is a shortcut for read.table that sets a few defaults and passes any extra parameters like colClasses directly to read.table
Slight problem where my as.Date function gives a different result when I put it in a for loop. I'm looking in a folder with subfolders (per date) that contain images. I build date_list to organize all the dates (for plotting options in a later stage). The Julian Day starts from the first of January of the year, so because I have 4 years of date, the year must be flexible.
# Set up list with 4 columns and counter Q. jan is used to set all dates to the first of january
date_list <- outer(1:52, 1:4)
q = 1
jan <- "-01-01"
for (scene in folders){
year <- as.numeric(substr(scene, start=10, stop=13))
day <- as.numeric(substr(scene, start=14, stop=16))
datum <- paste(year, day, sep='_')
date_list[q, 1] <- datum
date_list[q, 2] <- year
date_list[q, 3] <- day
date_list[q, 4] <- as.Date(day, origin = as.Date(paste(year,jan, sep="")))
q = q+1
}
Output final row:
[52,] "2016_267" "2016" "267" "17068"
What am i missing in date_list[q, 4] that doesn't transfer my integer to a date?
running the following code does work, but due to the large amount of scenes and folders I like to automate this:
as.Date(day, origin = as.Date(paste(year,jan, sep="")))
Thank you for your time!
Well, I assume this would answer your first question:
date_list[q, 4] <- as.character(as.Date(datum,format="%Y_%j"))
as.Date accept a format argument, (the %Y and %j are documented in strptime), the %jis the julian day, this is a little easier to read than using origin and multiple paste calls.
Your problem is actually linked to what a Date object is:
> dput(as.Date("2016-01-10"))
structure(16810, class = "Date")
When entered into a matrix (your date_list) it is coerced to character w
without special treatment before like this:
> d<-as.Date("2016-01-10")
> class(d)<-"character"
> d
[1] "16810"
Hence you get only the number of days since 1970-01-01. When you ask for the date as character representation with as.character, it gives the correct value because the Date class as a as.character method which first compute the date in human format before returning a character value.
Now if I understood well your problem I would go this way:
First create a function to work on one string:
name_to_list <- function(name) {
dpart <- substr(name, start=10, stop=16)
date <- as.POSIXlt(dpart, format="%Y%j")
c("datum"=paste(date$year+1900,date$yday,sep="_"), "year"=date$year+1900, "julian_day"=date$yday, "date"=as.character(date) )
}
this function just get your substring, and then convert it to POSIXlt class, which give us julian day, year and date in one pass. as the year is stored as integer since 1900 (could be negative), we have to add 1900 when storing the year in the fields.
Then if your folders variable is a vector of string:
lapply(folders,name_to_list)
wich for folders=c("LC81730382016267LGN00","LC81730382016287LGN00","LC81730382016167LGN00") gives:
[[1]]
datum year julian_day date
"2016_266" "2016" "266" "2016-09-23"
[[2]]
datum year julian_day date
"2016_286" "2016" "286" "2016-10-13"
[[3]]
datum year julian_day date
"2016_166" "2016" "166" "2016-06-15"
Do you mean to output your day as 3 numbers? Should it not be 2 numbers?
day <- as.numeric(substr(scene, start=15, stop=16))
or
day <- as.numeric(substr(scene, start=14, stop=15))
That could at least be part of the issue. Providing an example of what typical values of "scene" are would be helpful here.
I have a character string of the date in Year-week format as such:
weeks.strings <- c("2002-26", "2002-27", "2002-28", "2002-29", "2002-30", "2002-31")
However, converting this character to Date class results in a loss of week identifier:
> as.Date(weeks.strings, format="%Y-%U")
[1] "2002-08-28" "2002-08-28" "2002-08-28" "2002-08-28" "2002-08-28"
[6] "2002-08-28"
As shown above, the format is converted into year- concatenated with today's date, so any information about the original week is lost (ex - when using the format function or strptime to try and coerce back into the original format.
One solution I found in a help group is to specify the day of the week:
as.Date(weeks.strings, format="%Y-%u %U")
[1] "2002-02-12" "2002-02-19" "2002-02-26" "2002-03-05" "2002-01-02"
[6] "2002-01-09"
But it looks like this results in incorrect week numbering (doesn't match the original string).
Any guidance would be appreciated.
You just need to add a weekday to your weeks.strings in order to make the dates unambiguous (adapted from Jim Holtman's answer on R-help).
as.Date(paste(weeks.strings,1),"%Y-%U %u")
As pointed out in the comments, the Date class is not appropriate if the dates span a long horizon because--at some point--the chosen weekday will not exist in the first/last week of the year. In that case you could use a numeric vector where the whole portion is the year and the decimal portion is the fraction of weeks/year. For example:
wkstr <- sprintf("%d-%02d", rep(2000:2012,each=53), 0:52)
yrwk <- lapply(strsplit(wkstr, "-"), as.numeric)
yrwk <- sapply(yrwk, function(x) x[1]+x[2]/53)
Obviously, there's no unique solution, since each week could be represented by any of up to 7 different dates. That said, here's one idea:
weeks.strings <- c("2002-26", "2002-27", "2002-28", "2002-29",
"2002-30", "2002-31")
x <- as.Date("2002-1-1", format="%Y-%m-%d") + (0:52*7)
x[match(weeks.strings, format(x, "%Y-%U"))]
# [1] "2002-07-02" "2002-07-09" "2002-07-16" "2002-07-23"
# [5] "2002-07-30" "2002-08-06"