I have a text file containing:
Tue Feb 11 12:19:39 +0000 2014
Tue Feb 11 12:19:56 +0000 2014
Tue Feb 11 12:20:04 +0000 2014
and i read it into r
dataset <- read.csv("Time.txt")
and in order for R to recognise the timestamps in the file, i write:
time <- strptime(dataset[,1], format = "%a %b %d %H:%M:%S %z %Y")
and whenever i try to plot a histogram with:
hist(time, breaks = 100)
it produces an error together with a generated histogram
In breaks[-1L] + breaks[-nB] : NAs produced by integer overflow
What could be the issue that is prompting this error?
Since you asked what could be causing the error here it is:
The error is created when the hist.default function calculates the midpoints of the histogram. This vector mids <- 0.5 * (breaks[-1L] + breaks[-nB]) calculates the halfway point between each break. The issue arises because the breaks are generated as integers:
If the argument breaks is numeric and length == 1 then the hist.default function (which is called by hist.POSIXt) creates a vector of breaks based on the range of x and the number of breaks. This is done using the pretty command. For reasons I have not looked into too closely, if breaks is small enough that pretty(range(x),n=breaks, min.n = 1) returns only one of each value e.g.:
pretty(range(x), n = 35, min.n = 1)
#[1] 1392121179 1392121180 1392121181 1392121182 1392121183 1392121184
#[7] 1392121185 1392121186 1392121187 1392121188 1392121189 1392121190
#[13] 1392121191 1392121192 1392121193 1392121194 1392121195 1392121196
#[19] 1392121197 1392121198 1392121199 1392121200 1392121201 1392121202
#[25] 1392121203 1392121204
then the output is an integer type. If however, the number of breaks is larger and some of the outputs are duplicated:
pretty(range(x), n = 36, min.n = 1)
# [1] 1392121179 1392121180 1392121180 1392121181 1392121181 1392121182
# [7] 1392121182 1392121183 1392121183 1392121184 1392121184 1392121185
#[13] 1392121185 1392121186 1392121186 1392121187 1392121187 1392121188
#[19] 1392121188 1392121189 1392121189 1392121190 1392121190 1392121191
#[25] 1392121191 1392121192 1392121192 1392121193 1392121193 1392121194
#[31] 1392121194 1392121195 1392121195 1392121196 1392121196 1392121197
#[37] 1392121197 1392121198 1392121198 1392121199 1392121199 1392121200
#[43] 1392121200 1392121201 1392121201 1392121202 1392121202 1392121203
#[49] 1392121203 1392121204 1392121204
then the output is numeric.
Because R uses 32 bit integer types and POSIXt integers are large numbers, adding two POSIXt integers results in an overflow that R can't handle and returns NA. When pretty returns numeric, this is not a problem.
See also: What is integer overflow in R and how can it happen?
In practice, all this means is that, if you print out the hist structure returned, all of your mids values will be NA but I don't think it actually affects the plotting of the histogram. Thus it is only a warning.
EDIT:
pretty internally uses seq.int
In my environement, it does not generate any errors.
dataset <- read.csv("Time.txt", header = F)
time <- strptime(dataset[,1], format = "%a %b %d %H:%M:%S %z %Y")
hist(as.numeric(time), breaks = 100)
Perhaps if you just convert time into numeric as above, error will disappear. Then, it is straightforward to change the x-axis of the histogram.
EDIT : The ggplot2 should not face this issue and is much simpler and modern :
ggplot(dataset) + geom_histogram(aes(x = V1), stat = "count", bins = 100)
Where V1 is the default name of the unique column of dataset created by read.csv().
Related
I want to generate a sequence of dates with one quarter interval, with a starting date and ending date. I have below code :
> seq(as.Date('1980-12-31'), as.Date('1985-06-30'), by = 'quarter')
[1] "1980-12-31" "1981-03-31" "1981-07-01" "1981-10-01" "1981-12-31"
[6] "1982-03-31" "1982-07-01" "1982-10-01" "1982-12-31" "1983-03-31"
[11] "1983-07-01" "1983-10-01" "1983-12-31" "1984-03-31" "1984-07-01"
[16] "1984-10-01" "1984-12-31" "1985-03-31"
As you can see, this is not generating right sequence, as I dont understand how the date "1981-07-01" is coming here, I would expect "1981-06-30".
Is there any way to generate such sequence correctly with quarter interval?
Thanks for your time.
The from and to dates in the question are both end-of-quarter dates so we assume that that is the general case you are interested in.
1) Create a sequence of yearqtr objects yq and then convert them to Date class. frac=1 tells it s to use the end of the month. Alternately just use yq since that directly models years with quarters.
library(zoo)
from <- as.Date('1980-12-31')
to <- as.Date('1985-06-30')
yq <- seq(as.yearqtr(from), as.yearqtr(to), by = 1/4)
as.Date(yq, frac = 1)
giving;
[1] "1980-12-31" "1981-03-31" "1981-06-30" "1981-09-30" "1981-12-31"
[6] "1982-03-31" "1982-06-30" "1982-09-30" "1982-12-31" "1983-03-31"
[11] "1983-06-30" "1983-09-30" "1983-12-31" "1984-03-31" "1984-06-30"
[16] "1984-09-30" "1984-12-31" "1985-03-31" "1985-06-30"
2) or without any packages add 1 to from and to so that they are at the beginning of the next month, create the sequence (it has no trouble with first of month sequences) and then subtract 1 from the generated sequence giving the same result as above.
seq(from + 1, to + 1, by = "quarter") - 1
Using the clock package and R >= 4.1:
library(clock)
seq(year_quarter_day(1980, 4), year_quarter_day(1985, 2), by = 1) |>
set_day("last") |>
as_date()
# [1] "1980-12-31" "1981-03-31" "1981-06-30" "1981-09-30" "1981-12-31" "1982-03-31" "1982-06-30" "1982-09-30" "1982-12-31"
# [10] "1983-03-31" "1983-06-30" "1983-09-30" "1983-12-31" "1984-03-31" "1984-06-30" "1984-09-30" "1984-12-31" "1985-03-31"
# [19] "1985-06-30"
Note that this includes the final quarter. I don't know if that was your intent.
Different definition of "quarter". A quarter might well be (although it is not in R) 365/4 days. Look at output of :
as.Date('1980-12-31')+(365/4)*(0:12)
#[1] "1980-12-31" "1981-04-01" "1981-07-01" "1981-09-30" "1981-12-31" "1982-04-01" "1982-07-01" "1982-09-30"
#[9] "1982-12-31" "1983-04-01" "1983-07-01" "1983-09-30" "1983-12-31"
In order to avoid the days of the month from surprising you, you need to use a starting day of the month between 1 and 28, at least in non-leap years.
seq(as.Date('1981-01-01'), as.Date('1985-06-30'), by = 'quarter')
[1] "1981-01-01" "1981-04-01" "1981-07-01" "1981-10-01" "1982-01-01" "1982-04-01" "1982-07-01" "1982-10-01"
[9] "1983-01-01" "1983-04-01" "1983-07-01" "1983-10-01" "1984-01-01" "1984-04-01" "1984-07-01" "1984-10-01"
[17] "1985-01-01" "1985-04-01"
I want to create a vector in R, which contains month abbreviation and date:
Jan1, Jan2, Jan3, ..., Dec29, Dec30, Dec31.
How can I create such a vector?
I have tried different approaches. Using paste0(month.abb,1:31) gives me a vector containing Jan1,Feb2,Mar3.
I also created 12 different vectors for each month (df_jan <- paste0("Jan",1:31) and so on). Then I attempted to rbind the 12 vectors, but that also doesn't help.
Can you suggest any way to do this?
The solution is dependent on the year of the dates. A leap year would have Feb29 in it.
Here's a function which takes year as an argument and return dates in the required format.
monthday <- function(year) {
format(seq(as.Date(paste0(year, '-01-01')),
as.Date(paste0(year, '-12-31')), by = 'day'), '%b%d')
}
monthday(2021)
# [1] "Jan01" "Jan02" "Jan03" "Jan04" "Jan05" "Jan06" "Jan07" "Jan08" "Jan09"
#...
#[55] "Feb24" "Feb25" "Feb26" "Feb27" "Feb28" "Mar01" "Mar02" "Mar03" "Mar04"
#....
#[361] "Dec27" "Dec28" "Dec29" "Dec30" "Dec31"
monthday(2020)
# [1] "Jan01" "Jan02" "Jan03" "Jan04" "Jan05" "Jan06" "Jan07" "Jan08" "Jan09"
#....
#[55] "Feb24" "Feb25" "Feb26" "Feb27" "Feb28" "Feb29" "Mar01" "Mar02" "Mar03"
#....
#[361] "Dec26" "Dec27" "Dec28" "Dec29" "Dec30" "Dec31"
You can create a sequence of dates and convert it using format For more details on POSIX standard format use ?strptime
whole_year_dates <- seq.Date(as.Date("2021-01-01"), as.Date("2021-12-31"), by = "day")
whole_year_dates_abbr <- format(whole_year_dates, format= "%b%d")
I'm trying to get a vector of all the working days between to dates with the following code:
days_of_month = seq(as.Date("2017-01-01"), as.Date("2017-01-31"), by="days")
sundays = c(as.Date("2017-01-01"), as.Date("2017-01-08"), as.Date("2017-01-15"), as.Date("2017-01-22"), as.Date("2017-01-29"))
When I do:
working_days = setdiff(days_of_month, sundays)
The return value of setdiff is a vector of strange values:
[1] 17168 17169 17170 17171 17172 17173 17175 17176 17177 17178 17179 17180
[13] 17182 17183 17184 17185 17186 17187 17189 17190 17191 17192 17193 17194
[25] 17196 17197
What are those values? And how I get a vectors of the days that are in days_of_month but not in sundays?
Those are internal numeric value of R S3 class Date. You can see the numeric value by as.numeric(days_of_month). Or, you can convert the result to Date by as.Date(working_days, origin="1970-01-01").
I have a CSV file of 1000 daily prices
They are of this format:
1 1.6
2 2.5
3 0.2
4 ..
5 ..
6
7 ..
.
.
1700 1.3
The index is from 1:1700
But I need to specify a begin date and end date this way:
Start period is lets say, 25th january 2009
and the last 1700th value corresponds to 14th may 2013
So far Ive gotten this close to this problem:
> dseries <- ts(dseries[,1], start = ??time??, freq = 30)
How do I go about this? thanks
UPDATE:
managed to create a seperate object with dates as suggested in the answers and plotted it, but the y axis is weird, as shown in the screenshot
Something like this?
as.Date("25-01-2009",format="%d-%m-%Y") + (seq(1:1700)-1)
A better way, thanks to #AnandaMahto:
seq(as.Date("2009-01-25"), by="1 day", length.out=1700)
Plotting:
df <- data.frame(
myDate=seq(as.Date("2009-01-25"), by="1 day", length.out=1700),
myPrice=runif(1700)
)
plot(df)
R stores Date-classed objects as the integer offset from "1970-01-01" but the as.Date.numeric function needs an offset ('origin') which can be any staring date:
rDate <- as.Date.numeric(dseries[,1], origin="2009-01-24")
Testing:
> rDate <- as.Date.numeric(1:10, origin="2009-01-24")
> rDate
[1] "2009-01-25" "2009-01-26" "2009-01-27" "2009-01-28" "2009-01-29"
[6] "2009-01-30" "2009-01-31" "2009-02-01" "2009-02-02" "2009-02-03"
You didn't need to add the extension .numeric since R would automticallly seek out that function if you used the generic stem, as.Date, with an integer argument. I just put it in because as.Date.numeric has different arguments than as.Date.character.
I would like to generate a sequence of dates from 10,000 B.C.E. to the present. This is easy for 0 C.E. (or A.D.):
ADtoNow <- seq.Date(from = as.Date("0/1/1"), to = Sys.Date(), by = "day")
But I am stumped as to how to generate dates before 0 AD. Obviously, I could do years before present but it would be nice to be able to graph something as BCE and AD.
To expand on Ricardo's suggestion, here is some testing of how things work. Or don't work for that matter.
I will repeat Joshua's warning taken from ?as.Date for future searchers in big bold letters:
"Note: Years before 1CE (aka 1AD) will probably not be handled correctly."
as.integer(as.Date("0/1/1"))
[1] -719528
as.integer(seq(as.Date("0/1/1"),length=2,by="-10000 years"))
[1] -719528 -4371953
seq(as.Date(-4371953,origin="1970-01-01"),Sys.Date(),by="1000 years")
# nonsense
[1] "0000-01-01" "'000-01-01" "(000-01-01" ")000-01-01" "*000-01-01"
[6] "+000-01-01" ",000-01-01" "-000-01-01" ".000-01-01" "/000-01-01"
[11] "0000-01-01" "1000-01-01" "2000-01-01"
> as.integer(seq(as.Date(-4371953,origin="1970-01-01"),Sys.Date(),by="1000 years"))
# also possibly nonsense
[1] -4371953 -4006710 -3641468 -3276225 -2910983 -2545740 -2180498 -1815255
[9] -1450013 -1084770 -719528 -354285 10957
Though this does seem to work for graphing somewhat:
yrs1000 <- seq(as.Date(-4371953,origin="1970-01-01"),Sys.Date(),by="1000 years")
plot(yrs1000,rep(1,length(yrs1000)),axes=FALSE,ann=FALSE)
box()
axis(2)
axis(1,at=yrs1000,labels=c(paste(seq(10000,1000,by=-1000),"BC",sep=""),"0AD","1000AD","2000AD"))
title(xlab="Year",ylab="Value")
Quite some time has gone by since this question was asked. With that time came a new R package, gregorian which can handle BCE time values in the as_gregorian method.
Here's an example of piecewise constructing a list of dates that range from -10000 BCE to the current year.
library(lubridate)
library(gregorian)
# Container for the dates
dates <- c()
starting_year <- year(now())
# Add the CE dates to the list
for (year in starting_year:0){
date <- sprintf("%s-%s-%s", year, "1", "1")
dates <- c(dates, gregorian::as_gregorian(date))
}
starting_year <- "-10000"
# Add the BCE dates to the list
for (year in starting_year:0){
start_date <- gregorian::as_gregorian("-10000-1-1")
date <- sprintf("%s-%s-%s", year, "1", "1")
dates <- c(dates, gregorian::as_gregorian(date))
}
How you use the list is up to you, just know that the relevant properties of the date objects are year and bce. For example, you can loop over list of dates, parse the year, and determine if it's BCE or not.
> gregorian_date <- gregorian::as_gregorian("-10000-1-1")
> gregorian_date$bce
[1] TRUE
> gregorian_date$year
[1] 10001
Notes on 0AD
The gregorian package assumes that when you mean Year 0, you're really talking about year 1 (shown below). I personally think an exception should be thrown, but that's the mapping users needs to keep in mind.
> gregorian::as_gregorian("0-1-1")
[1] "Monday January 1, 1 CE"
This is also the case with BCE
> gregorian::as_gregorian("-0-1-1")
[1] "Saturday January 1, 1 BCE"
As #JoshuaUlrich commented, the short answer is no.
However, you can splice out the year into a separate column and then convert to integer. Would this work for you?
The package lubridate seems to handle "negative" years ok, although it does create a year 0, which from the above comments seems to be inaccurate. Try:
library(lubridate)
start <- -10000
stop <- 2013
myrange <- NULL
for (x in start:stop) {
myrange <- c(myrange,ymd(paste0(x,'-01-01')))
}