I have a csv file that consists of one column. The column presents the date of posting on a website. I want to plot a histogram to see how the number of posts varies over the years. The file contains the years (2012 to 2016) and consists of 11,000 rows.
sample of the file:
2 30/1/12 21:07
3 2/2/12 15:53
4 3/4/12 0:49
5 14/11/12 3:49
6 11/8/13 16:00
7 31/7/14 8:08
8 31/7/14 10:48
9 6/8/14 9:24
10 16/12/14 3:34
The data types is dataframe
class(postsData)
[1] "data.frame"
I tried converting the data to text using strptime function as below:
formatDate <- strptime(as.character(postsData$Date),format="“%d/%m/%y")
then plot the histogram
hist(formatDate,breaks=10,xlab="year")
Any tip or suggestion would be useful. Thank you,
use lubridate::dmy_hm()
strptime() is overly complicated in my opinion compared to { lubridate }.
library(lubridate)
d <- c("30/1/12 21:07",
"2/2/12 15:53",
"3/4/12 0:49",
"14/11/12 3:49",
"11/8/13 16:00",
"31/7/14 8:08",
"31/7/14 10:48",
"6/8/14 9:24",
"16/12/14 3:34")
d2 <- dmy_hm(d)
d2
Returns:
[1] "2012-01-30 21:07:00 UTC"
[2] "2012-02-02 15:53:00 UTC"
[3] "2012-04-03 00:49:00 UTC"
[4] "2012-11-14 03:49:00 UTC"
[5] "2013-08-11 16:00:00 UTC"
[6] "2014-07-31 08:08:00 UTC"
[7] "2014-07-31 10:48:00 UTC"
[8] "2014-08-06 09:24:00 UTC"
[9] "2014-12-16 03:34:00 UTC"
As you can see, lubridate functions return POSIXct objects.
class(d2)
[1] "POSIXct" "POSIXt"
Next, you can use lubridate::year() to get the year of each POSIXct object returned by dmy_hm(), and plot that histogram.
hist(year(d2))
Here's one approach. I think your date conversion is fine but you need to count the number of dates that occur in each year then plot that count as a histogram.
library(tidyverse)
# generate some data
date.seq <- tibble(xdate = seq(from = lubridate::ymd_hms('2000-01-01 00:00:00'), to=lubridate::ymd_hms('2016-12-31 24:59:59'), length.out = 100))
date.seq %>%
mutate(xyear = lubridate::year(xdate)) %>% # add a column of years
group_by(xyear) %>%
summarise(date_count = length(xdate)) %>% # Count the number of dates that occur in each year
ggplot(aes(x = xyear, y = date_count)) +
geom_col(colour = 'black', fill = 'blue') # plot as a column graph
There's no problem with strptime()*, however, the format option is intended to specify how the is formatted.
df1$date <- strptime(df1$date, format="%d/%m/%y %H:%M")
# [1] "2012-01-30 21:07:00 CET" "2012-02-02 15:53:00 CET"
# [3] "2012-04-03 00:49:00 CEST" "2012-11-14 03:49:00 CET"
# [5] "2013-08-11 16:00:00 CEST" "2014-07-31 08:08:00 CEST"
# [7] "2014-07-31 10:48:00 CEST" "2014-08-06 09:24:00 CEST"
# [9] "2014-12-16 03:34:00 CET"
What you probably want then is to use the format() function
formatDate <- format(df1$date, format="%F")
(or in this case simpler with formatDate <- as.Date(df1$date))
and then
hist(formatDate, breaks=10, xlab="year")
* credits to #MikkoMarttila
Data
df1 <- structure(list(id = 2:10, date = c("30/1/12 21:07", "2/2/12 15:53",
"3/4/12 0:49", "14/11/12 3:49", "11/8/13 16:00", "31/7/14 8:08",
"31/7/14 10:48", "6/8/14 9:24", "16/12/14 3:34")), class = "data.frame", row.names = c(NA,
-9L))
Related
This question already has answers here:
Convert date-time string to class Date
(4 answers)
Closed 3 years ago.
I have date&time stamp as a character variable
"2018-12-13 11:00:01 EST" "2018-10-23 22:00:01 EDT" "2018-11-03 14:15:00 EDT" "2018-10-04 19:30:00 EDT" "2018-11-10 17:15:31 EST" "2018-10-05 13:30:00 EDT"
How can I strip the time from this character vector?
PS: Can someone please help. I have tried using strptime but I am getting NA values as a result
It's a bit unclear whether you want the date or time but if you want the date then as.Date ignores any junk after the date so:
x <- c("2018-12-13 11:00:01 EST", "2018-10-23 22:00:01 EDT")
as.Date(x)
## [1] "2018-12-13" "2018-10-23"
would be sufficient to get a Date vector from the input vector x. No packages are used.
If you want the time then:
read.table(text = x, as.is = TRUE)[[2]]
## [1] "11:00:01" "22:00:01"
If you want a data frame with each part in a separate column then:
read.table(text = x, as.is = TRUE, col.names = c("date", "time", "tz"))
## date time tz
## 1 2018-12-13 11:00:01 EST
## 2 2018-10-23 22:00:01 EDT
I think the OP wants to extract the time from date-time variable (going by the title of the question).
x <- "2018-12-13 11:00:01 EST"
as.character(strptime(x, "%Y-%m-%d %H:%M:%S"), "%H:%M:%S")
[1] "11:00:01"
Another option:
library(lubridate)
format(ymd_hms(x, tz = "EST"), "%H:%M:%S")
[1] "11:00:01"
The package lubridate makes everything like this easy:
library(lubridate)
x <- "2018-12-13 11:00:01 EST"
as_date(ymd_hms(x))
You can use the as.Date function and specify the format
> as.Date("2018-12-13 11:00:01 EST", format="%Y-%m-%d")
[1] "2018-12-13"
If all values are in a vector:
x = c("2018-12-13 11:00:01 EST", "2018-10-23 22:00:01 EDT",
"2018-11-03 14:15:00 EDT", "2018-10-04 19:30:00 EDT",
"2018-11-10 17:15:31 EST", "2018-10-05 13:30:00 EDT")
> as.Date(x, format="%Y-%m-%d")
[1] "2018-12-13" "2018-10-23" "2018-11-03" "2018-10-04" "2018-11-10"
[6] "2018-10-05"
I have looked at this answer which lists all dates between time points. Ideally I would like to state start and end dates, and the number of elements I'd want in the vector, and get back random dates including replicates.
start_date <- as.Date('2015-01-01')
end_date <- as.Date('2017-01-01')
set.seed(1984)
as.Date(sample( as.numeric(start_date): as.numeric(end_date), 10,
replace = T),
origin = '1970-01-01')
[1] "2016-04-27" "2015-11-16" "2015-10-01" "2015-08-31" "2016-06-23"
[6] "2016-09-23" "2015-01-24" "2015-11-24" "2016-08-30" "2015-06-07"
Using the sample and seq functions as in Generating Random Dates seems to me the more straight forward approach:
set.seed(1984)
sample(seq(as.Date('2015-01-01'), as.Date('2017-01-01'), by = "day"), 10)
Output:
[1] "2016-04-27" "2015-11-16" "2015-09-30" "2015-08-30" "2016-06-20"
[6] "2016-09-19" "2015-01-24" "2015-11-21" "2016-08-23" "2015-06-05"
If we are working with the class POSIXct and we'd like randomized hours, minutes, and seconds, not just dates:
set.seed(1984)
sample(seq(as.POSIXct('2015-01-01'), as.POSIXct('2017-01-01'), by = "sec"), 10)
Output:
[1] "2016-04-26 15:04:13 CEST" "2015-11-16 10:17:23 CET"
[3] "2015-09-30 22:50:41 CEST" "2015-08-30 23:17:49 CEST"
[5] "2016-06-23 04:49:01 CEST" "2016-09-22 14:37:58 CEST"
[7] "2015-01-24 17:04:13 CET" "2015-11-24 07:13:42 CET"
[9] "2016-08-29 16:13:13 CEST" "2015-06-06 21:29:18 CEST"
I have a regular 5 minute interval datetime data sets (about 50). POSIXt/ lubridate functions convert my datetime very nicely to a 24 hour format as required. But I would like to add another column with my day's definition to be from 6 am to 6 am (which is currently midnight to midnight). I am trying to do this to capture after 12AM activity as a part of current date rather than the next one.
I am currently trying to create a group every 288th row (there are 288 5minute intervals in a day). But it creates a problem because my datasets don't necessarily start at a unique time.
I do not want to create offsets because that tampers with the values corresponding to the time.
Any efficient ways around this problem? Thank you.
You can efficiently do it by first generating a sequence of date/times, then using cut to find the bin in which each value falls:
set.seed(2)
dat <- Sys.time() + sort(runif(10, min=0, max=5*24*60*60))
dat
# [1] "2017-07-29 15:43:10 PDT" "2017-07-29 20:23:12 PDT" "2017-07-29 22:24:22 PDT" "2017-07-31 08:22:57 PDT"
# [5] "2017-07-31 18:13:06 PDT" "2017-07-31 21:01:10 PDT" "2017-08-01 12:30:19 PDT" "2017-08-02 04:14:03 PDT"
# [9] "2017-08-02 17:26:14 PDT" "2017-08-02 17:28:52 PDT"
sixs <- seq(as.POSIXct("2017-07-29 06:00:00", tz = "UTC"), as.POSIXct("2017-08-03 06:00:00", tz = "UTC"), by = "day")
sixs
# [1] "2017-07-29 06:00:00 UTC" "2017-07-30 06:00:00 UTC" "2017-07-31 06:00:00 UTC" "2017-08-01 06:00:00 UTC"
# [5] "2017-08-02 06:00:00 UTC" "2017-08-03 06:00:00 UTC"
cut(dat, sixs, label = FALSE)
# [1] 1 1 1 3 3 3 4 5 5 5
According to the help page (?seq.POSIXt), you might choose by="DSTday" instead.
Checkout this question and the corresponding answer: How to manipulate the time part of a date column?
It illustrates a more robust solution as it is independent of your data structure (e.g. repeatition).
Following #meenaparam's solution:
Convert all date columns to dmy_hms format from lubridate package. Please explore other options like dmy_hm or ymd_hms etc, as per your specific need.
mutate(DATE = dmy_hms(DATE))
Now create a column to identify the data points that need to be modified in different ways. Like your data points with 00:00:00 to 05:59:59 (hms) needs to be part of the previous date.
DAY_PAST = case_when(hour(DATE) < 6 ~ "yup", TRUE ~ "nope"))
Now convert the day value of these "yup" dates to day(DATE)-1
NEW_DATE = case_when(DAY_PAST == "yup"
~ make_datetime(year(DATE-86400), month(DATE-86400), day = day(DATE-86400), hour = hour(DATE)),
TRUE ~ DATE)
.
I am trying to do some simple operation in R, after loading a table i encountered a date column which has many formats combined.
**Date**
1/28/14 6:43 PM
1/29/14 4:10 PM
1/30/14 12:09 PM
1/30/14 12:12 PM
02-03-14 19:49
02-03-14 20:03
02-05-14 14:33
I need to convert this to format like 28-01-2014 18:43 i.e. %d-%m-%y %h:%m
I tried this
tablename$Date <- as.Date(as.character(tablename$Date), "%d-%m-%y %h:%m")
but doing this its filling NA in the entire column. Please help me to get this right!
The lubridate package makes quick work of this:
library(lubridate)
d <- parse_date_time(dates, names(guess_formats(dates, c("mdy HM", "mdy IMp"))))
d
## [1] "2014-01-28 18:43:00 UTC" "2014-01-29 16:10:00 UTC"
## [3] "2014-01-30 12:09:00 UTC" "2014-01-30 12:12:00 UTC"
## [5] "2014-02-03 19:49:00 UTC" "2014-02-03 20:03:00 UTC"
## [7] "2014-02-05 14:33:00 UTC"
# put in desired format
format(d, "%m-%d-%Y %H:%M:%S")
## [1] "01-28-2014 18:43:00" "01-29-2014 16:10:00" "01-30-2014 12:09:00"
## [4] "01-30-2014 12:12:00" "02-03-2014 19:49:00" "02-03-2014 20:03:00"
## [7] "02-05-2014 14:33:00"
You'll need to adjust the vector in guess_formats if you come across other format variations.
I have a separately created time series object with daily frequency:
my.timeseries= ts(data= 1:10, start= c(2014,1,1), frequency = 365.25)
How can I get back the dates as POSIXct vector ("2014-01-01 UTC" ...) from this time series object?
Here's one potential method. I'm not really sure if it should be done this way, but it seems to work.
With your existing time series, try
p <- paste(attr(my.timeseries, "tsp")[1], my.timeseries)
as.POSIXct(as.Date(p, "%Y %j"))
# [1] "2014-01-01 UTC" "2014-01-02 UTC" "2014-01-03 UTC"
# [4] "2014-01-04 UTC" "2014-01-05 UTC" "2014-01-06 UTC"
# [7] "2014-01-07 UTC" "2014-01-08 UTC" "2014-01-09 UTC"
# [10] "2014-01-10 UTC"
As noted by G. Grothendieck in the comments, here is a more general solution
p <- paste(start(my.timeseries), seq_along(my.timeseries))
as.Date(p, "%Y %j")
# [1] "2014-01-01" "2014-01-02" "2014-01-03" "2014-01-04"
# [5] "2014-01-05" "2014-01-06" "2014-01-07" "2014-01-08"
# [9] "2014-01-09" "2014-01-10"
as.Date might be better to avoid any time-zone issues.
I strongly advise you to use xts object instead of ts.
Here is a code replicating what you want :
library(xts)
my.index = seq(from = as.Date("2014-01-01"), by = "day", length.out = 10)
my.timeseries = xts(x = 1:10, order.by = my.index)
index(my.timeseries)
Let us know if that helps :)
Romain