I'm trying to figure out the way of creating sequence of dates and time in this format: 2018-01-01 01:00 till 2018-03-30 01:00
for each Patient and fill the new empty value with random numbers.
My data look like :
Patients temperature
Patient1 37
Patient2 36
Patient3 35.4
I want to get the data looks like
Patients temperature Time
Patient1 37 2018-01-01 01:00
Patient2 36 2018-01-01 01:00
Patient3 35.4 2018-01-01 01:00
Patient1 NA 2018-01-01 02:00
Patient2 NA 2018-01-01 02:00
Patient3 NA 2018-01-01 02:00
Patient1 NA 2018-01-01 03:00
Patient2 NA 2018-01-01 03:00
Patient3 NA 2018-01-01 03:00
So the Time variable will be till 2018-03-30 01:00 and the temperature can be NA and then I generate random numbers but not repeating the same values of the temperature of each Patient.
I tried this commands but didn't work and I don't know how to assign the time to each Patient
Time <- seq (from=as.POSIXct("2018-1-1 01:00"), to=as.POSIXct("2018-3-30 01:00", tz="UTC"), by="hour")
And I tried too this command but I got error message:
dt = data.table(ID = Sensor7$StationID,Time = seq (from=as.POSIXct("2018-01-01 02:00"), to=as.POSIXct("2018-03-30 01:00",format = "%Y-%m-%d %H:%M",by="hour")))
But it gave me error message:
Error in seq.POSIXt(from = as.POSIXct("2018-01-01 00:00"), to = as.POSIXct("2018-03-30 23:00", :
exactly two of 'to', 'by' and 'length.out' / 'along.with' must be specified
Does anyone have any idea how to get the data in the format I'm looking for pleas?
You weren't too far off. Try this:
# I reproduce your data:
library(data.table)
data = data.table::fread(input =
"Patients,temperature
Patient1,37
Patient2,36
Patient3,35.4")
library(dplyr)
Time <- seq (from=as.POSIXct("2018-1-1 01:00"), to=as.POSIXct("2018-3-30 01:00", tz="UTC"), by="hour")
And this should do what you want:
data %>%
group_by(Patients) %>%
do({data.frame("temperature" = c(.data$temperature, rep(NA,length(Time) - nrow(.data))), Time)})
Here's one way:
dat = data.frame(Patients=paste0("Patients", 1:3), temperature=c(37,36,35.4))
Time = seq(as.POSIXct("2018-01-01 01:00"), as.POSIXct("2018-03-30 01:00"), by="hour")
new.data = data.frame(
Patient = rep(dat$Patients, each=length(Time)),
Time = rep(Time, length(dat$Patients))
)
I'm not sure how you want to generate the random values, but here's a generic method:
new.data$Random.Temperature = rnorm(nrow(new.data), 35, 1)
Related
I have a time series, that spans almost 20 years with a resolution of 15 min.
I want to extract only hourly values (00:00:00, 01:00:00, and so on...) and plot the resulting time series.
The df looks like this:
3 columns: date, time, and discharge
How would you approach this?
a reproducible example would be good for this kind of question. Here is my code, hope it helps you:
#creating dummy data
df <- data.frame(time = seq(as.POSIXct("2018-01-01 00:00:00"), as.POSIXct("2018-01-01 23:59:59"), by = "15 min"), variable = runif(96, 0, 1))
example output: (only 5 rows)
time variable
1 2018-01-01 00:00:00 0.331546992
2 2018-01-01 00:15:00 0.407269290
3 2018-01-01 00:30:00 0.635367577
4 2018-01-01 00:45:00 0.808612045
5 2018-01-01 01:00:00 0.258801201
df %>% filter(format(time, "%M:%S") == "00:00")
output:
1 2018-01-01 00:00:00 0.76198532
2 2018-01-01 01:00:00 0.01304103
3 2018-01-01 02:00:00 0.10729465
4 2018-01-01 03:00:00 0.74534184
5 2018-01-01 04:00:00 0.25942667
plot(df %>% filter(format(time, "%M:%S") == "00:00") %>% ggplot(aes(x = time, y = variable)) + geom_line())
I currently have a dataset with multiple different time formats(AM/PM, numeric, 24hr format) and I'm trying to turn them all into 24hr format. Is there a way to standardize mixed format columns?
Current sample data
time
12:30 PM
03:00 PM
0.961469907
0.913622685
0.911423611
09:10 AM
18:00
Desired output
new_time
12:30:00
15:00:00
23:04:31
21:55:37
21:52:27
09:10:00
18:00:00
I know how to do them all individually(an example below), but is there a way to do it all in one go because I have a large amount of data and can't go line by line?
#for numeric time
> library(chron)
> x <- c(0.961469907, 0.913622685, 0.911423611)
> times(x)
[1] 23:04:31 21:55:37 21:52:27
The decimal times are a pain but we can parse them first, feed them back as a character then use lubridate's parse_date_time to do them all at once
library(tidyverse)
library(chron)
# Create reproducible dataframe
df <-
tibble::tibble(
time = c(
"12:30 PM",
"03:00 PM",
0.961469907,
0.913622685,
0.911423611,
"09:10 AM",
"18:00")
)
# Parse times
df <-
df %>%
dplyr::mutate(
time_chron = chron::times(as.numeric(time)),
time_chron = if_else(
is.na(time_chron),
time,
as.character(time_chron)),
time_clean = lubridate::parse_date_time(
x = time_chron,
orders = c(
"%I:%M %p", # HH:MM AM/PM 12 hour format
"%H:%M:%S", # HH:MM:SS 24 hour format
"%H:%M")), # HH:MM 24 hour format
time_clean = hms::as_hms(time_clean)) %>%
select(-time_chron)
Which gives us
> df
# A tibble: 7 × 2
time time_clean
<chr> <time>
1 12:30 PM 12:30:00
2 03:00 PM 15:00:00
3 0.961469907 23:04:31
4 0.913622685 21:55:37
5 0.911423611 21:52:27
6 09:10 AM 09:10:00
7 18:00 18:00:00
I have a dataframe (vlinder) like the following, whereby the date and the timestamp (in UTC) are in separate columns:
date time.utc variable
1/04/2020 0:00:00 12
1/04/2020 0:05:00 54
In a first step, I combined the date and time variables into one column called dateandtime using the following code:
vlinder$dateandtime <- paste(vlinder$date, vlinder$time.utc)
which resulted in an extra column in dataframe vlinder:
date time.utc variable dateandtime
1/04/2020 0:00:00 12 1/04/2020 0:00:00
1/04/2020 0:05:00 54 1/04/2020 0:05:00
I want to convert the time of UTC into local time (which is CEST, so a time difference of 2 hours).
I tried using the following code, but I get something totally different.
vlinder$dateandtime <- as.POSIXct(vlinder$dateandtime, tz = "UTC")
vlinder$dateandtime.cest <- format(vlinder$dateandtime, tz = "Europe/Brussels", usetz = TRUE)
which results in:
date time.utc variable dateandtime dateandtime.cest
1/04/2020 0:00:00 12 0001-04-20 0001-04-20 00:17:30 LMT
1/04/2020 0:05:00 54 0001-04-20 0001-04-20 00:17:30 LMT
How can I solve this?
Many thanks!
Here's a lubridate and tidyverse answer. Some data tidying, data type changes, and then bam. Check lubridate::OlsonNames() for valid time zones (tz). (I'm not positive I chose the correct tz.)
library(tidyverse)
library(lubridate)
df <- read.table(header = TRUE,
text = "date time.utc variable
1/04/2020 00:00:00 12
1/04/2020 00:05:00 54")
df <- df %>%
mutate(date = mdy(date),
datetime_utc = as_datetime(paste(date, time.utc)),
datetime_cest = as_datetime(datetime_utc, tz = 'Europe/Brussels'))
date time.utc variable datetime_utc datetime_cest
1 2020-01-04 00:00:00 12 2020-01-04 00:00:00 2020-01-04 01:00:00
2 2020-01-04 00:05:00 54 2020-01-04 00:05:00 2020-01-04 01:05:00
The default format of as.POSIXct expects an date ordered by Year-Month-Day. Therefore the date 01/04/2020 is translated into the 20th April of Year 1.
You just need to add your timeformat to as.POSIXct:
vlinder$dateandtime <- as.POSIXct(vlinder$dateandtime, tz = "UTC", format = "%d/%m/%Y %H:%M:%S")
format(vlinder$dateandtime, tz = "Europe/Brussels", usetz = TRUE)
In a dataframe, I have wind speed data measured four times a day, at 00:00, 06:00, 12:00 and 18:00 o'clock. To combine these with other data, I need to fill the time in between towards a resolution of 15 minutes. I would like to fill the gaps by simple interpolation.
The following example produces two corresponding sample dataframes. df1 and df2 need to be merged. In the resulting merged dataframe, the gap values between the 6-hourly values (where var == NA?) need to be filled by a simply mean interpolation. My problem is how to merge both and do the concrete interpolation between the given values.
First dataframe
Creation:
# create a corresponding sample data frame
df1 <- data.frame(
date = seq.POSIXt(
from = ISOdatetime(2015,10,1,0,0,0, tz = "GMT"),
to = ISOdatetime(2015,10,14,23,59,0, tz= "GMT"),
by = "6 hour"
),
windspeed = abs(rnorm(14*4, 10, 4)) # abs() because windspeed shoud be positive
)
Resulting dataframe:
> # show the head of the dataframe
> head(df1)
date windspeed
1 2015-10-01 00:00:00 17.928217
2 2015-10-01 06:00:00 11.306025
3 2015-10-01 12:00:00 6.648131
4 2015-10-01 18:00:00 10.320146
5 2015-10-02 00:00:00 2.138559
6 2015-10-02 06:00:00 9.076344
Second dataframe
Creation:
# create a 2nd corresponding sample data frame
df2 <- data.frame(
date = seq.POSIXt(
from = ISOdatetime(2015,10,1,0,0,0, tz = "GMT"),
to = ISOdatetime(2015,10,14,23,59,0, tz= "GMT"),
by = "15 min"
),
var = abs(rnorm(14*24*4, 300, 100))
)
Resulting dataframe:
> # show the head of the 2nd dataframe
> head(df2)
date var
1 2015-10-01 00:00:00 198.2657
2 2015-10-01 00:15:00 472.9041
3 2015-10-01 00:30:00 605.8776
4 2015-10-01 00:45:00 429.0949
5 2015-10-01 01:00:00 400.2390
6 2015-10-01 01:15:00 317.1503
This is a solution
First merge them to get using all = TRUE to get all values
df3 <- merge(df1, df2, all = TRUE)
Then use approx for Interpolation
df3$windspeed <- approx(x = df1$date, y = df1$windspeed, xout = df2$date)$y
The only problem there is that the las ones will be NA unless your last value of windspeed is there, but everything in between will be there
I have read in and formatted my data set like shown under.
library(xts)
#Read data from file
x <- read.csv("data.dat", header=F)
x[is.na(x)] <- c(0) #If empty fill in zero
#Construct data frames
rawdata.h <- data.frame(x[,2],x[,3],x[,4],x[,5],x[,6],x[,7],x[,8]) #Hourly data
rawdata.15min <- data.frame(x[,10]) #15 min data
#Convert time index to proper format
index.h <- as.POSIXct(strptime(x[,1], "%d.%m.%Y %H:%M"))
index.15min <- as.POSIXct(strptime(x[,9], "%d.%m.%Y %H:%M"))
#Set column names
names(rawdata.h) <- c("spot","RKup", "RKdown","RKcon","anm", "pp.stat","prod.h")
names(rawdata.15min) <- c("prod.15min")
#Convert data frames to time series objects
data.htemp <- xts(rawdata.h,order.by=index.h)
data.15mintemp <- xts(rawdata.15min,order.by=index.15min)
#Select desired subset period
data.h <- data.htemp["2013"]
data.15min <- data.15mintemp["2013"]
I want to be able to combine hourly data from data.h$prod.h with data, with 15 min resolution, from data.15min$prod.15min corresponding to the same hour.
An example would be to take the average of the hourly value at time 2013-12-01 00:00-01:00 with the last 15 minute value in that same hour, i.e. the 15 minute value from time 2013-12-01 00:45-01:00. I'm looking for a flexible way to do this with an arbitrary hour.
Any suggestions?
Edit: Just to clarify further: I want to do something like this:
N <- NROW(data.h$prod.h)
for (i in 1:N){
prod.average[i] <- mean(data.h$prod.h[i] + #INSERT CODE THAT FINDS LAST 15 MIN IN HOUR i )
}
I found a solution to my problem by converting the 15 minute data into hourly data using the very useful .index* function from the xts package like shown under.
prod.new <- data.15min$prod.15min[.indexmin(data.15min$prod.15min) %in% c(45:59)]
This creates a new time series with only the values occuring in the 45-59 minute interval each hour.
For those curious my data looked like this:
Original hourly series:
> data.h$prod.h[1:4]
2013-01-01 00:00:00 19.744
2013-01-01 01:00:00 27.866
2013-01-01 02:00:00 26.227
2013-01-01 03:00:00 16.013
Original 15 minute series:
> data.15min$prod.15min[1:4]
2013-09-30 00:00:00 16.4251
2013-09-30 00:15:00 18.4495
2013-09-30 00:30:00 7.2125
2013-09-30 00:45:00 12.1913
2013-09-30 01:00:00 12.4606
2013-09-30 01:15:00 12.7299
2013-09-30 01:30:00 12.9992
2013-09-30 01:45:00 26.7522
New series with only the last 15 minutes in each hour:
> prod.new[1:4]
2013-09-30 00:45:00 12.1913
2013-09-30 01:45:00 26.7522
2013-09-30 02:45:00 5.0332
2013-09-30 03:45:00 2.6974
Short answer
df %>%
group_by(t = cut(time, "30 min")) %>%
summarise(v = mean(value))
Long answer
Since, you want to compress the 15 minutes time series to a smaller resolution (30 minutes), you should use dplyr package or any other package that computes the "group by" concept.
For instance:
s = seq(as.POSIXct("2017-01-01"), as.POSIXct("2017-01-02"), "15 min")
df = data.frame(time = s, value=1:97)
df is a time series with 97 rows and two columns.
head(df)
time value
1 2017-01-01 00:00:00 1
2 2017-01-01 00:15:00 2
3 2017-01-01 00:30:00 3
4 2017-01-01 00:45:00 4
5 2017-01-01 01:00:00 5
6 2017-01-01 01:15:00 6
The cut.POSIXt, group_by and summarise functions do the work:
df %>%
group_by(t = cut(time, "30 min")) %>%
summarise(v = mean(value))
t v
1 2017-01-01 00:00:00 1.5
2 2017-01-01 00:30:00 3.5
3 2017-01-01 01:00:00 5.5
4 2017-01-01 01:30:00 7.5
5 2017-01-01 02:00:00 9.5
6 2017-01-01 02:30:00 11.5
A more robust way is to convert 15 minutes values into hourly values by taking average. Then do whatever operation you want to.
### 15 Minutes Data
min15 <- structure(list(V1 = structure(1:8, .Label = c("2013-01-01 00:00:00",
"2013-01-01 00:15:00", "2013-01-01 00:30:00", "2013-01-01 00:45:00",
"2013-01-01 01:00:00", "2013-01-01 01:15:00", "2013-01-01 01:30:00",
"2013-01-01 01:45:00"), class = "factor"), V2 = c(16.4251, 18.4495,
7.2125, 12.1913, 12.4606, 12.7299, 12.9992, 26.7522)), .Names = c("V1",
"V2"), class = "data.frame", row.names = c(NA, -8L))
min15
### Hourly Data
hourly <- structure(list(V1 = structure(1:4, .Label = c("2013-01-01 00:00:00",
"2013-01-01 01:00:00", "2013-01-01 02:00:00", "2013-01-01 03:00:00"
), class = "factor"), V2 = c(19.744, 27.866, 26.227, 16.013)), .Names = c("V1",
"V2"), class = "data.frame", row.names = c(NA, -4L))
hourly
### Convert 15min data into hourly data by taking average of 4 values
min15$V1 <- as.POSIXct(min15$V1,origin="1970-01-01 0:0:0")
min15 <- aggregate(. ~ cut(min15$V1,"60 min"),min15[setdiff(names(min15), "V1")],mean)
min15
names(min15) <- c("time","min15")
names(hourly) <- c("time","hourly")
### merge the corresponding values
combined <- merge(hourly,min15)
### average of hourly and 15min values
rowMeans(combined[,2:3])