I have weather data from 13 years, with 154 days each year (only interested in summer days), from which I would like to get the number of dry days occurred each year. This means I have a great data (df) set that I want to shrink (df_Ch). I've already done with the total rain per year:
df_Ch = df.groupby(by = ["distrito", "year"], as_index=False)["Chuvia"].sum()
I'm now trying with total dry days per year:
df_Ds = df.groupby(by = ["distrito", "year"], as_index=False)[when "Chuvia" == 0].count()
Of course, it didn't work
I spent some hours without success trying to get something... Any suggestion?
Apply the condition before using groupby, so you will already have your "filtered" df before grouping it.
df_Ds = df[df["Chuvia" == 0]].groupby(by = ["distrito", "year"], as_index=False)["Chuvia"].count()
Related
I have a set of many different kind of observations that contain the Date of observations on special days (sam1, sam2, sam3). My aim is now to perform a wilcox.test() to find out if there is a significant difference between the observations on these special days and not special days. So I need to find a method to find suitable days to use as a control group. I want to try out at least 3 different control groups. The special days are very different, some represent a whole season, some only rainy days, some only cold days, some stormy weather. So they can be spread over the whole 2 years, some might only occur in one single month.
Start <- as.Date("2016-01-01")
End <- as.Date("2017-12-31")
all_dates <- seq(from = Start, to = End, by = 1)
set.seed(1)
sam1<- sample(dates, 30)
sam2<- sample(dates, 5)
sam3<- sample(dates, 120)
all_dates represents my observation period. sam1-3 contain the days of different observations. What I want to do now is to find:
The closest days to my important observations (same number of days that are in the sample)
Random days in roughly the same time the important observations took place (also same number of days as in the sample). It must not be the closest days, but roughly around the same time like in the same months or only 1 before/later.
Any days (I know how to do that, no help needed there)
My idea was to cut out the days of important observations from my whole observation period and then apply a routine that selects my control group days. That is where I stuck now. Any ideas?
I have a dataframe with Precipitation data for every day from January 1961 to December 2017 that looks like this:
DF=data.frame(Years,Month,Day,Precipitation Value)
I want to create periods of 30 days starting with 1th of January of 1961 so the first period will be 1st january to 30th January 1961 and want R to calculate the number of days without rain (Precipitation Value=0). Then, I want to do the same with the next day: 2th January so the period will be 2nd january-31st January, etc. After that, I need R to create a data frame with all the results for the year 1961. So it should be a data frame with of only one column with values (those values will be the number of days without rain in every period).
Then I need to do the same thing with all the years. Which means I will end up with 56 dataframes (1 for each year) and after that I could make a matrix with all of them (putting each data frame as a row).
The thing is I DO NOT KNOW how to start. I have no idea about how making the loop. I know it should be really easy, but I am having trouble with doing it. Specially i do not know how to tell R to stop every different year and start over and make a NEW data frame/vector with values.
Please provide a reproducible subset of your data so others can help you more effectively. While I cannot teach you how to create a loop from scratch here is some code that I think will help. This code simply calculates the moving 30 day average of precipitation using a simple for loop. You can use dplyr to filter these moving averages by year and create data frames doing that. Note I'm not counting the number of no precipitation days here but you can modify the loop easily to do that if needed
df<-data.frame(year = rep(1967:2002, each =12*30),
month = rep(rep(1:12, each = 30), 36),
day = rep(seq(1,30, by = 1), 432),
precipitation = sample(1:2000, 12*36))
df
#create a column that goes from 1 to however long your dataframe is
df$marker <- 1:nrow(df)
#'Now we create a simple loop to calculate the mean precipitation for
#'every 30 day window. You can modify this to count the number of days with
#'0 precipitation
#'the new column moving precip will tell you the mean precipitation for the
#' past 30 days relative to its postion. So if your on row 55, it will give
#' you the mean precipitation from row 25 to 55
df$movingprecip<-NA
for(i in 1:nrow(df)){
start = i #this says we start at i
end = i + 30 #we end 30 days later from i
if(end > nrow(df)){
#here I tell R to print this if there is not enough days
#in the dataset (30 days) to calculate the 30 day window mean
#this happens at the beginning of the dataset because we need to get to the
#30th row to start calculating means
print("not able to calculate, not 30 days into the data yet")
}else{
#Here I calculate the mean the of the past 30 days of precip
df$movingprecip[end] = mean(df[start:end,4])}
}
I am trying to calculate utilization rates by relative employee lifespans. I need to assign a total number of hours available to this employee between the earliest and furthest date in which time was recorded. From there I will use this as the divisor in utilization rate = workhours / totalhours.
When testing the bizdays function, I tried a simple example.
bizdays::bizdays("2020-02-07","2020-02-14")
[1] 7
Any reason why the function is not returning the correct number of business days?
I am expecting 5 business days since 2/07 was a Friday so only 1 week should be included.
The goals is to use bizdays in the following function that will be applied to a grouped df with gapply.
timeentry = function(x){
end_date = max(x$terminus)#creates an end_date variable from further end date in the group
start_date = min(x$onset) #creates a start_date from earliest start date in the group
start_date %>% bizdays(end_date) * 8 #subtracts dates and multiple by 8 to get work hours between two dates
}
I will apply the function in this manner. Unfortunately, it returns an error suggesting it cannot allocate vector of size 4687 gb. This is a separate issue I hope someone can point out.
util = group %>% gapply(.,timeentry)
where group is the grouped df.
Try setting up your calendar with create.calendar
library(bizdays)
create.calendar(name = "demo", weekdays = c("saturday", "sunday"))
bizdays::bizdays("2020-02-07","2020-02-14", cal = "demo")
[1] 5
I've seen a lot of solutions to working with groups of times or date, like aggregate to sum daily observations into weekly observations, or other solutions to compute a moving average, but I haven't found a way do what I want, which is to pluck relative dates out of data keyed by an additional variable.
I have daily sales data for a bunch of stores. So that is a data.frame with columns
store_id date sales
It's nearly complete, but there are some missing data points, and those missing data points are having a strong effect on our models (I suspect). So I used expand.grid to make sure we have a row for every store and every date, but at this point the sales data for those missing data points are NAs. I've found solutions like
dframe[is.na(dframe)] <- 0
or
dframe$sales[is.na(dframe$sales)] <- mean(dframe$sales, na.rm = TRUE)
but I'm not happy with the RHS of either of those. I want to replace missing sales data with our best estimate, and the best estimate of sales for a given store on a given date is the average of the sales 7 days prior and 7 days later. E.g. for Sunday the 8th, the average of Sunday the 1st and Sunday the 15th, because sales is significantly dependent on day of the week.
So I guess I can use
dframe$sales[is.na(dframe$sales)] <- my_func(dframe)
where my_func(dframe) replaces every stores' sales data with the average of the store's sales 7 days prior and 7 days later (ignoring for the first go around the situation where one of those data points is also missing), but I have no idea how to write my_func in an efficient way.
How do I match up the store_id and the dates 7 days prior and future without using a terribly inefficient for loop? Preferably using only base R packages.
Something like:
with(
dframe,
ave(sales, store_id, FUN=function(x) {
naw <- which(is.na(x))
x[naw] <- rowMeans(cbind(x[naw+7],x[naw-7]))
x
}
)
)
I have this data frame in R:
steps day month
4758 Tuesday December
9822 Wednesday December
10773 Thursday December
I want to iterate over the data frame and apply a function to the steps column based on the value in the month column. I'm trying to work out the average number of steps per weekday for each month.
I want to output to a new data frame like so where the week days repeat but I only have the average values per day:
average.steps day month
4500 Tuesday December
9000 Wednesday December
1000 Thursday December
I can work out how to work out the averages for the data frame as a whole, but want to use a for loop to apply it just for step values from the same month.
avgsteps <- ddply(DATA, "day", summarise, msteps = mean(steps))
My basic idea for the for function was:
f <- function(m in month) {ddply(DATA, "day", summarise, msteps = mean(steps))}
But it won't process it and throws the error:
Error: unexpected 'in' in "f <- function(m in"
Any help would be greatly appreciated!
EDIT:
SO I've tried #agstudy's suggested fix (below) and it gets the right data structure (single value for each weekday for each month), but the value assigned to each day is identical. I'm a bit confused what could be going wrong.
steps.month.day.avg <- ddply(steps.month.day, .(fitbit.day,fitbit.month), summarise, msteps = mean(steps))
No need to loop here , you should just change the variables to split data frame by,
ddply(DATA, .(day,month), summarise, msteps = mean(steps))