R Function behaves differently than the code entered line by line - r

I am at a loss. Googling has failed me because I'm not sure I know the right question to ask.
I have a data frame (df1) and my goal is to use a function to get a moving average using forecast::ma.
Here is str(df1)
'data.frame': 934334 obs. of 6 variables:
$ clname : chr ...
$ dos : Date, format: "2011-10-05" ...
$ subpCode: chr
$ ch1 : chr "
$ prov : chr
$ ledger : chr
I have a function that I am trying to write.
process <- function(df, y, sub, ...) {
prog <- df %>%
filter(subpCode == sub) %>%
group_by(dos, subpCode) %>%
summarise(services = n())
prog$count_ts <- ts(prog[ , c('services')])
}
The problem is that when I run the function, my final result is data object that is 1x1798 and it's just a time series. If I go a run the code line by line I get what I need but my function that hypothetically does the same thing wont work.
Here is my desired result
Classes ‘grouped_df’, ‘tbl_df’, ‘tbl’ and 'data.frame': 1718 obs. of 4 variables:
$ dos : Date, format: "2010-09-21" "2010-11-18" "2010-11-19" "2010-11-30" ...
$ subpCode: chr "CII " "CII " "CII " "CII " ...
$ services: int 1 1 2 2 2 2 1 2 1 3 ...
$ count_ts: Time-Series [1:1718, 1] from 1 to 1718: 1 1 2 2 2 2 1 2 1 3 ...
..- attr(*, "dimnames")=List of 2
.. ..$ : NULL
.. ..$ : chr "services"
- attr(*, "vars")= chr "dos"
- attr(*, "drop")= logi TRU
And here is the code that gets it.
CII <- df1 %>%
filter(subpCode == "CII ") %>%
group_by(dos, subpCode) %>%
summarise(services = n())
CII$count_ts <- ts(CII[ , c('services')])
Could someone point me in the right direction. I've exhausted my usual places.
Thanks!

Following the vignette pointed out by #CalumYou, you should use more something like this:
process <- function(df, sub) {
## Enquoting sub
sub <- enquo(sub)
## Piping stuff
prog <- df %>%
filter(!! subpCode == sub) %>%
group_by(dos, subpCode) %>%
summarise(services = n())
prog$count_ts <- ts(prog[ , c('services')])
## Returning the prog object
return(prog)
}

Related

Dynamically Changing Data Type for a Data Frame

I have a set of data frames belonging to many countries consisting of 3 variables (year, AI, OAD). The example for Zimbabwe is shown as below,
>str(dframe_Zimbabwe_1955_1970)
'data.frame': 16 obs. of 3 variables:
$ year: chr "1955" "1956" "1957" "1958" ...
$ AI : chr "11.61568161" "11.34114927" "11.23639317" "11.18841409" ...
$ OAD : chr "5.740789488" "5.775882473" "5.800441036" "5.822536579" ...
I am trying to change the data type of the variables in data frame to below so that I can model the linear fit using lm(dframe_Zimbabwe_1955_1970$AI ~ dframe_Zimbabwe_1955_1970$year).
>str(dframe_Zimbabwe_1955_1970)
'data.frame': 16 obs. of 3 variables:
$ year: int 1955 1956 1957 1958 ...
$ AI : num 11.61568161 11.34114927 11.23639317 11.18841409 ...
$ OAD : num 5.740789488 5.775882473 5.800441036 5.822536579 ...
The below static code able to change AI from character (chr) to numeric (num).
dframe_Zimbabwe_1955_1970$AI <- as.numeric(dframe_Zimbabwe_1955_1970$AI)
However when I tried to automate the code as below, AI still remains as character (chr)
countries <- c('Zimbabwe', 'Afghanistan', ...)
for (country in countries) {
assign(paste('dframe_',country,'_1955_1970$AI', sep=''), eval(parse(text = paste('as.numeric(dframe_',country,'_1955_1970$AI)', sep=''))))
}
Can you advice what I could have done wrong?
Thanks.
42: Your code doesn't work as written but with some edits it will. in addition to the missing parentheses and wrong sep, you can't use $'column name' in assign, but you don't need it anyway
for (country in countries) {
new_val <- get(paste( 'dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
remove(new_val)
}
proof it works:
dframe_Zimbabwe_1955_1970 <- data.frame(year = c("1955", "1956", "1957"),
AI = c("11.61568161", "11.34114927", "11.23639317"),
OAD = c("5.740789488", "5.775882473", "5.800441036"),
stringsAsFactors = F)
str(dframe_Zimbabwe_1955_1970)
'data.frame': 3 obs. of 3 variables:
$ year: chr "1955" "1956" "1957"
$ AI : chr "11.61568161" "11.34114927" "11.23639317"
$ OAD : chr "5.740789488" "5.775882473" "5.800441036"
countries <- 'Zimbabwe'
for (country in countries) {
new_val <- get(paste( 'dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
remove(new_val)
}
str(dframe_Zimbabwe_1955_1970)
'data.frame': 3 obs. of 3 variables:
$ year: num 1955 1956 1957
$ AI : num 11.6 11.3 11.2
$ OAD : num 5.74 5.78 5.8
It's going to be considered fairly ugly code by teh purists but perhaps this:
for (country in countries) {
new_val <- get(paste('dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
}
Using the get('obj_name') function is considered cleaner than eval(parse(text=...)). It would get handled more R-naturally had you assembled these dataframes in a list.

How to speed up code with loop in R

Problem:
I have two data frames.
DF with payment log:
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 53682 obs. of 7 variables:
str(moneyDB)
$ user_id : num 59017170 57859746 58507536 59017667 59017795 ...
$ reg_date: Date, format: "2016-08-06" "2016-07-01" "2016-07-19" ...
$ date : Date, format: "2016-08-06" "2016-07-01" "2016-07-19" ...
$ money : num 0.293 0.05 0.03 0.03 7 ...
$ type : chr "1" "2" "2" "1" ...
$ quality : chr "VG" "no_quality" "no_quality" "VG" ...
$ geo : chr "Canada" "NO GEO" "NO GEO" "Canada" ...
Here is its structure. Its just a log of all transactions.
Also i have second data frame:
str(grPaysDB)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 335591 obs. of 9 variables:
$ reg_date : Date, format: "2016-05-01" "2016-05-01" "2016-05-01" ...
$ date : Date, format: "2016-05-01" "2016-05-01" "2016-05-01" ...
$ type : chr "1" "1" "1" "1" ...
$ quality : chr "VG" "VG" "VG" "VG" ...
$ geo : chr "Australia" "Canada" "Finland" "Canada" ...
$ uniqPayers : num 0 1 0 1 1 0 0 1 0 3 ...
Its Grouped data from first data frame + zero transactions. For example, there is a lot of rows in second data frame with zero payers. Thats why second data frame is greater then first.
I need to add column weeklyPayers to the second data frames. Weekly payers is sum unique payers for the last 7 days. I tried do it via loop, but it wooks too long. Is there any another vectorized ideas, how to realise this?
weeklyPayers <- vector()
for (i in 1:nrow(grPaysDB)) {
temp <- moneyDB %>%
filter(
geo == grPaysDB$geo[i],
reg_date == grPaysDB$reg_date[i],
quality == grPaysDB$quality[i],
type == grPaysDB$type[i],
between(date, grPaysDB$date[i] - 6, grPaysDB$date[i])
)
weeklyPayers <- c(weeklyPayers, length(unique(temp$user_id)))
}
grPaysDB <- cbind(grPaysDB, weeklyPayers)
In this loop for each row in second data frame i find rows in first data frame with right geo,type, quality and reg_date and range of dates. And then I can calculate number of unique payers.
I may be misunderstanding, but I think this should be fairly simple, using filter and summarise in dplyr. However, as #Hack-R mentioned, it would be helpful to have your dataset. But it would look something like:
library(dplyr)
weeklyPayers <- grPaysDB %>%
filter(date > ADD DATE IN QUESTION) %>%
summarise(sumWeeklyPayers = sum(uniqPayers))
Then again, I may well have misunderstood. If your question involves summing for each week, then you may want to investigate daily2weekly in the timeSeries package and then using group_by for the weekly variable that transpires.
I would try making a join on your datasets using merge on multiple columns (c('geo', 'reg_date', 'quality', 'type') and filter the result based on the dates. After that, aggregate using summarise.
But I am not completely sure why you want to add the weeklypayers to every transaction. Isn't it more informative or easier to aggregate your data on week number (with dplyr). Like so:
moneyDB %>% mutate(week = date- as.POSIXlt(date)$wday) %>%
group_by(geo, reg_date, quality, type, week) %>%
summarise(weeklyPayers = n())

replacement has x rows, data has y - paste() function

I am trying to group by the following sample values,
latitude | longitude | TotalGreenhouseGases | Amount | Branch |End Date
-37.80144| 144.95402| 42965.9868|32549.99|Arts and Culture| 07/31/2013 12:00:00 AM
-37.80144| 144.95402| 43246.6716|32762.63|Arts and Culture| 08/30/2013 12:00:00 AM
-37.80144| 144.95402| 21374.1264|16192.52|Arts and Culture| 09/31/2013 12:00:00 AM
mapdata <- aggregate(cbind(TotalGreenhouseGases,Amount) ~ latitude+longitude,data = dt2,FUN=function(dt2) c(mn =sum(dt2), n=length(dt2) ) )
163 obs. and 4 Variables are created as a result, now to plot it in a map using plot.ly i am trying to add a text for hovering,
mapdata$hover <- paste( mapdata$TotalGreenhouseGases, "CO2 Emission ",'<br>', "Resource Consumption ", mapdata$Amount)
but this results in the following error,
Error in `$<-.data.frame`(`*tmp*`, "hover", value = c("264.06428571 CO2 Emission <br> Resource Consumption 200", :
replacement has 326 rows, data has 163
can anyone let me know where I am going wrong or if it has been solved before can you please provide a link for that.
I think the the problem is that the way you created mapdata you end up with a list of length of 2 for both TotalGreenhouseGases and Amount.
> str(mapdata)
'data.frame': 1 obs. of 5 variables:
$ latitude : num -37.8
$ longitude : num 145
$ TotalGreenhouseGases: num [1, 1:2] 107587 3
..- attr(*, "dimnames")=List of 2
.. ..$ : NULL
.. ..$ : chr "mn" "n"
$ Amount : num [1, 1:2] 81505 3
..- attr(*, "dimnames")=List of 2
.. ..$ : NULL
.. ..$ : chr "mn" "n"
So if you want to use the sum of these values in your paste function then you will need to use [1] indexing, if you need to use the sample size n then use [2]. For example:
mapdata$hover <- paste( mapdata$TotalGreenhouseGases[1],
"CO2 Emission ",'<br>', "Resource Consumption ",
mapdata$Amount[1])
will give you
[1] "107586.7848 CO2 Emission <br> Resource Consumption 81505.14"

How to use "within" in R?

I've been struggling with finding a way to evaluate an R expression in an environment constructed from data. I have a dataframe:
head(DATA1)
COD_CLI ENDEUD_FINAL
1 01002901 Mediana Empresa
2 01002932 No Sobreendeudado
3 04203409 No Sobreendeudado
...
and I try to complete another data (DATA2):
head(DATA2)
COD_CLI_W ENDEUD_FINAL
1 01002190
2 01002913
3 04203401
...
DATA2 is larger than DATA1, if the same COD_CLI/COD_CLI_W is in both datas, I take the second column of DATA1, if not I must evaluate another dataframe "wallet":
> str(wallet)
'data.frame': 81101 obs. of 8 variables:
$ COD_CREDITO : chr "0040410166104" "00000363393" "0060030164135" "004023854M" ...
$ COD_CLI : chr "00402037" "00166750" "00178607" "40097700" ...
$ TIPO_DE_CREDITO : chr "12.-CONSUMO NO REVOLVENTE" "10.-MICROEMPRESA" "10.-MICROEMPRESA" "10.-MICROEMPRESA" ...
$ SITUACION_SAFI : chr "CASTIGADO" "CASTIGADO" "CASTIGADO" "CASTIGADO" ...
$ COD_TIP_PRESTAMO: chr "0747" "0748" "0748" "0747" ...
$ ATR_SOL : num 0 0 0 0 0 0 0 0 0 0 ...
$ CAP_SOL : num 313.37 3.16 1670.51 3010 2327.71 ...
$ NUM_ENT : num 3 1 2 1 1 3 2 1 4 2 ...
Now the code I run is:
DATA2 <- within(DATA2,{
CALIF_RCD <- ifelse(COD_CLI_W %in% DATA1$COD_CLI,DATA1$ENDEUD_FINAL[which(DATA1$COD_CLI %in% COD_CLI_W)],
ifelse(wallet$TIPO_DE_CREDITO[which(wallet$COD_CLI %in% COD_CLI_W)[1]] == "08.-MEDIANA EMPRESA","Mediana Empresa",
ifelse(wallet$NUM_ENT[which(wallet$COD_CLI %in% COD_CLI_W)[1]]<5,"No Sobreendeudado","Sobreendeudado")))
}
)
the output is wrong in most of the cases. I'm new to R and I would like to know how to code it in a properly manner. Any help would be much appreciated.
I took the first approach using "merge":
DATA3 <- merge(DATA1, DATA2, by.x = "COD_CLI", by.y = "COD_CLI_W", all.y=TRUE)
DATA3 <- DATA3[!complete.cases(DATA3),]
After that, I analysed the left outer side in DATA3:
w = NULL
for(i in 1:length(DATA3$ENDEUD_FINAL))
{
w = which(wallet$COD_CLI %in% DATA3$COD_CLI[i])[1]
DATA3$ENDEUD_FINAL[i] <- ifelse(wallet$TIPO_DE_CREDITO[w] == "08.-MEDIANA EMPRESA","Mediana Empresa",
ifelse(wallet$NUM_ENT[w]<5,"No Sobreendeudado","Sobreendeudado"
))
}
and finally "rbind" DATA1 and DATA2:
DATA2 <- rbind(DATA1, DATA3)

Trying to aggregate Queried data using a forloop in R

I am having a little bit of trouble creating a loop that
sqlQueries every device in the variable "sensorname" (this is roughly 30 elements, but will increase in the future)
takes the data table associated with the device query and puts it into a separate data frame "data1" but keeps adding into it.
Below is my sample loop and a sample of what data1 looks like which is "correct", but not complete. LSF20_3a0925 is the last element in the variable sensorname so essentially the loop runs 30 times each time overwriting the data in variable data1 until it runs for the last time.
library(RODBC)
ch <- odbcConnect("SweetLab", uid='---', pwd='------')
sqlQuery(ch, "use SweetDatabase")
sensorname <- sqlQuery(ch,paste("SELECT site_device.code
FROM site_device, device
WHERE site_device.did=device.id AND
device.name='LSF20'
LIMIT 0, 1000;",
sep="")
)
for(k in 1:length(sensorname[[1]])){
sqlQuery(ch, "use SweetAnalysis")
sql <- na.omit(sqlQuery(ch,paste("select * From ",sensorname[[1]][k],"_Events",sep="")));
if (is.null(sql))
{return(NULL)}
data1 <- merge(sensorname[[1]][k],sql)
}
#############################################
data1
x row_names PeaksP1Time PeaksP1
1 LSF20_3a0925 24 1346781683 5.076920
2 LSF20_3a0925 31 1358444323 0.043240
3 LSF20_3a0925 13 1358444463 0.133170
4 LSF20_3a0925 12 1358445120 5.286443
Any help would be most appretiated I am new to writing code in general so please excuse me if this is a dumb question. I've searched around for a bit on this topic, but honestly wasn't quite sure how to search for this topic.
After some tweaking this looks great!
library(RODBC)
ch <- odbcConnect("SweetLab", uid='***', pwd='******')
sqlQuery(ch, "use SweetDatabase")
sensorname <- sqlQuery(ch,paste("select site_device.code from site_device, device where site_device.did=device.id and device.name='LSF20' LIMIT 0, 1000;",sep=""));
sqlQuery(ch, "use SweetAnalysis")
Datalist <- lapply(sensorname[[1]],function(x){
query <- paste("SELECT PeaksP1Time,PeaksP1 FROM ",x,"_Events",sep="")
dat <- (na.omit(sqlQuery(ch,query)))
data2.nlist<-list(device=x,data=dat)
names(Datalist)<-sensorname$code[1:30]
})
close(ch)
Looking at the structure of this list I get
> str(Datalist[1:3])
List of 3
$ LSF20_39ecf7:List of 1
..$ :'data.frame': 306 obs. of 2 variables:
.. ..$ PeaksP1Time: num [1:306] 1.35e+09 1.35e+09 1.35e+09 1.35e+09 1.35e+09 ...
.. ..$ PeaksP1 : num [1:306] 4.5 4.379 0.706 3 0 ...
$ LSF20_39cd3e:List of 1
..$ :'data.frame': 202 obs. of 2 variables:
.. ..$ PeaksP1Time: num [1:202] 1.35e+09 1.35e+09 1.35e+09 1.35e+09 1.35e+09 ...
.. ..$ PeaksP1 : num [1:202] 0.664 3.235 5.765 4.636 2.936 ...
.. ..- attr(*, "na.action")=Class 'omit' Named int [1:24] 203 204 205 206 207 208 209 210 211 212 ...
.. .. .. ..- attr(*, "names")= chr [1:24] "203" "204" "205" "206" ...
$ LSF20_3a09ac:List of 1
..$ :'data.frame': 42 obs. of 2 variables:
.. ..$ PeaksP1Time: num [1:42] 1.35e+09 1.35e+09 1.35e+09 1.35e+09 1.35e+09 ...
.. ..$ PeaksP1 : num [1:42] 5.589 2.897 2.713 1.706 0.831 ...
Now I'm moving on to the next phase of this which is graphing multiple sets at the same time.
My problem is how do I tell R that it should graph the data with in each list or specific lists. I have a save file of the work history if anyone wants to work with something reproducible.
Ok..without a reproducible example it is not clear what you asked. here how I would do it..
I open the connection
I loop using lapply to create a list
I close the connection
I bind list elements into data.frame. I assume taht you have the same columns for different sensor table.
ch <- odbcConnect("SweetLab", uid='---', pwd='------')
ll <- lapply(sensorname[[1]],function(x){
query <- paste("SELECT * FROM ",x,"_Events",sep="")
dat <- na.omit(sqlQuery(ch,query))
data.frame(sensor=x,dat)
})
close(ch)
data1 <- do.call(rbind,ll)
This is resource intensive, but will work.
# before your for loop
results <- list()
#inside your for loop
for (k ......) {
....
....
results[[k]] <- sql
}
# after your for loop
Data1 <- do.call(rbind, results) # if same schema
# OR
Data1 <- do.call(merge, results) # if different schema

Resources