I Have few variables AGE ACT_TYPE GENDER in my data frame. Instead of printing each of these factor variable's level distribution, I have used for loop to print the distribution. However nothing seems to be printing. Please let me know how to resolve the issue ..
> str(combin)
Classes ‘data.table’ and 'data.frame': 500000 obs. of 333 variables:
$ CUSTOMER_ID : int 385793 286891 108751 278651 23637 130723 5694 275523 163723 469852 ...
$ ACT_TYPE : Factor w/ 2 levels "CSA","SA": 1 1 1 1 1 1 2 2 2 1 ...
$ GENDER : Factor w/ 3 levels "","F","M": 3 3 3 3 3 3 3 3 3 3 ...
$ LEGAL_ENTITY : Factor w/ 7 levels "ASSOCIATION",..: 3 3 3 3 3 3 3 3 3 3
combin[, prop.table(table(GENDER))]
GENDER
F M
0.000272 0.232436 0.767292
combin[, prop.table(table(ACT_TYPE))]
ACT_TYPE
CSA SA
0.710686 0.289314
If I replace the above printing to the display with forloop, I don't see any o/p.
Please let me know where I am going wrong...
for(i in names(combin)) {
combin[, prop.table(table(names(combin)[i]))]
}
Also suggest me how can I apply a condition in the for loop to only print the
distribution only if it's a factor variable.
You could use purrr to loop through each column of the data frame and return a list, where each item in the list corresponds to a column and the columns that are factors are the prop.tables
library(purrr)
#generate some random data like yours
mydf <- data_frame(
id = sample(1:100, 10,replace = F)
, ACT_TYPE = factor(sample(c("CSA", "SA"),10, replace = T))
, GENDER = factor(sample(c("", "F", "M"), 10, replace = T))
)
# use map_if to generate prop.tables when the column is a factor
map_if(mydf, ~class(.x) == "factor", ~prop.table(table(.x)) )
Related
I have a dataset with many columns and I'd like to locate the columns that have fewer than n unique responses and change just those columns into factors.
Here is one way I was able to do that:
#create sample dataframe
df <- data.frame("number" = c(1,2.7,8,5), "binary1" = c(1,0,1,1),
"answer" = c("Yes","No", "Yes", "No"), "binary2" = c(0,0,1,0))
n <- 3
#for each column
for (col in colnames(df)){
#check if the first entry is numeric
if (is.numeric(df[col][1,1])){
# check that there are fewer than 3 unique values
if ( length(unique(df[col])[,1]) < n ) {
df[[col]] <- factor(df[[col]])
}
}
}
What is another, hopefully more succinct, way of accomplishing this?
Here is a way using tidyverse.
We can make use of where within across to select the columns with logical short-circuit expression where we check
the columns are numeric - (is.numeric)
if the 1 is TRUE, check whether number of distinct elements less than the user defined n
if 2 is TRUE, then check all the unique elements in the column are 0 and 1
loop over those selected column and convert to factor class
library(dplyr)
df1 <- df %>%
mutate(across(where(~is.numeric(.) &&
n_distinct(.) < n &&
all(unique(.) %in% c(0, 1))), factor))
-checking
str(df1)
'data.frame': 4 obs. of 4 variables:
$ number : num 1 2.7 8 5
$ binary1: Factor w/ 2 levels "0","1": 2 1 2 2
$ answer : chr "Yes" "No" "Yes" "No"
$ binary2: Factor w/ 2 levels "0","1": 1 1 2 1
A base R option
out <- list2DF(
lapply(
df,
function(x) {
if (length(unique(x)) < n & all(x %in% c(0, 1))) as.factor(x) else x
}
)
)
gives
> str(out)
'data.frame': 4 obs. of 4 variables:
$ number : num 1 2.7 8 5
$ binary1: Factor w/ 2 levels "0","1": 2 1 2 2
$ answer : chr "Yes" "No" "Yes" "No"
$ binary2: Factor w/ 2 levels "0","1": 1 1 2 1
You can also use imap function to great advantage in this case. A thousand thanks to my dear friend #akrun who never ceases to inspire us:
library(dplyr)
library(purrr)
n <- 3
df %>%
imap_dfc(~ if(is.numeric(.x) & length(unique((.x)) < n)
& all(unique(.x) %in% c(0, 1))) {
factor(df[[.y]])
} else {
df[[.y]]
}
)
# A tibble: 4 x 4
number binary1 answer binary2
<dbl> <fct> <chr> <fct>
1 1 1 Yes 0
2 2.7 0 No 0
3 8 1 Yes 1
4 5 1 No 0
I am using the following code, which works fine (improvement suggestions very much welcome):
WeeklySlopes <- function(Year, Week){
DynamicQuery <- paste('select DayOfYear, Week, Year, Close from SourceData where year =', Year, 'and week =', Week, 'order by DayOfYear')
SubData = sqldf(DynamicQuery)
SubData$X <- as.numeric(rownames(SubData))
lmfit <- lm(Close ~ X, data = SubData)
lmfit <- tidy(lmfit)
Slope <- as.numeric(sqldf("select estimate from lmfit where term = 'X'"))
e <- globalenv()
e$WeeklySlopesDf[nrow(e$WeeklySlopesDf) + 1,] = c(Year,Week, Slope)
}
WeeklySlopesDf <- data.frame(Year = integer(), Week = integer(), Slope = double())
WeeklySlopes(2017, 15)
WeeklySlopes(2017, 14)
head(WeeklySlopesDf)
Is there really no other way to append a row to my existing dataframe. I seem to need to access the globalenv. On the other hand, why can sqldf 'see' the 'global' dataframe SourceData?
dfrm <- data.frame(a=1:10, b=letters[1:10]) # reproducible example
myfunc <- function(new_a=20){ g <- globalenv(); g$dfrm[3,1] <- new_a; cat(dfrm[3,1])}
myfunc()
20
dfrm
a b
1 1 a
2 2 b
3 20 c # so your strategy might work, although it's unconventional.
Now try to extend dataframe outside a function:
dfrm[11, ] <- c(a=20,b="c")
An occult disaster (conversion of numeric column to character):
str(dfrm)
'data.frame': 11 obs. of 2 variables:
$ a: chr "1" "2" "20" "4" ...
$ b: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
So use a list to avoid occult coercion:
dfrm <- data.frame(a=1:10, b=letters[1:10]) # start over
dfrm[11, ] <- list(a=20,b="c")
str(dfrm)
'data.frame': 11 obs. of 2 variables:
$ a: num 1 2 3 4 5 6 7 8 9 10 ...
$ b: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
Now try within a function:
myfunc <- function(new_a=20, new_b="ZZ"){ g <- globalenv(); g$dfrm[nrow(dfrm)+1, ] <- list(a=new_a,b=new_b)}
myfunc()
Warning message:
In `[<-.factor`(`*tmp*`, iseq, value = "ZZ") :
invalid factor level, NA generated
str(dfrm)
'data.frame': 12 obs. of 2 variables:
$ a: num 1 2 3 4 5 6 7 8 9 10 ...
$ b: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
So it succeeds, but if there are any factor columns, non-existent levels will get turned into NA values (with a warning). You method of using named access to objects in the global environment is rather unconventional but there is a set of tested methods that you might want to examine. Look at ?R6. Other options are <<- and assign which allows one to specify the environment in which the assignment is to occur.
I'm trying to get the error rates for a Naive Bayes classifier, by adding in each variable incrementally. For example I have 25 variables in my dataset. I want to get the error rates of the model as I add in one variable at a time. So you know it would output the error rate of the model with the first 2 columns, the error rate with the first 3 columns, then with the first 4 columns, and so on up to the last column.
Here is the pseudocode of what I'm trying to achieve
START
IMPORT DATASET WITH ALL VARIABLES
num_variables = num_dataset_cols
i= 1
WHILE (i <= num_variables)
{
CREATE NEW DATASET WITH x COLUMNs
BUILD THE MODEL
GET THE ERROR RATE
ADD IN NEXT COLUMN
i = i + 1
}
Here is a reproducible question. Obviously you can't build a NB classifier with this data, but that's not my problem. My problem is adding in the columns one by one. So far, the only way I can do it is by overwriting each column. For a NB classifier, the first column is the class node, so there must be at least 2 columns starting off in order for it to run.
#REPRODUCIBLE EXAMPLE
col1 <- c("A", "B", "C", "D", "E")
col2 <- c(1,2,3,4,5)
col3 <- c(TRUE, FALSE, FALSE, TRUE, FALSE)
col4 <- c("n","y","y","n","y")
col5 <- c("10", "15", "50", "100", "20")
dataset <- data.frame(col1, col2, col3, col4,col5)
num_variables <- ncol(dataset)
i <- 1
while i <= num_variables
{
data <- dataset[c(1, i+1)]
str(data)
#BUILD MODEL AND GET VALIDATION ERROR
#INCREMENT i TO GET NEXT COLUMN
i <- i + 1
}
You should be able to see from the str(data) that each time the column is overwritten. Does anyone know how I could go about adding each column without overwriting the previous one? Someone suggested an array to me, but I'm not too familiar with arrays in R. Would this work?
I think this is what you want.
col1 <- c("A", "B", "C", "D", "E")
col2 <- c(1,2,3,4,5)
col3 <- c(TRUE, FALSE, FALSE, TRUE, FALSE)
col4 <- c("n","y","y","n","y")
col5 <- c("10", "15", "50", "100", "20")
dataset <- data.frame(col1, col2, col3, col4,col5)
dataset
num_variables <- ncol(dataset)
num_variables
i <- 1
while (i <= num_variables) {
data <- dataset[, 1:i]
print(str(data))
#BUILD MODEL AND GET VALIDATION ERROR
#INCREMENT i TO GET NEXT COLUMN
i <- i + 1
}
Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
NULL
'data.frame': 5 obs. of 2 variables:
$ col1: Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
$ col2: num 1 2 3 4 5
NULL
'data.frame': 5 obs. of 3 variables:
$ col1: Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
$ col2: num 1 2 3 4 5
$ col3: logi TRUE FALSE FALSE TRUE FALSE
NULL
'data.frame': 5 obs. of 4 variables:
$ col1: Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
$ col2: num 1 2 3 4 5
$ col3: logi TRUE FALSE FALSE TRUE FALSE
$ col4: Factor w/ 2 levels "n","y": 1 2 2 1 2
NULL
'data.frame': 5 obs. of 5 variables:
$ col1: Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
$ col2: num 1 2 3 4 5
$ col3: logi TRUE FALSE FALSE TRUE FALSE
$ col4: Factor w/ 2 levels "n","y": 1 2 2 1 2
$ col5: Factor w/ 5 levels "10","100","15",..: 1 3 5 2 4
NULL
You can use append function after defining output variable
data <- dataset[c(1, i+1)]
append(output, data)
str(data)
Using the "assign" function within a while loop can be helpful for issues like this. You don't show the model syntax, but something like this should work:
dataset$errorrate <- [whatever makes this calculation, assuming it is vectorized]
name <- paste0(errorrate, i)
assign(name, dataset$errorrate)
...
This should leave you with i variables containing error estimate for each model run. If you are looking for one parameter estimate per model you can assign the single estimate a unique name within the global environment using the process above and then rbind them together after the loop has finished
I have a wide data.frame that is all character vectors (df1). I have a separate vector(vec1) that contains the column classes I'd like to assign to each of the columns in df1.
If I was using read.csv(), I'd use the colClasses argument and set it equal to vec1, but there doesn't appear to be a similar option for an existing data.frame.
Any suggestions for a fast way to do this besides a loop?
I don't know if it will be of help but I have run into the same need many times and I have created a function in case it helps:
reclass <- function(df, vec){
df[] <- Map(function(x, f){
#switch below shows the accepted values in the vector
#you can modify it and/or add more
f <- switch(f,
as.is = 'force',
factor = 'as.factor',
num = 'as.numeric',
char = 'as.character')
#takes the name of the function and fetches the function
f <- get(f)
#apply the function
f(x)
},
df,
vec)
df
}
It uses Map to pass in a vector of classes to the data.frame. Each element corresponds to the class of the column. The length of both the dataframe and the vector need to be the same.
I am using switch as well to make the corresponding classes shorter to type. Use as.is to keep the class the same, the rest are self explanatory I think.
Small example:
df1 <- data.frame(1:10, letters[1:10], runif(50))
> str(df1)
'data.frame': 50 obs. of 3 variables:
$ X1.10 : int 1 2 3 4 5 6 7 8 9 10 ...
$ letters.1.10.: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
$ runif.50. : num 0.0969 0.1957 0.8283 0.1768 0.9821 ...
And after the function:
df1 <- reclass(df1, c('num','as.is','char'))
> str(df1)
'data.frame': 50 obs. of 3 variables:
$ X1.10 : num [1:50] 1 2 3 4 5 6 7 8 9 10 ...
$ letters.1.10.: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10 ...
$ runif.50. : chr [1:50] "0.0968757788650692" "0.19566105119884" "0.828283685725182" "0.176784737734124" ...
I guess Map internally is a loop but it is written in C so it should be fast enough.
May be you could try this function that makes the same work.
reclass <- function (df, vec_types) {
for (i in 1:ncol(df)) {
type <- vec_types[i]
class(df[ , i]) <- type
}
return(df)
}
and this is an example of vec_types (vector of types):
vec_types <- c('character', rep('integer', 3), rep('character', 2))
you can test the function (reclass) whith this table (df):
table <- data.frame(matrix(sample(1:10,30, replace = T), nrow = 5, ncol = 6))
str(table) # original column types
# apply the function
table <- reclass(table, vec_types)
str(table) # new column types
I use ddply a lot. I use ordered factors occasionally. Calling ddply on a data frame that contains an ordered factor drops any ordering in the recombined data frame.
I wrote the following wrapper for ddply that records level ordering and then re-applies it on any columns that were ordered originally:
dat <- data.frame(a=runif(10),b=factor(letters[10:1],
levels=letters[10:1],ordered=TRUE),
c = rep(letters[1:2],times=5),
d = factor(rep(c('lev1','lev2'),times=5),ordered=TRUE))
#Drops ordering on b and d
dat1 <- ddply(dat,.(c),transform,log_a = log(a))
ddplyKeepOrder <- function(dat,...){
orderedCols <- colnames(dat)[sapply(dat,is.ordered)]
levs <- lapply(dat[,orderedCols,drop=FALSE],levels)
result <- ddply(.data = dat,...)
ind <- match(orderedCols,colnames(result))
levs <- levs[!is.na(ind)]
orderedCols <- orderedCols[!is.na(ind)]
ind <- ind[!is.na(ind)]
if (length(ind) > 0){
for (i in 1:length(ind)){
result[,orderedCols[i]] <- factor(result[,orderedCols[i]],
levels=levs[[i]],ordered=TRUE)
}
}
return(droplevels(result))
}
#Preserves ordering on b and d
dat2 <- ddplyKeepOrder(dat,.variables = .(c),.fun = transform,log_a = log(a))
I haven't checked this function thoroughly so there might be cases it doesn't handle. Is there a better/more complete way to handle this? I could probably remove the for loop if I thought about it a bit, I suppose.
In particular, the checking I do after the ddply call to see if there are still any of the original ordered factors present seems really ugly, but I would like the function to be able to handle cases where ddply alters which columns are present, possibly removing ordered factors.
Thoughts?
I use the code below for these types of problems ("ddply" not "ordered factor") and it seems to handle your specific example without issue (other than different row names).
> dat2 <- do.call(rbind, lapply(split(dat, dat$c), transform, log_a=log(a)))
> str(dat2)
'data.frame': 10 obs. of 5 variables:
$ a : num 0.216 0.607 0.197 0.171 0.797 ...
$ b : Ord.factor w/ 10 levels "j"<"i"<"h"<"g"<..: 1 3 5 7 9 2 4 6 8 10
$ c : Factor w/ 2 levels "a","b": 1 1 1 1 1 2 2 2 2 2
$ d : Ord.factor w/ 2 levels "lev1"<"lev2": 1 1 1 1 1 2 2 2 2 2
$ log_a: num -1.532 -0.499 -1.625 -1.767 -0.227 ...