Related
I am working with a modest survey dataset (190 x 2162). The platform we are using exports to csv, but when imported every column is a factor. I could assign each column a class on import, but a recent change of hats at work has put many more surveys in my future. So, out of respect for my future self's sanity, I am looking to build a small library of functions to convert ranges of columns as needed.
Goals of the initial function:
Convert from factor to numeric
Take column names as input, rather than numbers
The core of the function appears to work fine:
startNum <- match("Q7_1", names(rawNum))
endNum <- match("Q7_19", names(rawNum))
for(i in c(startNum:endNum)){
rawNum[,i] <- as.character(rawNum[,i])
rawNum[,i] <- as.numeric(rawNum[,i])
}
However, when I attempt to wrap this in a function, it falls apart. I believe the issue is in passing the *_col_name arguments into the function, but I can't seem to find where it is going wrong.
facToNum <- function(frame_name, start_col_name, end_col_name){
startNum <- match(start_col_name, names(frame_name))
endNum <- match(end_col_name, names(frame_name))
for(i in c(startNum:endNum)){
frame_name[,i] <- as.character(frame_name[,i])
frame_name[,i] <- as.numeric(frame_name[,i])
}
}
What am I missing here? I'm sure it's something obvious, but I've had to soldier on with my partial solution, and it rankles.
I don't see any obvious mistakes in your function except that you need to return the changed dataframe at the end of the function.
facToNum <- function(frame_name, start_col_name, end_col_name){
startNum <- match(start_col_name, names(frame_name))
endNum <- match(end_col_name, names(frame_name))
for(i in c(startNum:endNum)){
frame_name[,i] <- as.character(frame_name[,i])
frame_name[,i] <- as.numeric(frame_name[,i])
}
return(frame_name)
}
You can also simplify your approach using dplyr library.
library(dplyr)
facToNum <- function(frame_name, start_col_name, end_col_name){
frame_name %>%
mutate(across(start_col_name:end_col_name, ~as.numeric(as.character(.x))))
}
df <- data.frame(a = factor(rnorm(5)), b = factor(runif(5)), c = 1:5)
str(df)
#data.frame': 5 obs. of 3 variables:
# $ a: Factor w/ 5 levels "-1.64324479436782",..: 3 1 5 4 2
# $ b: Factor w/ 5 levels "0.11049344507046",..: 1 4 3 2 5
# $ c: int 1 2 3 4 5
result <- facToNum(df, "a", "b")
str(result)
#'data.frame': 5 obs. of 3 variables:
# $ a: num -0.11733 -1.64324 1.26266 0.00823 -0.9531
# $ b: num 0.11 0.614 0.545 0.228 0.902
# $ c: int 1 2 3 4 5
I have created a function, which works well on dummy data. But, when I run this function on real data, I've got back an error
Error in wilcox.test.formula(tab[[dependent]] ~ as.factor(tab$group), :
grouping factor must have exactly 2 levels
and warning messages:
In wilcox.test.default(x = c(11.2558701380866, 31.8401548036613, : cannot compute exact p-value with ties
So, "thresholding" in my function seems not correctly split real data in two groups. Also, the sub-setting of the real data is not correct. But I don't understand why?? The dummy and real tables structure seem the same:
Structure of dummy and real data:
Dummy:
> str(tab)
'data.frame': 80 obs. of 3 variables:
$ infGrad : num 14.15 12.53 3.03 9.21 16.36 ...
$ distance : int 1 1 1 1 1 1 1 1 1 1 ...
$ uniqueGroup: Factor w/ 2 levels "x","y": 1 2 1 2 1 2 1 2 1 2 ...
Real:
> str(tab)
'data.frame': 142 obs. of 10 variables:
$ distance : num 100 100 100 100 100 100 100 100 100 100 ...
$ infGrad : num 11.3 17.4 31.8 11.1 47.8 ...
$ uniqueGroup: Factor w/ 6 levels "x",..: 5 2 5 2 5 5 5 5 3 6 ...
I have found that NAs might cause these problems, or specification of formula of the wilcox.test(y ~ x).
So, I tried to add na.omit to my function, and instead of wilcox.test(y~x) use wilcox.test(y, x). None of these have worked.
Do you have any ideas how to make my function work or how to make it more robust to accept my real data? Your help is highly appreciated.
What the code does:
classify data in two groups by "moving threshold"
test statistical differences between those groups.
I run the function with nested lapply to vary my thresholds and different data subsets.
My dummy data:
set.seed(10)
infGrad <- c(rnorm(20, mean=14, sd=8),
rnorm(20, mean=13, sd=5),
rnorm(20, mean=8, sd=2),
rnorm(20, mean=7, sd=1))
distance <- rep(c(1:4), each = 20)
uniqueGroup <- rep(c("x", "y"), 40)
tab<-data.frame(infGrad, distance, uniqueGroup)
# Create moving threshols function
movThreshold <- function(th, tab, dependent, ...) {
tab<-na.omit(tab)
# Classify data
tab$group<- ifelse(tab$distance < th, "a", "b") # does not WORK on REAL data
# Calculate wincoxon test
test<-wilcox.test(tab[[dependent]] ~ as.factor(tab$group), # specify column name
data = tab)
# Put results in a vector
c(th, dependent, round(test$p.value, 3))
}
# Define two vectors to run through
# unique group
gr.list<-unique(tab$uniqueGroup)
# unique threshold
th.list<-c(2,3,4)
# apply function over threshols and subset
res<-lapply(gr.list, function(x) lapply(th.list,
movThreshold,
tab = tab[uniqueGroup == x,], # does not work on REAL data
dependent = "infGrad"))
What seems not working on real data:
Groups classification within the function
tab$group<- ifelse(tab$distance < th, "a", "b")
Data subsetting in nested lapply loop
subsetting: tab = tab[uniqueGroup == x,]
The issue probably happens because of a single value group.
You can reproduce the error for instance adding a high value to th.list.
# unique threshold
th.list<-c(2,3,4,100)
The easiest way to avoid this is checking for the length of tab$group before performing the test.
This change in the function should suffice:
movThreshold <- function(th, tab, dependent, ...) {
tab<-na.omit(tab)
# Classify data
tab$group<- ifelse(tab$distance < th, "a", "b") # does not WORK on REAL data
# Check there are two groups
if(length(unique(tab$group))<2){return(NA)}
# Calculate wincoxon test
test<-wilcox.test(tab[[dependent]] ~ as.factor(tab$group), # specify column name
data = tab)
# Put results in a vector
c(th, dependent, round(test$p.value, 3))
}
I have the following code
anna.table<-data.frame (anna1,anna2)
write.table<-(anna.table, file="anna.file.txt",sep='\t', quote=FALSE)
my table in the end contains numbers such as the following
chr start end score
chr2 41237927 41238801 151
chr1 36976262 36977889 226
chr8 83023623 83025129 185
and so on......
after that i am trying to to get only the values which fit some criteria such as score less than a specific value
so i am doing the following
anna3<-"data/anna/anna.file.txt"
anna.total<-read.table(anna3,header=TRUE)
significant.anna<-subset(anna.total,score <=0.001)
Error: In Ops.factor(score, 0.001) <= not meaningful for factors
so i guess the problem is that my table has factors and not integers
I guess that my anna.total$score is a factor and i must make it an integer
If i read correctly the as.numeric might solve my problem
i am reading about the as.numeric function but i cannot understand how i can use it
Hence could you please give me some advices?
thank you in advance
best regards
Anna
PS : i tried the following
anna3<-"data/anna/anna.file.txt"
anna.total<-read.table(anna3,header=TRUE)
anna.total$score.new<-as.numeric (as.character(anna.total$score))
write.table(anna.total,file="peak.list.numeric.v3.txt",append = FALSE ,quote = FALSE,col.names =TRUE,row.names=FALSE, sep="\t")
anna.peaks<-subset(anna.total,fdr.new <=0.001)
Warning messages:
1: In Ops.factor(score, 0.001) : <= not meaningful for factors
again i have the same problem......
With anna.table (it is a data frame by the way, a table is something else!), the easiest way will be to just do:
anna.table2 <- data.matrix(anna.table)
as data.matrix() will convert factors to their underlying numeric (integer) levels. This will work for a data frame that contains only numeric, integer, factor or other variables that can be coerced to numeric, but any character strings (character) will cause the matrix to become a character matrix.
If you want anna.table2 to be a data frame, not as matrix, then you can subsequently do:
anna.table2 <- data.frame(anna.table2)
Other options are to coerce all factor variables to their integer levels. Here is an example of that:
## dummy data
set.seed(1)
dat <- data.frame(a = factor(sample(letters[1:3], 10, replace = TRUE)),
b = runif(10))
## sapply over `dat`, converting factor to numeric
dat2 <- sapply(dat, function(x) if(is.factor(x)) {
as.numeric(x)
} else {
x
})
dat2 <- data.frame(dat2) ## convert to a data frame
Which gives:
> str(dat)
'data.frame': 10 obs. of 2 variables:
$ a: Factor w/ 3 levels "a","b","c": 1 2 2 3 1 3 3 2 2 1
$ b: num 0.206 0.177 0.687 0.384 0.77 ...
> str(dat2)
'data.frame': 10 obs. of 2 variables:
$ a: num 1 2 2 3 1 3 3 2 2 1
$ b: num 0.206 0.177 0.687 0.384 0.77 ...
However, do note that the above will work only if you want the underlying numeric representation. If your factor has essentially numeric levels, then we need to be a bit cleverer in how we convert the factor to a numeric whilst preserving the "numeric" information coded in the levels. Here is an example:
## dummy data
set.seed(1)
dat3 <- data.frame(a = factor(sample(1:3, 10, replace = TRUE), levels = 3:1),
b = runif(10))
## sapply over `dat3`, converting factor to numeric
dat4 <- sapply(dat3, function(x) if(is.factor(x)) {
as.numeric(as.character(x))
} else {
x
})
dat4 <- data.frame(dat4) ## convert to a data frame
Note how we need to do as.character(x) first before we do as.numeric(). The extra call encodes the level information before we convert that to numeric. To see why this matters, note what dat3$a is
> dat3$a
[1] 1 2 2 3 1 3 3 2 2 1
Levels: 3 2 1
If we just convert that to numeric, we get the wrong data as R converts the underlying level codes
> as.numeric(dat3$a)
[1] 3 2 2 1 3 1 1 2 2 3
If we coerce the factor to a character vector first, then to a numeric one, we preserve the original information not R's internal representation
> as.numeric(as.character(dat3$a))
[1] 1 2 2 3 1 3 3 2 2 1
If your data are like this second example, then you can't use the simple data.matrix() trick as that is the same as applying as.numeric() directly to the factor and as this second example shows, that doesn't preserve the original information.
I know this is an older question, but I just had the same problem and may be it helps:
In this case, your score column seems like it should not have become a factor column. That usually happens after read.table when it is a text column. Depending on which country you are from, may be you separate floats with a "," and not with a ".". Then R thinks that is a character column and makes it a factor. AND in that case Gavins answer won't work, because R won't make "123,456" to 123.456 . You can easily fix that in a text editor with replace "," with "." though.
I'm trying to do chi square analysis for all combinations of variables in the data and my code is:
Data <- esoph[ , 1:3]
OldStatistic <- NA
for(i in 1:(ncol(Data)-1)){
for(j in (i+1):ncol(Data)){
Statistic <- data.frame("Row"=colnames(Data)[i], "Column"=colnames(Data)[j],
"Chi.Square"=round(chisq.test(Data[ ,i], Data[ ,j])$statistic, 3),
"df"=chisq.test(Data[ ,i], Data[ ,j])$parameter,
"p.value"=round(chisq.test(Data[ ,i], Data[ ,j])$p.value, 3),
row.names=NULL)
temp <- rbind(OldStatistic, Statistic)
OldStatistic <- Statistic
Statistic <- temp
}
}
str(Data)
'data.frame': 88 obs. of 3 variables:
$ agegp: Ord.factor w/ 6 levels "25-34"<"35-44"<..: 1 1 1 1 1 1 1 1 1 1 ...
$ alcgp: Ord.factor w/ 4 levels "0-39g/day"<"40-79"<..: 1 1 1 1 2 2 2 2 3 3 ...
$ tobgp: Ord.factor w/ 4 levels "0-9g/day"<"10-19"<..: 1 2 3 4 1 2 3 4 1 2 ...
Statistic
Row Column Chi.Square df p.value
1 agegp tobgp 2.400 15 1
2 alcgp tobgp 0.619 9 1
My code gives my the chi square analysis output for variable 1 vs variable 3, and variable 2 vs variable 3 and is missing for variable 1 vs variable 2. I tried hard but could not fixed the code. Any comment and suggestion will be highly appreciated. I'd like like to do cross tabulation for all possible combinations. Thanks in advance.
EDIT
I used to do this kind of analysis in SPSS but now I want to switch to R.
A sample of your data would be appreciated, but I think this will work for you. First, create a combination of all columns with combn. Then write a function to use with an apply function to iterate through the combos. I like to use plyr since it is easy to specify what you want for a data structure on the back end. Also note you only need to compute the chi square test once for each combination of columns, which should speed things up quite a bit as well.
library(plyr)
combos <- combn(ncol(Dat),2)
adply(combos, 2, function(x) {
test <- chisq.test(Dat[, x[1]], Dat[, x[2]])
out <- data.frame("Row" = colnames(Dat)[x[1]]
, "Column" = colnames(Dat[x[2]])
, "Chi.Square" = round(test$statistic,3)
, "df"= test$parameter
, "p.value" = round(test$p.value, 3)
)
return(out)
})
I wrote my own function. It creates a matrix where all nominal variables are tested against each other. It can also save the results as excel file. It displays all the pvalues that are smaller than 5%.
funMassChi <- function (x,delFirst=0,xlsxpath=FALSE) {
options(scipen = 999)
start <- (delFirst+1)
ds <- x[,start:ncol(x)]
cATeND <- ncol(ds)
catID <- 1:cATeND
resMat <- ds[1:cATeND,1:(cATeND-1)]
resMat[,] <- NA
for(nCc in 1:(length(catID)-1)){
for(nDc in (nCc+1):length(catID)){
tryCatch({
chiRes <- chisq.test(ds[,catID[nCc]],ds[,catID[nDc]])
resMat[nDc,nCc]<- chiRes[[3]]
}, error=function(e){cat(paste("ERROR :","at",nCc,nDc, sep=" "),conditionMessage(e), "\n")})
}
}
resMat[resMat > 0.05] <- ""
Ergebnis <- cbind(CatNames=names(ds),resMat)
Ergebnis <<- Ergebnis[-1,]
if (!(xlsxpath==FALSE)) {
write.xlsx(x = Ergebnis, file = paste(xlsxpath,"ALLChi-",Sys.Date(),".xlsx",sep=""),
sheetName = "Tabelle1", row.names = FALSE)
}
}
funMassChi(categorialDATA,delFirst=3,xlsxpath="C:/folder1/folder2/")
delFirst can delete the first n columns. So if you have an count index or something you dont want to test.
I hope this can help anyone else.
Recoding is a common practice for survey data, but the most obvious routes take more time than they should.
The fastest code that accomplishes the same task with the provided sample data by system.time() on my machine wins.
## Sample data
dat <- cbind(rep(1:5,50000),rep(5:1,50000),rep(c(1,2,4,5,3),50000))
dat <- cbind(dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat)
dat <- as.data.frame(dat)
re.codes <- c("This","That","And","The","Other")
Code to optimize.
for(x in 1:ncol(dat)) {
dat[,x] <- factor(dat[,x], labels=re.codes)
}
Current system.time():
user system elapsed
4.40 0.10 4.49
Hint: dat <- lapply(1:ncol(dat), function(x) dat[,x] <- factor(dat[,x],labels=rc))) is not any faster.
Combining #DWin's answer, and my answer from Most efficient list to data.frame method?:
system.time({
dat3 <- list()
# define attributes once outside of loop
attrib <- list(class="factor", levels=re.codes)
for (i in names(dat)) { # loop over each column in 'dat'
dat3[[i]] <- as.integer(dat[[i]]) # convert column to integer
attributes(dat3[[i]]) <- attrib # assign factor attributes
}
# convert 'dat3' into a data.frame. We can do it like this because:
# 1) we know 'dat' and 'dat3' have the same number of rows and columns
# 2) we want 'dat3' to have the same colnames as 'dat'
# 3) we don't care if 'dat3' has different rownames than 'dat'
attributes(dat3) <- list(row.names=c(NA_integer_,nrow(dat)),
class="data.frame", names=names(dat))
})
identical(dat2, dat3) # 'dat2' is from #Dwin's answer
My computer is obviously much slower, but structure is a pretty fast way to do this:
> system.time({
+ dat1 <- dat
+ for(x in 1:ncol(dat)) {
+ dat1[,x] <- factor(dat1[,x], labels=re.codes)
+ }
+ })
user system elapsed
11.965 3.172 15.164
>
> system.time({
+ m <- as.matrix(dat)
+ dat2 <- data.frame( matrix( re.codes[m], nrow = nrow(m)))
+ })
user system elapsed
2.100 0.516 2.621
>
> system.time(dat3 <- data.frame(lapply(dat, structure, class='factor', levels=re.codes)))
user system elapsed
0.484 0.332 0.820
# this isn't because the levels get re-ordered
> all.equal(dat1, dat2)
> all.equal(dat1, dat3)
[1] TRUE
Try this:
m <- as.matrix(dat)
dat <- data.frame( matrix( re.codes[m], nrow = nrow(m)))
A data.table answer for your consideration. We're just using setattr() from it, which works on data.frame, and columns of data.frame. No need to convert to data.table.
The test data again :
dat <- cbind(rep(1:5,50000),rep(5:1,50000),rep(c(1L,2L,4L,5L,3L),50000))
dat <- cbind(dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat)
dat <- as.data.frame(dat)
re.codes <- c("This","That","And","The","Other")
Now change the class and set the levels of each column directly, by reference :
require(data.table)
system.time(for (i in 1:ncol(dat)) {
setattr(dat[[i]],"levels",re.codes)
setattr(dat[[i]],"class","factor")
}
# user system elapsed
# 0 0 0
identical(dat, <result in question>)
# [1] TRUE
Does 0.00 win? As you increase the size of the data, this method stays at 0.00.
Ok, I admit, I changed the input data slightly to be integer for all columns (the question has double input data in a third of the columns). Those double columns have to be converted to integer because factor is only valid for integer vectors. As mentioned in the other answers.
So, strictly with the input data in the question, and including the double to integer conversion :
dat <- cbind(rep(1:5,50000),rep(5:1,50000),rep(c(1,2,4,5,3),50000))
dat <- cbind(dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat,dat)
dat <- as.data.frame(dat)
re.codes <- c("This","That","And","The","Other")
system.time(for (i in 1:ncol(dat)) {
if (!is.integer(dat[[i]]))
set(dat,j=i,value=as.integer(dat[[i]]))
setattr(dat[[i]],"levels",re.codes)
setattr(dat[[i]],"class","factor")
})
# user system elapsed
# 0.06 0.01 0.08 # on my slow netbook
identical(dat, <result in question>)
# [1] TRUE
Note that set also works on data.frame, too. You don't have to convert to data.table to use it.
These are very small times, clearly. Since it's only a small input dataset :
dim(dat)
# [1] 250000 36
object.size(dat)
# 68.7 Mb
Scaling up from this should reveal larger differences. But even so I think it should be (just about) measurably fastest. Not a significant difference that anyone minds about, at this size, though.
The setattr function is also in the bit package, btw. So the 0.00 method can be done with either data.table or bit. To do the type conversion by reference (if required) either set or := (both in data.table) is needed, afaik.
The help page for class() says that class<- is deprecated and to use as. methods. I haven't quite figured out why the earlier effort was reporting 0 observations when the data was obviously in the object, but this method results in a complete object:
system.time({ dat2 <- vector(mode="list", length(dat))
for (i in 1:length(dat) ){ dat2[[i]] <- dat[[i]]
storage.mode(dat2[[i]]) <- "integer"
attributes(dat2[[i]]) <- list(class="factor", levels=re.codes)}
names(dat2) <- names(dat)
dat2 <- as.data.frame(dat2)})
#--------------------------
user system elapsed
0.266 0.290 0.560
> str(dat2)
'data.frame': 250000 obs. of 36 variables:
$ V1 : Factor w/ 5 levels "This","That",..: 1 2 3 4 5 1 2 3 4 5 ...
$ V2 : Factor w/ 5 levels "This","That",..: 5 4 3 2 1 5 4 3 2 1 ...
$ V3 : Factor w/ 5 levels "This","That",..: 1 2 4 5 3 1 2 4 5 3 ...
$ V4 : Factor w/ 5 levels "This","That",..: 1 2 3 4 5 1 2 3 4 5 ...
$ V5 : Factor w/ 5 levels "This","That",..: 5 4 3 2 1 5 4 3 2 1 ...
$ V6 : Factor w/ 5 levels "This","That",..: 1 2 4 5 3 1 2 4 5 3 ...
$ V7 : Factor w/ 5 levels "This","That",..: 1 2 3 4 5 1 2 3 4 5 ...
$ V8 : Factor w/ 5 levels "This","That",..: 5 4 3 2 1 5 4 3 2 1 ...
snipped
All 36 columns are there.
Making factors is expensive; only doing it once is comparable with the commands using structure, and in my opinion, preferable as you don't have to depend on how factors happen to be constructed.
rc <- factor(re.codes, levels=re.codes)
dat5 <- as.data.frame(lapply(dat, function(d) rc[d]))
EDIT 2: Interestingly, this seems to be a case where lapply does speed things up. This for loop is substantially slower.
for(i in seq_along(dat)) {
dat[[i]] <- rc[dat[[i]]]
}
EDIT 1: You can also speed things up by being more precise with your types. Try any of the solutions (but especially your original one) creating your data as integers, as follows. For details, see a previous answer of mine here.
dat <- cbind(rep(1:5,50000),rep(5:1,50000),rep(c(1L,2L,4L,5L,3L),50000))
This is also a good idea as converting to integers from floating points, as is being done in all of the faster solutions here, can give unexpected behavior, see this question.