I am trying to convert missing factor values to NA in a data frame, and create a new data frame with replaced values but when I try to do that, previously character factors are all converted to numbers. I cannot figure out what I am doing wrong and cannot find a similar question. Could anybody please help?
Here are my codes:
orders <- c('One','Two','Three', '')
ids <- c(1, 2, 3, 4)
values <- c(1.5, 100.6, 19.3, '')
df <- data.frame(orders, ids, values)
new.df <- as.data.frame(matrix( , ncol = ncol(df), nrow = 0))
names(new.df) <- names(df)
for(i in 1:nrow(df)){
row.df <- df[i, ]
print(row.df$orders) # "One", "Two", "Three", ""
print(str(row.df$orders)) # Factor
# Want to replace "orders" value in each row with NA if it is missing
row.df$orders <- ifelse(row.df$orders == "", NA, row.df$orders)
print(row.df$orders) # Converted to number
print(str(row.df$orders)) # int or logi
# Add the row with new value to the new data frame
new.df[nrow(new.df) + 1, ] <- row.df
}
and I get this:
> new.df
orders ids values
1 2 1 2
2 4 2 3
3 3 3 4
4 NA 4 1
but I want this:
> new.df
orders ids values
1 One 1 1.5
2 Two 2 100.6
3 Three 3 19.3
4 NA 4
Convert empty values to NA and use type.convert to change their class.
df[df == ''] <- NA
df <- type.convert(df)
df
# orders ids values
#1 One 1 1.5
#2 Two 2 100.6
#3 Three 3 19.3
#4 <NA> 4 NA
str(df)
#'data.frame': 4 obs. of 3 variables:
#$ orders: Factor w/ 4 levels "","One","Three",..: 2 4 3 1
#$ ids : int 1 2 3 4
#$ values: num 1.5 100.6 19.3 NA
Thanks to the hint from Ronak Shah, I did this and it gave me what I wanted.
df$orders[df$orders == ''] <- NA
This will give me:
> df
orders ids values
1 One 1 1.5
2 Two 2 100.6
3 Three 3 19.3
4 <NA> 4
> str(df)
'data.frame': 4 obs. of 3 variables:
$ orders: Factor w/ 4 levels "","One","Three",..: 2 4 3 NA
$ ids : num 1 2 3 4
$ values: Factor w/ 4 levels "","1.5","100.6",..: 2 3 4 1
In case you are curious about the difference between NA and as I was, you can find the answer here.
Your suggestion
df$orders[is.na(df$orders)] <- NA
did not work maybe becasuse missing entry is not NA?
Related
I am using R for a research project that requires me to input a sequence of 1-5 of varying length and then calculate a score from that sequence.
The data frame I have stores the sequences as a factor. If I take a single entry and convert it to a numeric vector, I can input it into the formula. But if I try to do this for all rows I run into errors.
I have searched SO and other sources but only found information on how to convert factors to numeric if they contain one value per cell. My data contains a sequence of numbers per cell separated by commas.
If I take input from one cell and use as.numeric(strsplit(as.character it works. But I don't want to do all cells manually. How can I solve this?
This is what I did:
df <- read.csv2("example_seq_logs.csv", na.strings = "n/a")
df$seqtext <- as.character(df$hmm)
This is what the data frame looks like:
head(df)
lesson hmm
1 A 1,2,3,3,3,4,3,4,5,4,4,5,5,2,2,1,2,3,4,2,3
2 B 2,2,3,4,1,1,3,3,3,5,5,4,4,4,2,1
3 C 1,3,1,3,2,3,2,2,3,3,4,1,3,2,3,3,5,4,4,3,3
4 D 1,3,2,2,3,3,2,3,1,4,4,5,5,2,4,4,4,3
5 E 1,4,2,5,1,3,1,3,1,4,3,4,4
str(df)
'data.frame': 5 obs. of 2 variables:
$ lesson: Factor w/ 5 levels "A","B","C","D",..: 1 2 3 4 5
$ hmm : Factor w/ 5 levels "1,2,3,3,3,4,3,4,5,4,4,5,5,2,2,1,2,3,4,2,3",..: 1 5 2 3 4
sapply(df, mode)
lesson hmm
"numeric" "numeric"
Now if I take a single entry I can do this:
testseq <- as.numeric(strsplit(df$seqtext)[1],",")[[1]])
str(testseq)
num [1:21] 1 2 3 3 3 4 3 4 5 4 ...
and then I can input the testseq sequence into the function I need.
But when I try the same for the whole column it results in an error
df$seq <- as.numeric(strsplit(df$seqtext, ","))[[1:58]]
Error: (list) object cannot be coerced to type 'double'
Thank you for your help!
Edit:
The first suggestion yields this error:
df$seq <- as.numeric(unlist(strsplit(paste(df$seqtext, collapse = ","), ",")))
Error in `$<-.data.frame`(`*tmp*`, seq, value = c(1, 2, 3, 3, 3, 4, 3, :
replacement has 89 rows, data has 5
It seems it turns the entire column into one long string.
a <- as.numeric(unlist(strsplit(paste(df$seqtext, collapse = ","), ",")))
print(a)
[1] 1 2 3 3 3 4 3 4 5 4 4 5 5 2 2 1 2 3 4 2 3 2 2 3 4 1 1 3 3 3 5 5 4 4 4 2 1 1 3 1 3 2 3 2 2 3 3 4 1 3 2 3
[53] 3 5 4 4 3 3 1 3 2 2 3 3 2 3 1 4 4 5 5 2 4 4 4 3 1 4 2 5 1 3 1 3 1 4 3 4 4
But I need each sequence to turn up in the right row as a string.
Edit:
I found that the function I need to calculate results with doesn't need numerics so now I've solved the issue using a for loop:
df$score <- 0
for (i in 1:nrow(df)) {
seq <- as.array(strsplit(as.character(df$hmm),","))
session_seq <- seq[i]
res = computehmm(session_seq)
df$score[i] <- res$score
}
But now it stops calculating once it reaches an empty df$hmm field.
I understand sapply would be better but I don't understand how to get it to work.
You can use paste as:
as.numeric(unlist(strsplit(paste(df$seqtext, collapse = ","), ",")))
I have a data frame like this:
> str(dynamics)
'data.frame': 3517 obs. of 3 variables:
$ id : int 1 2 3 4 5 6 7 8 9 10 ...
$ y2015: int 245 129 301 162 123 125 115 47 46 135 ...
$ y2016: int NA 385 420 205 215 295 130 NA NA 380 ...
I take out the 3 vectors and name them differently,
Column 1:
> plantid <- dynamics$id
> head(plantid)
[1] 1 2 3 4 5 6
Column 2:
(I divide it into different classes and label them 2,3,4 and 5)
> y15 <- dynamics$y2015
> year15 <- cut(y15, breaks = c(-Inf, 50, 100, 150, Inf), labels = c("2", "3", "4", "5"))
> str(year15)
Factor w/ 4 levels "2","3","4","5": 4 3 4 4 3 3 3 1 1 3 ...
> head(year15)
[1] 5 4 5 5 4 4
Levels: 2 3 4 5
Column 3:
(Same here)
> y16 <- dynamics$y2016
> year16 <- cut(y16, breaks = c(-Inf, 50, 100, 150, Inf), labels = c("2", "3", "4", "5"))
> str(year16)
Factor w/ 4 levels "2","3","4","5": NA 4 4 4 4 4 3 NA NA 4 ...
> head(year16)
[1] <NA> 5 5 5 5 5
Levels: 2 3 4 5
So far so good!
The problem arises when I combine the above 3 vectors by cbind() to form a new data frame, the newly created vector levels are gone
Look at my code:
SD1 = data.frame(cbind(plantid, year15, year16))
head(SD1)
and I get a data frame like this:
> head(SD1)
plantid year15 year16
1 1 4 NA
2 2 3 4
3 3 4 4
4 4 4 4
5 5 3 4
6 6 3 4
as you can see the levels of 2nd and 3rd column have changed from 2, 3, 4, 5 back to 1, 2, 3, 4
How do I fix that?
cbind is most commonly used to combine objects into matrices. It strips out special attributes from the inputs to help ensure that they are compatible for combining into a single object. This means that data types with special attributes (such as the name and format attributes for factors and Dates) will be simplified to their underlying numerical representations. This is why cbind turns your factors into numbers.
Conversely, data.frame() by itself will preserve the individual object attributes. In this case, your use of cbind is unnecessary. To preserve your factor levels, simply use:
SD1 <- data.frame(plantid, year15, year16)
I am looking to create a function that will convert any factor variable with more than 4 levels into a dummy variable. The dataset has ~2311 columns, so I would really need to create a function. Your help would be immensely appreciated.
I have compiled the code below and was hoping to get it to work.
library(dummies)
# example function
for(i in names(Final_Dataset)){
if(count (Final_Dataset[i])>4){
y <- Final_Dataset[i]
Final_Dataset <- cbind(Final_Dataset, dummy(y, sep = "_"))
}
}
I was also considering an alternative approach where I would get all the number of columns that need to be dummied and then loop through all the columns and if the column number is in that array then create dummy variables out of the variable.
Example data
fct = data.frame(a = as.factor(letters[1:10]), b = 1:10, c = as.factor(sample(letters[1:4], 10, replace = T)), d = as.factor(letters[10:19]))
str(fct)
'data.frame': 10 obs. of 4 variables:
$ a: Factor w/ 10 levels "a","b","c","d",..: 1 2 3 4 5 6 7 8 9 10
$ b: int 1 2 3 4 5 6 7 8 9 10
$ c: Factor w/ 4 levels "a","b","c","d": 2 4 1 3 1 1 2 3 1 2
$ d: Factor w/ 10 levels "j","k","l","m",..: 1 2 3 4 5 6 7 8 9 10
# keep columns with more than 4 factors
fact_cols = sapply(fct, function(x) is.factor(x) && length(levels(x)) > 4)
# create dummy variables for subset (omit intercept)
dummy_cols = model.matrix(~. -1, fct[, fact_cols])
# cbind new data
out_df = cbind(fct[, !fact_cols], dummy_cols)
You could get all the columns with more than a given number of levels (n = 4) with something like
which(sapply(Final_Dataset, function (c) length(levels(c)) > n))
My dataset is pretty big. I have about 2,000 variables and 1,000 observations.
I want to run a model for each variable using other variables.
To do so, I need to drop variables which have missing values where the dependent variable doesn't have.
I meant that for instance, for variable "A" I need to drop variable C and D because those have missing values where variable A doesn't have. for variable "C" I can keep variable "D".
data <- read.table(text="
A B C D
1 3 9 4
2 1 3 4
NA NA 3 5
4 2 NA NA
2 5 4 3
1 1 1 2",header=T,sep="")
I think I need to make a loop to go through each variable.
I think this gets what you need:
for (i in 1:ncol(data)) {
# filter out rows with NA's in on column 'i'
# which is the column we currently care about
tmp <- data[!is.na(data[,i]),]
# now column 'i' has no NA values, so remove other columns
# that have NAs in them from the data frame
tmp <- tmp[sapply(tmp, function(x) !any(is.na(x)))]
#run your model on 'tmp'
}
For each iteration of i, the tmp data frame looks like:
'data.frame': 5 obs. of 2 variables:
$ A: int 1 2 4 2 1
$ B: int 3 1 2 5 1
'data.frame': 5 obs. of 2 variables:
$ A: int 1 2 4 2 1
$ B: int 3 1 2 5 1
'data.frame': 4 obs. of 2 variables:
$ C: int 3 3 4 1
$ D: int 4 5 3 2
'data.frame': 5 obs. of 1 variable:
$ D: int 4 4 5 3 2
I'll provide a way to get the usable vadiables for each column you choose:
getVars <- function(data, col){
tmp<-!sapply(data[!is.na(data[[col]]),], function(x) { any(is.na(x)) })
names(data)[tmp & names(data) != col]
}
PS: I'm on my phone so I didn't test the above nor had the chance for a good code styling.
EDIT: Styling fixed!
I'm looking for a way to remove rows in a data frame with less than 3 observations. Let me explain the matter in a better way.
I have a dataframe with 6 indipendent variables and 1 dependent. As I'm doing a density plot in ggplot2 using faceting, variables with less than 3 observations are not plotted (obviously). I'm looking for a way to delete these rows with less than 3 observations. this is an example of the data:
'data.frame': 432 obs. of 6 variables:
$ ID : Factor w/ 439 levels "1","2","3","4",..: 1 2 3 4 5 6 7 8 9 10 ...
$ Forno : Factor w/ 8 levels "Micro","Macro",..: 1 1 1 6 6 6 4 4 4 5 ...
$ Varieta: Factor w/ 11 levels "cc","dd",..: 11 11 11 6 6 6 1 1 1 6 ...
$ Impiego: Factor w/ 5 levels "aperto","chiuso",..: 2 2 2 3 3 3 2 2 2 5 ...
$ MediaL : num 60.7 58.9 60.5 55.9 56.1 ...
$ MediaL.sd : num 4.81 4.79 4.84 5.27 5.64 ...
ggplot code:
ggplot(d1,aes(MediaL))+geom_density(aes(fill=Varieta),colour=NA,alpha=0.5)+
scale_fill_brewer(palette="Set1")+facet_grid(Forno~Impiego)+
theme(axis.text.x=element_text(angle=90,hjust=1))+theme_mio +xlim(45,65)+
stat_bin(geom="text",aes(y=0,label=..count..),size=2,binwidth=2)
I would like to remove all the interactions with less than 3 observations.
Providing the actual output of your sample data would be useful. You can provide this via dput(yourObject) instead of the text representation you provided. However, it does seem like the same basic approach below works equally well with a matrix, data.frame, and table data structure.
#Matrix
x <- matrix(c(5,4,4,3,1,5,1,8,2), ncol = 3, byrow = TRUE)
x[x < 3] <- NA
#----
[,1] [,2] [,3]
[1,] 5 4 4
[2,] 3 NA 5
[3,] NA 8 NA
#data.frame
xd <- as.data.frame(matrix(c(5,4,4,3,1,5,1,8,2), ncol = 3, byrow = TRUE))
xd[xd < 3] <- NA
#----
V1 V2 V3
1 5 4 4
2 3 NA 5
3 NA 8 NA
#Table. Simulate some data first
set.seed(1)
samp <- data.frame(x1 = sample(c("acqua", "fango", "neve"), 20, TRUE),
x2 = sample(c("pippo", "pluto", "paperino"), 20, TRUE))
x2 <-table(samp)
x2[x2 < 3] <- NA
#----
x2
x1 paperino pippo pluto
acqua 3
fango 3
neve 3 3
ggplot generally likes data to be in long format, most often achieved via the melt() command in reshape2. If you provide your plotting code, that may illustrate a better way to remove the data you don't want to plot.