I have a table that look like this
A B C
AB ABC CBS
AB ABC
ADS
BBB
A want to use the columns as a character so is used this
A= as.character(table$A)
this results in c(“AB”, “AB”, “”) my goal was c(“AB”, “AB”), so without the empty cell "". To get wit of the empty cell I used this A=A[!A==""] which gives the results I want, but there must be a more elegant way of accomplishing the same goal.
May questions are 1) is there a better way of removing empty characters/cells.
Or more general 2) is there a way to transform the 3 columns (A,B,C) into characters A, B, C without the empty cells.
Thanks
'data.frame': 3 obs. of 3 variables:
$ A: Factor w/ 2 levels "","AB": 2 2 1
$ B: Factor w/ 3 levels "","ABC","ADS": 2 1 3
$ C: Factor w/ 3 levels "ABC","BBB","CBS": 3 1 2
Try specifying the argument na.strings during data import. Also, instead of using read.csv(), you could write read.csv2() which uses sep = ";" by default.
# Import data
data <- read.csv2("/path/to/data.csv", header = TRUE,
na.strings = "", stringsAsFactors = FALSE)
str(data)
'data.frame': 4 obs. of 3 variables:
$ A: chr "AB" "AB" NA NA
$ B: chr "ABC" NA "ADS" NA
$ C: chr "CBS" "ABC" NA "BBB"
# Exclude NAs
as.character(na.exclude(data$A))
[1] "AB" "AB"
If you prefer not to read your data set again, you can use:
# not in ('') or ("")
A <- table$A[!table$A %in% '']
Related
Recently I have come across a problem where my data has been converted to factors.
This is a large nuisance, as it's not (always) easily picked up on.
I am aware that I can convert them back with solutions such as as.character(paste(x)) or as.character(paste(x)), but that seems really unnecessary.
Example code:
nums <- c(1,2,3,4,5)
chars <- c("A","B","C,","D","E")
str(nums)
#> num [1:5] 1 2 3 4 5
str(chars)
#> chr [1:5] "A" "B" "C," "D" "E"
df <- as.data.frame(cbind(a = nums, b = chars))
str(df)
#> 'data.frame': 5 obs. of 2 variables:
#> $ a: Factor w/ 5 levels "1","2","3","4",..: 1 2 3 4 5
#> $ b: Factor w/ 5 levels "A","B","C,","D",..: 1 2 3 4 5
Don't cbind as it converts data to matrix and matrix can hold data of only one type, so it converts numbers to characters.
Use data.frame because as.data.frame(a = nums, b = chars) returns an error.
Use stringsAsFactors = FALSE because in data.frame default value of
stringsAsFactors is TRUE which converts characters to factors. The numbers also change to factors because in 1) they have been changed to characters.
df <- data.frame(a = nums, b = chars, stringsAsFactors = FALSE)
str(df)
#'data.frame': 5 obs. of 2 variables:
# $ a: num 1 2 3 4 5
# $ b: chr "A" "B" "C," "D" ...
EDIT: As of the newest version of R, the default value of stringAsFactors has changed to FALSE.
This should no longer happen if you have updated R: data frames don't automatically turn chr to fct. In a way, data frames are now more similar to tibbles.
After reading in data and cleaning it, I ended up with factor columns that have levels that should no longer be there.
For example, d below has one blank cell in excel. When it’s read in, the factor columns have a level "", which shouldn’t be part of the data.
d <- read.csv(header = TRUE, text='
x,y,value
a,one,1
,,5
b,two,4
c,three,10
')
d
#> x y value
#> 1 a one 1
#> 2 5
#> 3 b two 4
#> 4 c three 10
str(d)
#> 'data.frame': 4 obs. of 3 variables:
#> $ x : Factor w/ 4 levels "","a","b","c": 2 1 3 4
#> $ y : Factor w/ 4 levels "","one","three",..: 2 1 4 3
#> $ value: int 1 5 4 10
How do I remove this level, "" from the factors which are about 20 factors in the data frame, without deleting the entire row that has just one empty cell, cause this will reduce my sample size from 299000 to just 7 observation(which I have tried before).
One way would be to replace the '' with NA and use droplevels to remove the unused levels
d[1:2] <- lapply(d[1:2], function(x) droplevels(replace(x, x=="", NA)))
levels(d$x)
#[1] "a" "b" "c"
levels(d$y)
#[1] "one" "three" "two"
Another option while reading the dataset (as we assume the OP wanted factor columns would be
d <- read.csv("yourfile.csv", na.strings = "")
This should make sure that the '' will be read as NA.
Update
Suppose, there are numeric columns in between and we need to do the replace/droplevels only for the factor columns
d[] <- lapply(d, function(x) if(is.factor(x)) droplevels(replace(x, x== "", NA))
else x)
E.g.
chr <- c("a", "b", "c")
intgr <- c(1, 2, 3)
str(chr)
str(base::merge(chr,intgr, stringsAsFactors = FALSE))
gives:
> str(base::merge(chr,intgr, stringsAsFactors = FALSE))
'data.frame': 9 obs. of 2 variables:
$ x: Factor w/ 3 levels "a","b","c": 1 2 3 1 2 3 1 2 3
$ y: num 1 1 1 2 2 2 3 3 3
I originally thought it has something to do with how merge coerces arguments into data frames. However, I thought that adding the argument stringsAsFactors = FALSE would override the default coercion behaviour of char -> factor, yet this is not working.
EDIT: Doing the following gives me expected behaviour:
options(stringsAsFactors = FALSE)
str(base::merge(chr,intgr))
that is:
> str(base::merge(chr,intgr))
'data.frame': 9 obs. of 2 variables:
$ x: chr "a" "b" "c" "a" ...
$ y: num 1 1 1 2 2 2 3 3 3
but this is not ideal as it changes the global stringsAsFactors setting.
You can accomplish this particular "merge" using expand.grid(), since you're really just taking the cartesian product. This allows you to pass the stringsAsFactors argument:
sapply(expand.grid(x=chr,y=intgr,stringsAsFactors=F),class);
## x y
## "character" "numeric"
Here's a way of working around this limitation of merge():
sapply(merge(data.frame(x=chr,stringsAsFactors=F),intgr),class);
## x y
## "character" "numeric"
I would argue that it never makes sense to pass an atomic vector to merge(), since it is only really designed for merging data.frames.
We can use CJ from data.table as welll
library(data.table)
str(CJ(chr, intgr))
Classes ‘data.table’ and 'data.frame': 9 obs. of 2 variables:
#$ V1: chr "a" "a" "a" "b" ...
#$ V2: num 1 2 3 1 2 3 1 2 3
I have a data frame which has one column and column has some data and some empty cells.
When I am checking the levels of that column it is showing three levels as it is taking empty cells as one level. I want to delete that level.
suppose I have
## editor note: starting from R 4.0.0, `stringsAsFactors` defaults to FALSE
## we now explicitly need `stringsAsFactors = TRUE`
df <- data.frame(fan = c("a","b"," ","a","b"), stringsAsFactors = TRUE)
I have tried this code
droplevels(df)
but it is not working.
'droplevels' does work. No need for complex code:
df <- data.frame(fan = c("a","b"," ","a","b"))
df
# fan
#1 a
#2 b
#3
#4 a
#5 b
df$fan[df$fan==' ']=NA
df$fan = droplevels(df$fan)
str(df)
#'data.frame': 5 obs. of 1 variable:
# $ fan: Factor w/ 2 levels "a","b": 1 2 NA 1 2
When you read your file to R, you may avoid 'empty cell' being included as a factor level in the first place, by using the na.strings argument in read.csv (or in read.xxx). The na.strings argument defines "strings which are to be interpreted as NA values".
Here is an example where I read a text file (foo.csv) which I created from your 'df':
read.csv(file = "foo.csv", na.strings = " ")
# fan
# 1 a
# 2 b
# 3 <NA>
# 4 a
# 5 b
str(as.factor(df2$fan))
# Factor w/ 2 levels "a","b": 1 2 NA 1 2
When the file is read the empty fields are now treated as NA, and 'blank' is thus not included as a factor level.
From ?read.table: "Blank fields are [...] considered to be missing values in logical, integer, numeric and complex fields". However, in your data, the variable "fan" is a character. If you then have stringsAsFactors = TRUE in options or in read.xxx, the character vector is converted to a factor.
Try:
df$fan[grepl("^\\s*$", df$fan)] <- NA #in case you have c(" ", "", "a", "b", " ")
Explanation
^(|\\s+)$- matches if there is an empty quote'' or spaces within quotes(" ", " ", " "). Hence, more general.
str(droplevels(df))
#'data.frame': 5 obs. of 1 variable:
#$ fan: Factor w/ 2 levels "a","b": 1 2 NA 1 2
If you want to create a new dataset with the empty cells deleted
df1 <- droplevels(df[!grepl("^\\s*$", df$fan),,drop=FALSE] )
str(df1)
#'data.frame': 4 obs. of 1 variable:
#$ fan: Factor w/ 2 levels "a","b": 1 2 1 2
If you are using csv, this might help:
data<-read.csv(file = "data.csv", na.strings = "", stringsAsFactors = T)
I modified the prior response and added , stringsAsFactors = T
So, later it will not report NA in any subsequent analysis as in Createtableone
I am curious about the behaviour of transform. Two ways I might try creating a new column as character not as factor:
x <- data.frame(Letters = LETTERS[1:3], Numbers = 1:3)
y <- transform(x, Alphanumeric = as.character(paste(Letters, Numbers)))
x$Alphanumeric = with(x, as.character(paste(Letters, Numbers)))
x
y
str(x$Alphanumeric)
str(y$Alphanumeric)
The results "look" the same:
> x
Letters Numbers Alphanumeric
1 A 1 A 1
2 B 2 B 2
3 C 3 C 3
> y
Letters Numbers Alphanumeric
1 A 1 A 1
2 B 2 B 2
3 C 3 C 3
But look inside and only one has worked:
> str(x$Alphanumeric) # did convert to character
chr [1:3] "A 1" "B 2" "C 3"
> str(y$Alphanumeric) # but transform didn't
Factor w/ 3 levels "A 1","B 2","C 3": 1 2 3
I didn't find ?transform very useful to explain this behaviour - presumably Alphanumeric was coerced back to being a factor - or find a way to stop it (something like stringsAsFactors = FALSE for data.frame). What is the safest way to do this? Are there similar pitfalls to beware of, for instance with the apply or plyr functions?
This is not so much an issue with transform as much as it is with data.frames, where stringsAsFactors is set, by default, to TRUE. Add an argument that it should be FALSE and you'll be on your way:
y <- transform(x, Alphanumeric = paste(Letters, Numbers),
stringsAsFactors = FALSE)
str(y)
# 'data.frame': 3 obs. of 3 variables:
# $ Letters : Factor w/ 3 levels "A","B","C": 1 2 3
# $ Numbers : int 1 2 3
# $ Alphanumeric: chr "A 1" "B 2" "C 3"
I generally use within instead of transform, and it seems to not have this problem:
y <- within(x, {
Alphanumeric = paste(Letters, Numbers)
})
str(y)
# 'data.frame': 3 obs. of 3 variables:
# $ Letters : Factor w/ 3 levels "A","B","C": 1 2 3
# $ Numbers : int 1 2 3
# $ Alphanumeric: chr "A 1" "B 2" "C 3"
This is because it takes an approach similar to your with approach: Create a character vector and add it (via [<-) into the existing data.frame.
You can view the source of each of these by typing transform.data.frame and within.data.frame at the prompt.
As for other pitfalls, that's much too broad of a question. One thing that comes to mind right waya is that apply would create a matrix from a data.frame, so all the columns would be coerced to a single type.