Related
I am trying to convert a numeric vector with discreete values into a factor in R.
x <- c(1,2,3,4,8,9,10,88,89,90)
I need this vector to be converted into a factor variable with 4 levels as follows:
1,2 (level 1)
3,4 (level 2)
8,9,10 (level 3)
88,89, 90 (level 4)
I have tried using factor in R as follows:
y <- factor(x, levels = c(1:2, 3:4, 8:10, 88:90))
This returns a factor with 10 levels instead of a factor with 4 levels that I want.
str(y)
Factor w/ 10 levels "1","2","3","4",..: 1 2 3 4 5 6 7 8 9 10
I have also tried using cut as follows:
bins <- c(1,3,5,8,11,88,90)
y <- cut(x, breaks = bins, right = F, include.lowest = T)
table(y)
This also does not return the desired result as it creates a level for ranges such as [5-8) and [11-88) that I dont need.
y
[1,3) [3,5) [5,8) [8,11) [11,88) [88,90]
2 2 0 3 0 3
Is there a way to convert a range of numeric values into a factor in R?
Drop unused levels:
# as per your code
bins <- c(1,3,5,8,11,88,90)
y <- cut(x, breaks = bins, right = FALSE, include.lowest = TRUE)
levels(y)
# [1] "[1,3)" "[3,5)" "[5,8)" "[8,11)" "[11,88)" "[88,90]"
# drop unused levels
y1 <- droplevels(y)
levels(y1)
#[1] "[1,3)" "[3,5)" "[8,11)" "[88,90]"
We can use case_when
library(dplyr)
case_when(x %in% 1:2 ~ 1, x %in% 3:4 ~ 2, x %in% 8:10 ~ 3, x%in% 88:90 ~ 4)
Here are three vectors.
vec1 <- 1:6
vec2 <- c('radio', 'newspaper', 'web-page', 'chat', 'tv', 'web-page')
vec3 <- c(0, 0, 1, 1, 0, 1)
The task is to form a data frame with the following structure using these vectors.
'data.frame': 6 obs. of 3 variables:
$ id : int 1 2 3 4 5 6
$ response: Factor w/ 2 levels "No","Yes": 1 1 2 2 1 2
$ medium : chr "radio" "newspaper" "web-page" "chat" ...
Here is my solution.
dfr <- data.frame(id = vec1, response = vec3, medium = vec2, stringsAsFactors = FALSE)
dfr$response <- factor(x = , levels = , labels = )
My question is: "What values should the arguments (x, levels, labels) have and why?"
Talking about this line:
dfr$response <- factor(x = , levels = , labels = )
We can assign labels to vec3 as levels are by default taken from unique values of vec3.
df <- data.frame(id = vec1, response = factor(vec3, labels = c('No', 'Yes')),
medium = vec2, stringsAsFactors = FALSE)
str(df)
#'data.frame': 6 obs. of 3 variables:
#$ id : int 1 2 3 4 5 6
#$ response: Factor w/ 2 levels "No","Yes": 1 1 2 2 1 2
#$ medium : chr "radio" "newspaper" "web-page" "chat" ...
You can read ?factor for more details.
In this:
x is the vector of data that you want to turn into a factor, in this case the responses x=df$response
Levels is a vector of values that x might have taken. The default is a list of the distinct values of x, in ascending order (numeric or alphabetical), so the default would be c(0, 1). You don't need to include the levels, as it will automatically detect them, however as you're adding labels then it's good practice to add the levels so your labels match up in case you have lots of levels and manage to get the order mixed up.
Labels can either be a single string or a vector of all labels for the levels, you can use labels to map multiple values to the same Label. For your task you would use c("No", "Yes"). the default for Labels is the levels i.e. no label.
So your final code will be
dfr$response <- factor(x=dfr$response, levels=c(0,1), labels=c("No", "Yes"))
As a minor aside, people generally use df to represent a data frame, rather than dfr. It doesn't make any difference, but is just the commonly used notation.
dfOrig <- data.frame(rbind("1",
"C",
"531404",
"3",
"B",
"477644"))
setnames(dfOrig, "Value")
I have a single column vector, which actually comprises two observations of three variables. How do I convert it to a data.frame with the following structure:
ID Code Tag
"1" "C" "531404"
"3" "B" "477644"
Obviously, this is just a toy example to illustrate a real-world problem with many more observations and variables.
Here's another approach - it does rely on the dfOrig column being ordered 1,2,3,1,2,3 etc.
x <- c("ID", "Code", "Tag") # new column names
n <- length(x) # number of columns
res <- data.frame(lapply(split(as.character(dfOrig$Value), rep(x, nrow(dfOrig)/n)),
type.convert))
The resulting data is:
> str(res)
#'data.frame': 2 obs. of 3 variables:
# $ Code: Factor w/ 2 levels "B","C": 2 1
# $ ID : int 1 3
# $ Tag : int 531404 477644
As you can see, the column classes have been converted. In case you want the Code column to be character instead of factor you can specify stringsAsFactors = FALSE in the data.frame call.
And it looks like this:
> res
# Code ID Tag
#1 C 1 531404
#2 B 3 477644
Note: You have to get the column name order in x in line with the order of the entries in dfOrig$Value.
If you want to get the column order of res as specified in x, you can use the following:
res <- res[, match(x, names(res))]
Maybe convert to matrix with nrow:
# set number of columns
myNcol <- 3
# convert to matrix, then dataframe
res <- data.frame(matrix(dfOrig$Value, ncol = myNcol, byrow = TRUE),
stringsAsFactors = FALSE)
# convert the type and add column names
res <- as.data.frame(lapply(res, type.convert),
col.names = c("resID", "Code", "Tag"))
res
# resID Code Tag
# 1 1 C 531404
# 2 3 B 477644
You can create a sequence of numbers
x <- seq(1:nrow(dfOrig)) %% 3 #you can change this 3 to number of columns you need
data.frame(ID = dfOrig$Value[x == 1],
Code = dfOrig$Value[x == 2],
Tag = dfOrig$Value[x == 0])
#ID Code Tag
#1 1 C 531404
#2 3 B 477644
Another approach would be splitting the dataframe according to the sequence generated above and then binding the columns using do.call
x <- seq(1:nrow(dfOrig))%%3
res <- do.call("cbind", split(dfOrig,x))
You can definitely change the column names
colnames(res) <- c("Tag", "Id", "Code")
# Tag Id Code
#3 531404 1 C
#6 477644 3 B
In python, scikit has a great function called LabelEncoder that maps categorical levels (strings) to integer representation.
Is there anything in R to do this? For example if there is a variable called color with values {'Blue','Red','Green'} the encoder would translate:
Blue => 1
Green => 2
Red => 3
and create an object with this mapping to then use for transforming new data in a similar fashion.
Add:
It doesn't seem like just factors will work because there is no persisting of the mapping. If the new data has an unseen level from the training data, the entire structure changes. Ideally I would like the new levels labeled missing or 'other' somehow.
sample_dat <- data.frame(a_str=c('Red','Blue','Blue','Red','Green'))
sample_dat$a_int<-as.integer(as.factor(sample_dat$a_str))
sample_dat$a_int
#[1] 3 1 1 3 2
sample_dat2 <- data.frame(a_str=c('Red','Blue','Blue','Red','Green','Azure'))
sample_dat2$a_int<-as.integer(as.factor(sample_dat2$a_str))
sample_dat2$a_int
# [1] 4 2 2 4 3 1
Create your vector of data:
colors <- c("red", "red", "blue", "green")
Create a factor:
factors <- factor(colors)
Convert the factor to numbers:
as.numeric(factors)
Output: (note that this is in alphabetical order)
# [1] 3 3 1 2
You can also set a custom numbering system: (note that the output now follows the "rainbow color order" that I defined)
rainbow <- c("red","orange","yellow","green","blue","purple")
ordered <- factor(colors, levels = rainbow)
as.numeric(ordered)
# [1] 1 1 5 4
See ?factor.
Try CatEncoders package. It replicates the Python sklearn.preprocessing functionality.
# variable to encode values
colors = c("red", "red", "blue", "green")
lab_enc = LabelEncoder.fit(colors)
# new values are transformed to NA
values = transform(lab_enc, c('red', 'red', 'yellow'))
values
# [1] 3 3 NA
# doing the inverse: given the encoded numbers return the labels
inverse.transform(lab_enc, values)
# [1] "red" "red" NA
I would add the functionality of reporting the non-matching labels with a warning.
PS: It also has the OneHotEncoder function.
If I correctly understand what do you want:
# function which returns function which will encode vectors with values of 'vec'
label_encoder = function(vec){
levels = sort(unique(vec))
function(x){
match(x, levels)
}
}
colors = c("red", "red", "blue", "green")
color_encoder = label_encoder(colors) # create encoder
encoded_colors = color_encoder(colors) # encode colors
encoded_colors
new_colors = c("blue", "green", "green") # new vector
encoded_new_colors = color_encoder(new_colors)
encoded_new_colors
other_colors = c("blue", "green", "green", "yellow")
color_encoder(other_colors) # NA's are introduced
# save and restore to disk
saveRDS(color_encoder, "color_encoder.RDS")
c_encoder = readRDS("color_encoder.RDS")
c_encoder(colors) # same result
# dealing with multiple columns
# create data.frame
set.seed(123) # make result reproducible
color_dataframe = as.data.frame(
matrix(
sample(c("red", "blue", "green", "yellow"), 12, replace = TRUE),
ncol = 3)
)
color_dataframe
# encode each column
for (column in colnames(color_dataframe)){
color_dataframe[[column]] = color_encoder(color_dataframe[[column]])
}
color_dataframe
I wrote the following which I think works, the efficiency of which and/or how it will scale is not yet tested
str2Int.fit_transform<-function(df, plug_missing=TRUE){
list_of_levels=list() #empty list
#loop through the columns
for (i in 1: ncol(df))
{
#only
if (is.character(df[,i]) || is.factor(df[,i]) ){
#deal with missing
if(plug_missing){
#if factor
if (is.factor(df[,i])){
df[,i] = factor(df[,i], levels=c(levels(df[,i]), 'MISSING'))
df[,i][is.na(df[,i])] = 'MISSING'
}else{ #if character
df[,i][is.na(df[,i])] = 'MISSING'
}
}#end missing IF
levels<-unique(df[,i]) #distinct levels
list_of_levels[[colnames(df)[i]]] <- levels #set list with name of the columns to the levels
df[,i] <- as.numeric(factor(df[,i], levels = levels))
}#end if character/factor IF
}#end loop
return (list(list_of_levels,df)) #return the list of levels and the new DF
}#end of function
str2Int.transform<-function(df,list_of_levels,plug_missing=TRUE)
{
#loop through the columns
for (i in 1: ncol(df))
{
#only
if (is.character(df[,i]) || is.factor(df[,i]) ){
#deal with missing
if(plug_missing){
#if factor
if (is.factor(df[,i])){
df[,i] = factor(df[,i], levels=c(levels(df[,i]), 'MISSING'))
df[,i][is.na(df[,i])] = 'MISSING'
}else{ #if character
df[,i][is.na(df[,i])] = 'MISSING'
}
}#end missing IF
levels=list_of_levels[[colnames(df)[i]]]
if (! is.null(levels)){
df[,i] <- as.numeric(factor(df[,i], levels = levels))
}
}# character or factor
}#end of loop
return(df)
}#end of function
######################################################
# Test the functions
######################################################
###Test fit transform
# as strings
sample_dat <- data.frame(a_fact=c('Red','Blue','Blue',NA,'Green'), a_int=c(1,2,3,4,5), a_str=c('a','b','c','a','v'),stringsAsFactors=FALSE)
result<-str2Int.fit_transform(sample_dat)
result[[1]] #list of levels
result[[2]] #transformed df
#as factors
sample_dat <- data.frame(a_fact=c('Red','Blue','Blue',NA,'Green'), a_int=c(1,2,3,4,5), a_str=c('a','b','c','a','v'),stringsAsFactors=TRUE)
result<-str2Int.fit_transform(sample_dat)
result[[1]] #list of levels
result[[2]] #transformed df
###Test transform
str2Int.transform(sample_dat,result[[1]])
It's hard to believe why no one has mentioned caret's dummyVars function.
This is a widely searched question, and people don't want to write their own methods or copy and paste other users methods, they want a package, and caret is the closest thing to sklearn in R.
EDIT: I now realize that what the user actually want's is to turn strings into a counting number, which is just as.numeric(as.factor(x)) but I'm going to leave this here because using hot-one encoding is the more accurate method of encoding categorical data.
# input P to the function below is a dataframe containing only categorical variables
numlevel <- function(P) {
n <- dim(P)[2]
for(i in 1: n) {
m <- length(unique(P[[i]]))
levels(P[[i]]) <- c(1:m)
}
return(P)
}
Q <- numlevel(P)
df<- mtcars
head(df)
df$cyl <- factor(df$cyl)
df$carb <- factor(df$carb)
vec <- sapply(df, is.factor)
catlevels <- sapply(df[vec], levels)
#store the levels for each category
#level appearing first is coded as 1, second as 2 so on
df <- sapply(df, as.numeric)
class(df) #matrix
df <- data.frame(df)
#converting back to dataframe
head(df)
# Data
Country <- c("France", "Spain", "Germany", "Spain", "Germany", "France")
Age <- c(34, 27, 30, 32, 42, 30)
Purchased <- c("No", "Yes", "No", "No", "Yes", "Yes")
df <- data.frame(Country, Age, Purchased)
df
# Output
Country Age Purchased
1 France 34 No
2 Spain 27 Yes
3 Germany 30 No
4 Spain 32 No
5 Germany 42 Yes
6 France 30 Yes
Using CatEncoders package : Encoders for Categorical Variables
library(CatEncoders)
# Saving names of categorical variables
factors <- names(which(sapply(df, is.factor)))
# Label Encoder
for (i in factors){
encode <- LabelEncoder.fit(df[, i])
df[, i] <- transform(encode, df[, i])
}
df
# Output
Country Age Purchased
1 1 34 1
2 3 27 2
3 2 30 1
4 3 32 1
5 2 42 2
6 1 30 2
Using R base : factor function
# Label Encoder
levels <- c("France", "Spain", "Germany", "No", "Yes")
labels <- c(1, 2, 3, 1, 2)
for (i in factors){
df[, i] <- factor(df[, i], levels = levels, labels = labels, ordered = TRUE)
}
df
# Output
Country Age Purchased
1 1 34 1
2 2 27 2
3 3 30 1
4 2 32 1
5 3 42 2
6 1 30 2
Here is an easy and neat solution:
From the superml package:
https://www.rdocumentation.org/packages/superml/versions/0.5.3
There is a LabelEncoder class:
https://www.rdocumentation.org/packages/superml/versions/0.5.3/topics/LabelEncoder
install.packages("superml")
library(superml)
lbl <- LabelEncoder$new()
lbl$fit(sample_dat$column)
sample_dat$column <- lbl$fit_transform(sample_dat$column)
decode_names <- lbl$inverse_transform(sample_dat$column)
I'm having trouble assigning a dataframe to a subset of another. In the example below, the line
ds[cavities,] <- join(ds[cavities,1:4], fillings, by="ZipCode", "left")
only modifies one column instead of two. I would expect it either to modify no columns or both, not only one. I wrote the function to fill in the PrefName and CountyID columns in dataframe ds where they are NA by joining ds to another dataframe cs.
As you can see if you run it, the test is failing because PrefName is not getting filled in. After doing a bit of debugging, I realized that join() is doing exactly what it is expected to do, but the actual assignment of the result of that join somehow drops the PrefName back to a NA.
# fully copy-paste-run-able (but broken) code
suppressMessages({
library("plyr")
library("methods")
library("testthat")
})
# Fill in the missing PrefName/CountyIDs in delstat
# - Find the missing values in Delstat
# - Grab the CityState Primary Record values
# - Match on zipcode to fill in the holes in the delstat data
# - Remove any codes that could not be fixed
# - #param ds: delstat dataframe with 6 columns (see test case)
# - #param cs: citystate dataframe with 6 columns (see test case)
getMissingCounties <- function(ds, cs) {
if (length(is.na(ds$CountyID))) {
cavities <- which(is.na(ds$CountyID))
fillings <- cs[cs$PrimRec==TRUE, c(1,3,4)]
ds[cavities,] <- join(ds[cavities,1:4], fillings, by="ZipCode", "left")
ds <- ds[!is.na(ds$CountyID),]
}
return(ds)
}
test_getMissingCounties <- function() {
ds <- data.frame(
CityStateKey = c(1, 2, 3, 4 ),
ZipCode = c(11, 22, 33, 44 ),
Business = c(1, 1, 1, 1 ),
Residential = c(1, 1, 1, 1 ),
PrefName = c("One", NA , NA, NA),
CountyID = c(111, NA, NA, NA))
cs <- data.frame(
ZipCode = c(11, 22, 22, 33, 55 ),
Name = c("eh", "eh?", "eh?", "eh!?", "ah." ),
PrefName = c("One", "To", "Two", "Three", "Five"),
CountyID = c(111, 222, 222, 333, 555 ),
PrimRec = c(TRUE, FALSE, TRUE, TRUE, TRUE ),
CityStateKey = c(1, 2, 2, 3, 5 ))
expected <- data.frame(
CityStateKey = c(1, 2, 3 ),
ZipCode = c(11, 22, 33 ),
Business = c(1, 1, 1 ),
Residential = c(1, 1, 1 ),
PrefName = c("One", "Two", "Three"),
CountyID = c(111, 222, 333 ))
expect_equal(getMissingCounties(ds, cs), expected)
}
# run the test
test_getMissingCounties()
The results are:
CityStateKey ZipCode Business Residential PrefName CountyID
1 11 1 1 One 111
2 22 1 1 <NA> 222
3 33 1 1 <NA> 333
Any ideas why PrefName is getting set to NA by the assignment or how to do the assignment so I don't lose data?
The short answer is that you can avoid this problem by making sure that there are no factors in your data frames. You do this by using stringsAsFactors=FALSE in the call(s) to data.frame(...). Note that many of the data import functions, including read.table(...) and read.csv(...) also convert character to factor by default. You can defeat this behavior the same way.
This problem is actually quite subtle, and is also a good example of how R's "silent coercion" between data types creates all sorts of problems.
The data.frame(...) function converts any character vectors to factors by default. So in your code ds$PerfName is a factor with one level, and cs$PerfName is a factor with 5 levels. So in your assignment statement:
ds[cavities,] <- join(ds[cavities,1:4], fillings, by="ZipCode", "left")
the 5th column on the LHS is a factor with 1 level, and the 5th column on the RHS is a factor with 5 levels.
Under some circumstances, when you assign a factor with more levels to a factor with fewer levels, the missing levels are set to NA. Consider this:
x <- c("A","B",NA,NA,NA) # character vector
y <- LETTERS[1:5] # character vector
class(x); class(y)
# [1] "character"
# [1] "character"
df <- data.frame(x,y) # x and y coerced to factor
sapply(df,class) # df$x and df$y are factors
# x y
# "factor" "factor"
# assign rows 3:5 of col 2 to col 1
df[3:5,1] <- df[3:5,2] # fails with a warning
# Warning message:
# In `[<-.factor`(`*tmp*`, iseq, value = 3:5) :
# invalid factor level, NA generated
df # missing levels set to NA
# x y
# 1 A A
# 2 B B
# 3 <NA> C
# 4 <NA> D
# 5 <NA> E
The example above is equivalent to your assignment statement. However, notice what happens if you assign all of column 2 to column 1.
# assign all of col 2 to col 1
df <- data.frame(x,y)
df[,1] <- df[,2] # succeeds!!
df
# x y
# 1 A A
# 2 B B
# 3 C C
# 4 D D
# 5 E E
This works.
Finally, a note on debugging: if you are debugging a function, sometimes it is useful to run through the statements line by line at the command line (e.g., in the global environment). If you did that, you would have gotten the warning above, whereas inside a function call the warnings are suppressed.
The constraints of the test can be satisfied by reimplementing getMissingCountries with:
merge(ds[1:4], subset(subset(cs, PrimRec)[c(1, 3, 4)]), by="ZipCode")
Caveat: the ZipCode column is always emitted first, which differs from your expected result.
But to answer the subassignment question: it breaks, because the level sets of PrefName are incompatible between ds and cs. Either avoid using a factor or relevel them. You might have missed R's warning about this, because testthat was somehow suppressing warnings.