Find/replace or map, using a lookup table in R - r

Question

Total R-newbie, here. Please be gentle.
I have a column in a dataframe with numerical values representing ethnicity (UK Census data).
# create example data
id = c(1, 2, 3, 4, 5, 6, 7, 8, 9)
ethnicode = c(0, 1, 2, 3, 4, 5, 6, 7, 8)
df = data.frame(id, ethnicode)
I can do a mapping (or find/replace) to create a column (or edit an existing column) that contains a human-readable value:
# map values one-to-one from numeric to string
df$ethnicity <- mapvalues(df$ethnicode,
from = c(8, 7, 6, 5, 4, 3, 2, 1, 0),
to = c("Other", "Black", "Asian", "Mixed",
"WhiteOther", "WhiteIrish", "WhiteUK",
"WhiteTotal", "All"))
Of all of the things I tried this seemed to be the quickest (around 20 seconds for 9 million rows as opposed to over a minute with some approaches).
What I can’t seem to find (or understand from what I’ve read), is how to reference a lookup table instead.
# create lookup table
ethnicode = c(8, 7, 6, 5, 4, 3, 2, 1, 0)
ethnicity = c(("Other", "Black", "Asian", "Mixed", "WhiteOther",
"WhiteIrish", "WhiteUK", "WhiteTotal", "All")
lookup = data.frame(ethnicode, ethnicity)
The point being, if I want to change the human readable strings, or do anything else to the process, I’d rather do it once to the look-up table, than have to do it in several places in several scripts... and if I can do it more efficiently (under 20 seconds for 9 million rows) that would be good, too.
I also want to easily make sure that “8” still equals ‘Other’ (or whatever equivalent), and “0” still equals ‘All’, etc., which is more difficult, visually, with longer lists using the above approach.
Thanks in advance.

Answer 1

You could use named vectors for this. However, you would need to convert the ethnicode to character.
df = data.frame(
id = c(1, 2, 3, 4, 5, 6, 7, 8, 9),
ethnicode = as.character(c(0, 1, 2, 3, 4, 5, 6, 7, 8)),
stringsAsFactors=FALSE
)
# create lookup table
ethnicode = c(8, 7, 6, 5, 4, 3, 2, 1, 0)
ethnicity = c("Other", "Black", "Asian", "Mixed", "WhiteOther",
"WhiteIrish", "WhiteUK", "WhiteTotal", "All")
lookup = setNames(ethnicity, as.character(ethnicode))
Then you can do
df <- transform(df, ethnicity=lookup[ethnicode], stringsAsFactors=FALSE)
and you are done.
For working with 9 million rows, I suggest you use a database like sqlite or monetdb. For sqlite, the following code might be helpful:
library(RSQLite)
dbname <- "big_data_mapping.db" # db to create
csvname <- "data/big_data_mapping.csv" # large dataset
ethn_codes = data.frame(
ethnicode= c(8, 7, 6, 5, 4, 3, 2, 1, 0),
ethnicity= c("Other", "Black", "Asian", "Mixed", "WhiteOther", "WhiteIrish", "WhiteUK", "WhiteTotal", "All")
)
# build db
con <- dbConnect(SQLite(), dbname)
dbWriteTable(con, name="main", value=csvname, overwrite=TRUE)
dbWriteTable(con, name="ethn_codes", ethn_codes, overwrite=TRUE)
# join the tables
dat <- dbGetQuery(con, "SELECT main.id, ethn_codes.ethnicity FROM main JOIN ethn_codes ON main.ethnicode=ethn_codes.ethnicode")
# finish
dbDisconnect(con)
#file.remove(dbname)
monetdb is said to be more suitable for the tasks you usually do with R, so it is definitly worth a look.