The csv below is from a much longer data table, call it temp. I want to cast it to temp.wide with region_code as columns and with the vertical order of region_code (SAS, SSA, EUR, ...) as the order of the columns. I just noticed that dcast orders the new columns alphabetically.
scenario region_code region_name value
1: 2010 SAS South Asia 61.17716
2: 2010 SSA Africa south of the Sahara 62.08588
3: 2010 EUR Europe 63.76123
4: 2010 LAC Latin America and Caribbean 68.84806
5: 2010 FSU Former Soviet Union 59.04499
6: 2010 EAP East Asia and Pacific 64.00579
7: 2010 NAM North America 66.18235
8: 2010 MEN Middle East and North Africa 58.03167
9: SSP2-NoCC-REF SAS South Asia 57.29973
10: SSP2-NoCC-REF SSA Africa south of the Sahara 65.14987
11: SSP2-NoCC-REF EUR Europe 63.99204
12: SSP2-NoCC-REF LAC Latin America and Caribbean 68.21118
13: SSP2-NoCC-REF FSU Former Soviet Union 60.10807
14: SSP2-NoCC-REF EAP East Asia and Pacific 63.86103
15: SSP2-NoCC-REF NAM North America 65.97859
16: SSP2-NoCC-REF MEN Middle East and North Africa 58.98356
temp = setDT(structure(list(scenario = c("2010", "2010", "2010", "2010", "2010",
"2010", "2010", "2010", "SSP2-NoCC-REF", "SSP2-NoCC-REF", "SSP2-NoCC-REF",
"SSP2-NoCC-REF", "SSP2-NoCC-REF", "SSP2-NoCC-REF", "SSP2-NoCC-REF",
"SSP2-NoCC-REF"), region_code = c("SAS", "SSA", "EUR", "LAC",
"FSU", "EAP", "NAM", "MEN", "SAS", "SSA", "EUR", "LAC", "FSU",
"EAP", "NAM", "MEN"), region_name = c("South Asia", "Africa south of the Sahara",
"Europe", "Latin America and Caribbean", "Former Soviet Union",
"East Asia and Pacific", "North America", "Middle East and North Africa",
"South Asia", "Africa south of the Sahara", "Europe", "Latin America and Caribbean",
"Former Soviet Union", "East Asia and Pacific", "North America",
"Middle East and North Africa"), value = c(61.1771623260257,
62.0858809906661, 63.7612306428217, 68.84805628195, 59.0449875464304,
64.0057851485101, 66.182351351389, 58.0316719859857, 57.299725759211,
65.1498720847705, 63.9920412193261, 68.2111842947542, 60.1080745513644,
63.86103368494, 65.9785850777114, 58.9835574681585)), .Names = c("scenario",
"region_code", "region_name", "value"), row.names = c(NA, -16L
), class = "data.frame"))
Here's the code I used.
formula.wide <- "scenario ~ region_code"
temp.wide <- data.table::dcast(
data = temp,
formula = formula.wide,
value.var = "value")
scenario EAP EUR FSU LAC MEN NAM SAS SSA
1: 2010 64.00579 63.76123 59.04499 68.84806 58.03167 66.18235 61.17716 62.08588
2: SSP2-NoCC-REF 63.86103 63.99204 60.10807 68.21118 58.98356 65.97859 57.29973 65.14987
The new column names are scenario, EAP, EUR, FSU, LAC, MEN, NAM, SAS, SSA.
I can grab the correct order from temp and then use setcolorder to give temp.wide the correct column order. But I was wondering if there some way to not have the new column order alphabetized.
Also, the help text for dcast says
Names for columns that are being cast are generated in the same order
(separated by an underscore, _) from the (unique) values in each
column mentioned in the formula RHS.
If I am understanding this correctly, I don't think it describes what dcast actually does. But I don't understand what the parenthetical phrase (separated by an underscore, _) means.
with the vertical order of region_code (SAS, SSA, EUR, ...) as the order of the columns
Just pass a factor with appropriate levels:
dcast(temp, scenario ~ factor(region_code, levels=unique(region_code)))
scenario SAS SSA EUR LAC FSU EAP NAM MEN
1: 2010 61.17716 62.08588 63.76123 68.84806 59.04499 64.00579 66.18235 58.03167
2: SSP2-NoCC-REF 57.29973 65.14987 63.99204 68.21118 60.10807 63.86103 65.97859 58.98356
The documentation quoted in the OP sounds correct to me; in z ~ x + y -- x's unique values come before y's unique values in the order of the resulting column names.
Related
I have a dataframe with data about the US States.
One of the columns in the df is "Division", which tells the location where each state belongs to ("East North Central", "East South Central", "Middle Atlantic", "Mountain", "New England", "Pacific", "South Atlantic", "West North Central", "West South Central").
I created an array with the average expectancy life for each division, using an existing column called "Life Exp:
avg.life.exp = tapply(df[["Life Exp"]], df$Division, mean, na.rm=TRUE)
Which returns the following:
East North Central East South Central Middle Atlantic
70.99000 69.33750 70.63667
Mountain New England Pacific
70.94750 71.57833 71.69400
South Atlantic West North Central West South Central
69.52625 72.32143 70.43500
Now I would like to add a new column to the df, with the average life expectancy of each Division. So basically I would like to do a Left Join, where if the state belonged to the East Noth Central, it would return 70.99000, and so on.
I need to do this without using packages.
Thank you in advance for any help you can provide!
One option would be to use merge:
merge(df, data.frame(Division = names(avg.life.exp), avg.life.exp), all.x = TRUE)
A second option would be to use match
df$avg.life.exp <- avg.life.exp[match(df$Division, names(avg.life.exp))]
Using the gapminder dataset as example data:
library(gapminder)
# Example data
df <- gapminder[gapminder$year == 2007, c("country", "continent", "lifeExp")]
avg.life.exp <- tapply(df[["lifeExp"]], df$continent, mean, na.rm=TRUE)
avg.life.exp
#> Africa Americas Asia Europe Oceania
#> 54.80604 73.60812 70.72848 77.64860 80.71950
# Using merge
df1 <- merge(df, data.frame(continent = names(avg.life.exp), avg.life.exp), all.x = TRUE)
head(df1)
#> continent country lifeExp avg.life.exp
#> 1 Africa Reunion 76.442 54.80604
#> 2 Africa Eritrea 58.040 54.80604
#> 3 Africa Algeria 72.301 54.80604
#> 4 Africa Congo, Rep. 55.322 54.80604
#> 5 Africa Equatorial Guinea 51.579 54.80604
#> 6 Africa Malawi 48.303 54.80604
# Using match
df$avg.life.exp <- avg.life.exp[match(df$continent, names(avg.life.exp))]
head(df)
#> # A tibble: 6 × 4
#> country continent lifeExp avg.life.exp
#> <fct> <fct> <dbl> <dbl>
#> 1 Afghanistan Asia 43.8 70.7
#> 2 Albania Europe 76.4 77.6
#> 3 Algeria Africa 72.3 54.8
#> 4 Angola Africa 42.7 54.8
#> 5 Argentina Americas 75.3 73.6
#> 6 Australia Oceania 81.2 80.7
I have the following hypothetical list
test <- list(a = c("United", "States", "of", "America", "2021", "North", "America"),
b = c("Canada", "2021", "North", "America"),
c = c("Morocco", "2021", "Africa"),
d = c("South", "Africa", "2021", "Africa"),
e = c("Faroe", "Islands", "2021", "Europe"),
f = c("Spain", "2021", "Europe"))
I would like produce the following tibble:
country
year
continent
United States of America
2021
North America
Canada
2021
North America
Morocco
2021
Africa
South Africa
2021
Africa
Faroe Islands
2021
Europe
Spain
2021
Europe
I tried to use the ldply() function of the plyr package. However, my list elements have unequal lengths, because of compound names.
How could I join this data in a tibble with the variables: country, year and continent, for example?
An option is to use rleid to create a grouping based on the occurence of digits in the list, then paste the list elements and rbind them
library(data.table)
out <- type.convert(do.call(rbind.data.frame, lapply(test, function(x)
tapply(x, rleid(grepl('\\d+', x)), paste, collapse=' '))), as.is = TRUE)
colnames(out) <- c('country', 'year', 'continent')
row.names(out) <- NULL
-output
out
# country year continent
#1 United States of America 2021 North America
#2 Canada 2021 North America
#3 Morocco 2021 Africa
#4 South Africa 2021 Africa
#5 Faroe Islands 2021 Europe
#6 Spain 2021 Europe
Or use a similar option with rle from base R
out <- type.convert(do.call(rbind.data.frame,
lapply(test, function(x) tapply(x, with(rle(grepl('\\d+', x)),
rep(seq_along(values), lengths)), FUN = paste, collapse=' '))),
as.is = TRUE)
colnames(out) <- c('country', 'year', 'continent')
Slightly less general/efficient than the other solutions here but maybe more transparent?
cfun <- function(x) {
## find position of numeric value
numpos <- grep("^[0-9]+$", x)
## combine elements appropriately
list(country=paste(x[1:(numpos-1)], collapse=" "),
year=x[numpos],
continent=paste(x[(numpos+1):length(x)], collapse=" "))
}
purrr::map_dfr(test,cfun)
In base R you could do:
a <- do.call(rbind, lapply(test, function(x) paste(sub("(\\d+)",",\\1,", x), collapse = " ")))
read.csv(text=a, col.names = c("Country","Year","Continent"), h=FALSE)
Country Year Continent
1 United States of America 2021 North America
2 Canada 2021 North America
3 Morocco 2021 Africa
4 South Africa 2021 Africa
5 Faroe Islands 2021 Europe
6 Spain 2021 Europe
Here is another base R option
type.convert(setNames(
data.frame(
do.call(
rbind,
lapply(
test,
function(v) {
tapply(v,
cumsum(c(1, diff(grepl("\\d+", v)) != 0)), paste0,
collapse = " "
)
}
)
)
), c("Country", "Year", "Continent")
),
as.is = TRUE
)
which gives
Country Year Continent
a United States of America 2021 North America
b Canada 2021 North America
c Morocco 2021 Africa
d South Africa 2021 Africa
e Faroe Islands 2021 Europe
f Spain 2021 Europe
I have a question very similar to this question
country continent
<chr> <chr>
1 Taiwan Asia
2 New Zealand Oceania
3 Bulgaria Europe
4 Bahamas Americas
5 Serbia Europe
6 Tajikistan Asia
7 Southern Sub-Saharan Africa NA
8 Cameroon Africa
9 Indonesia Asia
10 Democratic Republic of Congo Africa
How do I use a function/write a loop so that when the country is "Bahamas" that it converts the continent so that it now says South America?
The page that I linked was the closest answer I could find but it differed from my question because I am trying to manipulate one column based on the values in a different column.
I tried using ifelse() but that did not work:
gm %>%
ifelse(country == "Bahamas", continent == "S America", continent)
Any insight would be greatly appreciated!
You need to mutate:
library(dplyr)
gm %>%
mutate(continent = ifelse(country == "Bahamas", "S America", continent))
This works:
gm[,'continent'][gm[,'country'] == "Bahamas"] <- "South America"
You might get a warning message like this if "South America" is not already in the dataframe:
Warning message:
In `[<-.factor`(`*tmp*`, gm[, "country"] == "Bahamas", value = c(2L, :
invalid factor level, NA generated
This means you need to add the level first, you are trying to issue a level which doesn't exist:
levels(gm$continent) <- c(levels(gm$continent), "South America")
gm[,'continent'][gm[,'country'] == "Bahamas"] <- "South America"
(run time on this approach [5M entries in a dataframe, 10 repeated measures] was 4x faster than the dplyr method)
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I've been given a set of country groups and I'm trying to get a set of mutually exclusive regions so that I can compare them. The problem is that my data contains several groups, many of which overlap. How can I get a set of groups which contain all countries, but do not overlap with each other?
For example, assume that this is the list of countries in the world:
World <- c("Angola", "France", "Germany", "Australia", "New Zealand")
Assume that this is my set of groups:
df <- data.frame(group = c("Africa", "Western Europe", "Europe", "Europe", "Oceania", "Oceania", "Commonwealth Countries"),
element = c("Angola", "France", "Germany", "France", "Australia", "New Zealand", "Australia"))
group element
1 Africa Angola
2 Western Europe France
3 Europe Germany
4 Europe France
5 Oceania Australia
6 Oceania New Zealand
7 Commonwealth Countries Australia
How could I remove overlapping groups (in this case Western Europe) to get a set of groups that contains all countries like the following:
df_solved <- data.frame(group = c("Africa", "Europe", "Europe", "Oceania", "Oceania"),
element = c("Angola", "France", "Germany", "Australia", "New Zealand"))
group element
1 Africa Angola
2 Europe France
3 Europe Germany
4 Oceania Australia
5 Oceania New Zealand
One possible rule could be to minimize the number of groups, e.g. to associate an element with that group which includes the most elements.
library(data.table)
setDT(df)[, n.elements := .N, by = group][
order(-n.elements), .(group = group[1L]), by = element]
element group
1: Germany Europe
2: France Europe
3: Australia Oceania
4: New Zealand Oceania
5: Angola Africa
Explanation
setDT(df)[, n.elements := .N, by = group][]
returns
group element n.elements
1: Africa Angola 1
2: Western Europe France 1
3: Europe Germany 2
4: Europe France 2
5: Oceania Australia 2
6: Oceania New Zealand 2
7: Commonwealth Countries Australia 1
Now, the rows are ordered by decreasing number of elements and for each country the first, i.e., the "largest", group is picked. This should return a group for each country as requested.
In case of ties, i.e., one group contains equally many elements, you can add additional citeria when ordering, e.g., length of the group name, or just alphabetical order.
1) If you want to simply eliminate duplicate elements then use !duplicated(...) as shown. No packages are used.
subset(df, !duplicated(element))
giving:
group element
1 Africa Angola
2 Europe France
3 Europe Germany
5 Oceania Australia
6 Oceania New Zealand
2) set partitioning If each group must be wholly in or wholly out and each element may only appear once then this is a set partitioning problem:
library(lpSolve)
const.mat <- with(df, table(element, group))
obj <- rep(1L, ncol(const.mat))
res <- lp("min", obj, const.mat, "=", 1L, all.bin = TRUE)
subset(df, group %in% colnames(const.mat[, res$solution == 1]))
giving:
group element
1 Africa Angola
2 Europe France
3 Europe Germany
5 Oceania Australia
6 Oceania New Zealand
3) set covering Of course there may be no exact set partition so we could consider the set covering problem (same code exceept "=" is replaced by ">=" in the lp line.
library(lpSolve)
const.mat <- with(df, table(element, group))
obj <- rep(1L, ncol(const.mat))
res <- lp("min", obj, const.mat, ">=", 1L, all.bin = TRUE)
subset(df, group %in% colnames(const.mat[, res$solution == 1]))
giving:
group element
1 Africa Angola
2 Europe France
3 Europe Germany
5 Oceania Australia
6 Oceania New Zealand
and we could optionally then apply (1) to remove any duplicates in the cover.
4) Non-dominated groups Another approach is to remove any group whose elements form a strict subset of the elements of some other group. For example, every element in Western Europe is in Europe and Europe has more elements than Western Europe so the elements of Western Europe are a strict subset of the elements of Europe and we remove Western Europe. Using const.mat from above:
# returns TRUE if jth column of const.mat is dominated by some other column
is_dom_fun <- function(j) any(apply(const.mat[, j] <= const.mat[, -j], 2, all) &
sum(const.mat[, j]) < colSums(const.mat[, -j]))
is_dom <- sapply(seq_len(ncol(const.mat)), is_dom_fun)
subset(df, group %in% colnames(const.mat)[!is_dom])
giving:
group element
1 Africa Angola
3 Europe Germany
4 Europe France
5 Oceania Australia
6 Oceania New Zealand
If there are any duplicates left we can use (1) to remove them.
library(dplyr)
df %>% distinct(element, .keep_all=TRUE)
group element
1 Africa Angola
2 Europe France
3 Europe Germany
4 Oceania Australia
5 Oceania New Zealand
Shoutout to Axeman for beating me with this answer.
Update
Your question is ill-defined. Why is 'Europe' preferred over 'Western Europe'? Put another way, each country is assigned several groups. You want to reduce it to one group per country. How do you decide which group?
Here's one way, we always prefer the biggest:
groups <- df %>% count(group)
df %>% inner_join(groups, by='group') %>%
arrange(desc(n)) %>% distinct(elemenet, .keep_all=TRUE)
group element n
1 Europe France 2
2 Europe Germany 2
3 Oceania Australia 2
4 Oceania New Zealand 2
5 Africa Angola 1
Here is one option with data.table
library(data.table)
setDT(df)[, head(.SD, 1), element]
Or with unique
unique(setDT(df), by = 'element')
# group element
#1: Africa Angola
#2: Europe France
#3: Europe Germany
#4: Oceania Australia
#5: Oceania New Zealand
Packages are used and it is data.table
A completely different approach would be to ignore the given groups but to look up just the country names in the catalogue of UN regions which are available in the countrycodes or ISOcodes packages.
The countrycodes package seems to offer the simpler interface and it also warns about country names which can not be found in its database:
# given country names - note the deliberately misspelled last entry
World <- c("Angola", "France", "Germany", "Australia", "New Zealand", "New Sealand")
# regions
countrycode::countrycode(World, "country.name.en", "region")
[1] "Middle Africa" "Western Europe" "Western Europe" "Australia and New Zealand"
[5] "Australia and New Zealand" NA
Warning message:
In countrycode::countrycode(World, "country.name.en", "region") :
Some values were not matched unambiguously: New Sealand
# continents
countrycode::countrycode(World, "country.name.en", "continent")
[1] "Africa" "Europe" "Europe" "Oceania" "Oceania" NA
Warning message:
In countrycode::countrycode(World, "country.name.en", "continent") :
Some values were not matched unambiguously: New Sealand
I have a data frame (df) that lists the countries associated with every site
Site Country
Site1 USA
Site2 Vietnam
Site3 Spain
Site4 Germany
Site5 China
I want to attach a column, where for each country I associate its corresponding continent. I wrote a simple if loop to do this:
df$Continent <- NA
if(df$Country == "USA" |df$Country == "Canada" |df$Country == "Mexico")
{df$Continent <- "North America"}
if(df$Country == "Spain" |df$Country == "France" |df$Country == "Germany")
{df$Continent <- "Europe"}
## .. etc
summary(df)
However, each time I run it the df, I find that it assigns North America to all the countries. I understand that this may sound trivial, but does it make a difference if I use if statments everywhere and not else or if else? Any suggestions for correcting this?
Build a lookup table and merge() it with the data.
For example:
lookup <- data.frame(Country = c("USA", "Canada", "Mexico",
"Spain", "France", "Germany",
"Vietnam", "China"),
Continent = rep(c("North America", "Europe", "Asia"),
times = c(3,3,2)))
Using your snippet of data as data frame df, we can add Continent via merge() (a join in database terminology):
> merge(df, lookup, sort = FALSE, all.x = TRUE)
Country Site Continent
1 USA Site1 North America
2 Vietnam Site2 Asia
3 Spain Site3 Europe
4 Germany Site4 Europe
5 China Site5 Asia
If you're working with a factor you can also do some nonsense with levels, or levels<- to be exact:
`levels<-`(dat$Country, list(
`North America` = c("USA","Canada","Mexico"),
`Europe` = c("Spain","France","Germany"),
`Asia` = c("Vietnam","China")
))
#[1] North America Asia Europe Europe Asia
#Levels: North America Europe Asia
I like ifelse() for things like this. You could use it with the %in% operator like this:
df$Continent <- ifelse(df$Country %in% c("USA", "Canada", "Mexico"),
"North America", df$Continent)
df$Continent <- ifelse(df$Country %in% c("Spain", "France", "Germany"),
"Europe", df$Continent)
df
Site Country Continent
1 Site1 USA North America
2 Site2 Vietnam <NA>
3 Site3 Spain Europe
4 Site4 Germany Europe
5 Site5 China <NA>