individuating rows based on conditions in nested data - r

Question

I am new to r, and I am having some trouble manipulating the data in the way I need it for my analysis. I would be grateful if anyone could help, because this is essential for my research.
I already asked a similar question but the answer I got did not fully address my problem, I will try to be more clear this time to see if anyone can help.
my data looks something like this:
df<- data.frame(
"Reporter" = c("USA", "USA", "USA", "USA", "USA","USA"),
"Partner" = c( "EU", "EU","EU","EU", "EU","EU"),
"Product.cat" = c("1", "11", "111", "112", "12", "2"),
"Product Description" = c("Food", "Fruit", "Apple",
"Banana", "Meat", "Manifactured"),
"Year" = c(1970, 1970, 1970, 1970, 1970, 1970),
"trade value" = c( 100, 50, 30, 20, 50, 220),
stringsAsFactors = FALSE)
I have country-year observations about trade.
The vector 'product.cat' indicates what kind of commodity is exported. The more digits the product.cat has, the more the trade information is disaggregated.
For example product.cat. 111 (eg. apple) and 112 (e.g. bananas) are sub-product categories of product category 11 (e.g. fruit).
The same holds for the higher levels of aggregation. Product category 11 (fruit) is a subcategory of product.cat 1(food) together with product.cat 12 (meat).
To note that data in lower categories is nested in higher level of aggregation. Hence the value of product.cat 11 (50) is equal to the value of product.cat 111 (30) + product.cat 112 (20).
To do my analysis I need to identify those values that are not reported at the most disaggregated possible level - i.e. I need to identify the data not reported at the 3 digit level.
My problem is that for some country-year observation I have data reported accurately at all levels of aggregation (e.g. 1,11,111,112) while for others i only have data at the higher level of aggregation (e.g. 12 and 2). For instance, in my example, I only have product.cat 12 (meat), but not data on what kind of meat product.cat 121(pork), product.cat 122 (veal).
Similarly, in the example, data on product.cat 2 (manufacturing), is not reported at lower levels.
we do not know whether is product.cat 21 (clothing) or product.cat 22 (wood products).
In other words, I have data reported at the 2 digit (12) or first digit level (2) that could be reported at the 3 digit level. To note that every category should be disaggregated at the 3 level digit
What I would like to do is to find a way to individuate all the data exclusively reported at a higher level of aggregation and change their product.cat name adding an "m" to the end.
After manipulation the product.cat 12 should become* 12m to indicate that data was reported only at the 2nd digit.
Similarly I would like to identify exports that are reported only at the first digit. product.cat 2 should become 2mm to reflect that the data was reported only at the first digit.
To be sure, only the data for which I have information exclusively at a higher level of aggregation - i.e. in the example 12 and 2 - should include "m"s.
For instance, in the example, I do not want to have 1mm, since I have data at a lower level of aggregation (11,12). Similarly, I do not want to have 11m, because I have data at lower levels of aggregation (111,112). What I would like to have is 12m and 2mm because the data is reported only at a higher level of aggregation (12 and 2).
I know that this is a very specific question but I would really appreciate if anyone could help.
Note: in the real dataset, due to for measurement errors, the sum of the disaggregated values do not always perfectly add up to the higher level of aggregation. (for instance, 111+112 can be > 11). Hence, ideally to solve the issue the, I am looking to a function that is able to specify when to add the m based on the number of digits divided by country, partner, year, rather than the sum of the traded value.
I really thank everyone that could give me a help with this, it would be a huge step forward for my research.
---- attempts
I have been working on this function, but it does not seem to do what I am looking for. Maybe someone can find out what is going wrong
fillLevel <- function(x, width = 3, fill = "m"){
sp <- split(x, substr(x, 1, 1))
sp <- lapply(seq_along(sp), function(i){
n <- nchar(sp[[i]])
if(all(n < 3)){
j <- which(n == max(n))
sp[[i]][j] <- gsub(" ", "m", formatC(sp[[i]][j], width = -3))
}
sp[[i]]
})
unname(unlist(sp))
}
df <- df%>% mutate(prdcat2 = fillLevel(df$Product.cat.))
As you can see it only individuates 2mm but not 12m. Moreover when I run it on more complex codes it mess up the order of my data. I think this relates to sp <- lapply(seq_along(sp) but i am not sure how to go about it.
Best

Answer 1

Here's one way to do it:
library(data.table)
setDT(df)
# tag levels
df[, lvl := nchar(Product.cat)]
df[lvl < 3L, has_subcat := FALSE]
# use level-3 observations to flag level-2s as okay
df[
df[lvl == 3, .(Reporter, Partner, Year, Product.cat = substr(Product.cat, 1, 2))],
on=.(Reporter, Partner, Year, Product.cat),
has_subcat := TRUE
]
# use level-2 observations to flag level-1s as okay
df[
df[lvl == 2, .(Reporter, Partner, Year, Product.cat = substr(Product.cat, 1, 1))],
on=.(Reporter, Partner, Year, Product.cat),
has_subcat := TRUE
]
# create new cat, flagging observations with no subcategories
df[, newcat := Product.cat]
df[has_subcat == FALSE, newcat := paste0(Product.cat, strrep("m", 3-lvl))]
Reporter Partner Product.cat Product.Description Year trade.value lvl has_subcat newcat
1: USA EU 1 Food 1970 100 1 TRUE 1
2: USA EU 11 Fruit 1970 50 2 TRUE 11
3: USA EU 111 Apple 1970 30 3 NA 111
4: USA EU 112 Banana 1970 20 3 NA 112
5: USA EU 12 Meat 1970 50 2 FALSE 12m
6: USA EU 2 Manifactured 1970 220 1 FALSE 2mm
I'm assuming that this should be done separately per Reporter-Partner-Year.