Within my group (the "name" variable), I want cut the value into quartile. And create a quartile label column for variable "value". Since the group size varies, for the quartile range for different group changes as well. But below code, only cut the quartile by the overall value, resulting the same quartile range for all groups.
dt<-data.frame(name=c(rep('a',8),rep('b',4),rep('c',5)),value=c(1:8,1:4,1:5))
dt
dt.2<-dt%>% group_by(name)%>% mutate(newcol=
cut(value,breaks=quantile(value,probs=seq(0,1,0.25),na.rm=TRUE),include.lowest=TRUE))
dt.2
str(dt.2)
Data:
name value
1 a 1
2 a 2
3 a 3
4 a 4
5 a 5
6 a 6
7 a 7
8 a 8
9 b 1
10 b 2
11 b 3
12 b 4
13 c 1
14 c 2
15 c 3
16 c 4
17 c 5
output from above code.
Update: the problem is not that newcol is factor but the necol has the same quartile range across all the different group. For example name b, the value is 1-4 but the quartile range has 3-5, which is derived from min(value) to max(value) regardless of the group.
name value newcol
<fctr> <int> <fctr>
1 a 1 [1,2]
2 a 2 [1,2]
3 a 3 (2,3]
4 a 4 (3,5]
5 a 5 (3,5]
6 a 6 (5,8]
7 a 7 (5,8]
8 a 8 (5,8]
9 b 1 [1,2]
10 b 2 [1,2]
11 b 3 (2,3]
12 b 4 (3,5]
13 c 1 [1,2]
14 c 2 [1,2]
15 c 3 (2,3]
16 c 4 (3,5]
17 c 5 (3,5]
Desired output
name value newcol/quartile label
1 a 1 1
2 a 2 1
3 a 3 2
4 a 4 2
5 a 5 3
6 a 6 3
7 a 7 4
8 a 8 4
9 b 1 1
10 b 2 2
11 b 3 3
12 b 4 4
13 c 1 1
14 c 2 2
15 c 3 3
16 c 4 4
17 c 5 4
Here's a way you can do it, following the split-apply-combine framework.
dt<-data.frame(name=c(rep('a',8),rep('b',4),rep('c',5)),value=c(1:8,1:4,1:5))
split_dt <- lapply(split(dt, dt$name),
transform,
quantlabel = as.numeric(
cut(value, breaks = quantile(value, probs = seq(0,1,.25)), include.lowest = T)))
dt <- unsplit(split_dt, dt$name)
name value quantlabel
1 a 1 1
2 a 2 1
3 a 3 2
4 a 4 2
5 a 5 3
6 a 6 3
7 a 7 4
8 a 8 4
9 b 1 1
10 b 2 2
11 b 3 3
12 b 4 4
13 c 1 1
14 c 2 1
15 c 3 2
16 c 4 3
17 c 5 4
edit: there's a data.table way
following this post, we can use the data.table package, if performance is a concern:
library(data.table)
dt<-data.frame(name=c(rep('a',8),rep('b',4),rep('c',5)),value=c(1:8,1:4,1:5))
dt.t <- as.data.table(dt)
dt.t[,quantlabels := as.numeric(cut(value, breaks = quantile(value, probs = seq(0,1,.25)), include.lowest = T)), name]
name value quantlabels
1: a 1 1
2: a 2 1
3: a 3 2
4: a 4 2
5: a 5 3
6: a 6 3
7: a 7 4
8: a 8 4
9: b 1 1
10: b 2 2
11: b 3 3
12: b 4 4
13: c 1 1
14: c 2 1
15: c 3 2
16: c 4 3
17: c 5 4
edit: and there's a dplyr way
We can follow #akrun's advice and use as.numeric (which is what we've done for the other solutions):
dt %>%
group_by(name) %>%
mutate(quantlabel =
as.numeric(
cut(value,
breaks = quantile(value, probs = seq(0,1,.25)),
include.lowest = T)))
Note that if you instead wanted the labels themselves, use as.character:
dt %>%
group_by(name) %>%
mutate(quantlabel = as.character(cut(value, breaks = quantile(value, probs = seq(0,1,.25)), include.lowest = T)))
Source: local data frame [17 x 3]
Groups: name [3]
name value quantlabel
<fctr> <int> <chr>
1 a 1 [1,2.75]
2 a 2 [1,2.75]
3 a 3 (2.75,4.5]
4 a 4 (2.75,4.5]
5 a 5 (4.5,6.25]
6 a 6 (4.5,6.25]
7 a 7 (6.25,8]
8 a 8 (6.25,8]
9 b 1 [1,1.75]
10 b 2 (1.75,2.5]
11 b 3 (2.5,3.25]
12 b 4 (3.25,4]
13 c 1 [1,2]
14 c 2 [1,2]
15 c 3 (2,3]
16 c 4 (3,4]
17 c 5 (4,5]
Related
I have run across similar questions, but have not been able to find an answer for my specific needs.
I have a data set with a nested group design and I need to include a unique non-repeating ID to nested groups that can have identical values. While I regularly conduct this type of data wrangling, both the structure of this data set as well as the required outcome are beyond my skillset at this time.
Below I have provided an example data set (df) and what the results should look like.
I used the below code in my actual data set, but realized that it fails under certain circumstances...which are exaggerated in the example data set provided here. I prefer the ID to be sequentially numbered.
df$ID = cumsum(c(TRUE, diff(df$LENGTH) != 0))
I am open to all options (e.g., library(data.table), library(boot), etc) as it would be great if others find this post useful. However, I prefer solutions that do not require the installation and loading of additional packages.
Thanks in advance for you help.
Take care.
df <- read.table(text = "GROUP REGION TIME LENGTH
a x 1 3
a x 2 3
a x 3 3
a y 4 3
a y 5 3
a y 6 3
a z 7 2
a z 8 2
b z 1 2
b z 2 2
b x 3 2
b x 4 2
c x 1 2
c x 2 2
c y 3 2
c y 4 2
c x 5 2
c x 6 2
c z 7 1", header = TRUE)
result <- read.table(text = "GROUP REGION TIME LENGTH ID
a x 1 3 1
a x 2 3 1
a x 3 3 1
a y 4 3 2
a y 5 3 2
a y 6 3 2
a z 7 2 3
a z 8 2 3
b z 1 2 4
b z 2 2 4
b x 3 2 5
b x 4 2 5
c x 1 2 6
c x 2 2 6
c y 3 2 7
c y 4 2 7
c x 5 2 8
c x 6 2 8
c z 7 1 9", header = TRUE)
Paste GROUP and REGION columns and use rle to create a sequential ID column.
transform(df,ID = with(rle(paste(GROUP, REGION)),rep(seq_along(values),lengths)))
In data.table we can use rleid.
library(data.table)
setDT(df)[, ID := rleid(GROUP, REGION)]
# GROUP REGION TIME LENGTH ID
# 1: a x 1 3 1
# 2: a x 2 3 1
# 3: a x 3 3 1
# 4: a y 4 3 2
# 5: a y 5 3 2
# 6: a y 6 3 2
# 7: a z 7 2 3
# 8: a z 8 2 3
# 9: b z 1 2 4
#10: b z 2 2 4
#11: b x 3 2 5
#12: b x 4 2 5
#13: c x 1 2 6
#14: c x 2 2 6
#15: c y 3 2 7
#16: c y 4 2 7
#17: c x 5 2 8
#18: c x 6 2 8
#19: c z 7 1 9
Another base R option, but without rle
transform(
df,
ID = cumsum(c(1, (s <- paste0(GROUP, REGION))[-1] != head(s, -1)))
)
gives
GROUP REGION TIME LENGTH ID
1 a x 1 3 1
2 a x 2 3 1
3 a x 3 3 1
4 a y 4 3 2
5 a y 5 3 2
6 a y 6 3 2
7 a z 7 2 3
8 a z 8 2 3
9 b z 1 2 4
10 b z 2 2 4
11 b x 3 2 5
12 b x 4 2 5
13 c x 1 2 6
14 c x 2 2 6
15 c y 3 2 7
16 c y 4 2 7
17 c x 5 2 8
18 c x 6 2 8
19 c z 7 1 9
With dplyr
library(dplyr)
library(data.table)
df %>%
mutate(ID = rleid(GROUP, REGION))
This is my data frame
df <- data.frame(
id = 1:14,
group_id = c(rep(1:2, each = 3), rep(3:4, each = 4)),
type = rep("A", 14), stringsAsFactors = FALSE)
df[c(2,4,8,12),"type"] <- "B"
id group_id type
1 1 1 A
2 2 1 B
3 3 1 A
4 4 2 B
5 5 2 A
6 6 2 A
7 7 3 A
8 8 3 B
9 9 3 A
10 10 3 A
11 11 4 A
12 12 4 B
13 13 4 A
14 14 4 A
I'd like to keep all rows with type B as well as the following row.
I could do...
B <- which(df$type=="B")
afterB <- B+1
df_sel <- df[c(B, afterB), ]
df_sel <- df_sel[order(df_sel$id),]
df_sel
...to get what I want.
id group_id type
2 2 1 B
3 3 1 A
4 4 2 B
5 5 2 A
8 8 3 B
9 9 3 A
12 12 4 B
13 13 4 A
How can this be done in a more generic way.
Another way, very similar to what you do but in one step and without the need to reorder:
df_sel <- df[rep(which(df$type=="B"), e=2)+c(0, 1), ]
df_sel
# id group_id type
# 2 2 1 B
# 3 3 1 A
# 4 4 2 B
# 5 5 2 A
# 8 8 3 B
# 9 9 3 A
# 12 12 4 B
# 13 13 4 A
Using lag from dplyr
library(dplyr)
df[df$type == "B" | lag(df$type == "B", default = FALSE), ]
# id group_id type
#2 2 1 B
#3 3 1 A
#4 4 2 B
#5 5 2 A
#8 8 3 B
#9 9 3 A
#12 12 4 B
#13 13 4 A
using grep will provide a row index of all instances of B - rows; concatenate (c()) this with rows + 1 to select from df will work.
rows <- grep("B", df[, "type"])
df[sort(c(rows, rows + 1)), ]
gives:
id group_id type
2 2 1 B
3 3 1 A
4 4 2 B
5 5 2 A
8 8 3 B
9 9 3 A
12 12 4 B
13 13 4 A
This question already has answers here:
Using dplyr to get cumulative count by group
(3 answers)
Closed 5 years ago.
I come to an issue with numbering the duplicated rows in data.frame and could not find a similar post.
Let's say we have a data like this
df <- data.frame(gr=gl(7,2),x=c("a","a","b","b","c","c","a","a","c","c","d","d","a","a"))
> df
gr x
1 1 a
2 1 a
3 2 b
4 2 b
5 3 c
6 3 c
7 4 a
8 4 a
9 5 c
10 5 c
11 6 d
12 6 d
13 7 a
14 7 a
and want to add new column called x_dupl to show that first occurrence of x values is numbered as 1 and second time 2 and third time 3 and so on..
thanks in advance!
The expected output
> df
gr x x_dupl
1 1 a 1
2 1 a 1
3 2 b 1
4 2 b 1
5 3 c 1
6 3 c 1
7 4 a 2
8 4 a 2
9 5 c 2
10 5 c 2
11 6 d 1
12 6 d 1
13 7 a 3
14 7 a 3
Your example data (plus rows where gr = 7 as in your output), and named df1, not df:
df1 <- data.frame(gr = gl(7,2),
x = c("a","a","b","b","c","c","a","a","c","c","d","d","a","a"))
library(dplyr)
df1 %>%
group_by(x) %>%
mutate(x_dupl = dense_rank(gr)) %>%
ungroup()
# A tibble: 14 x 3
gr x x_dupl
<fctr> <fctr> <int>
1 1 a 1
2 1 a 1
3 2 b 1
4 2 b 1
5 3 c 1
6 3 c 1
7 4 a 2
8 4 a 2
9 5 c 2
10 5 c 2
11 6 d 1
12 6 d 1
13 7 a 3
14 7 a 3
A base R solution:
df <- data.frame(gr=gl(7,2),x=c("a","a","b","b","c","c","a","a","c","c","d","d","a","a"))
x <- rle(as.numeric(df$x))
x$values <- ave(x$values, x$values, FUN = seq_along)
df$x_dupl <- inverse.rle(x)
# gr x x_dupl
# 1 1 a 1
# 2 1 a 1
# 3 2 b 1
# 4 2 b 1
# 5 3 c 1
# 6 3 c 1
# 7 4 a 2
# 8 4 a 2
# 9 5 c 2
# 10 5 c 2
# 11 6 d 1
# 12 6 d 1
# 13 7 a 3
# 14 7 a 3
I have a data frame,I want to create a variable z,count duplicate of "y variable", if y have 1,1 set z = 2,2, if y have 3,3,3, set z = 3,3,3.
x = c("a","b","c","d","e","a","b","c","d","e","a","b","c")
y = c(1,1,2,2,2,3,3,4,4,4,5,5,5)
data <- data.frame(x,y)
data
x y z
1 a 1 2
2 b 1 2
3 c 2 3
4 d 2 3
5 e 2 3
6 a 3 2
7 b 3 2
8 c 4 3
9 d 4 3
10 e 4 3
11 a 5 3
12 b 5 3
13 c 5 3
Thanks for your help.
You can try the rle:
data$z <- with(data, unlist(mapply(rep, rle(y)$lengths, rle(y)$lengths)))
data
x y z
1 a 1 2
2 b 1 2
3 c 2 3
4 d 2 3
5 e 2 3
6 a 3 2
7 b 3 2
8 c 4 3
9 d 4 3
10 e 4 3
11 a 5 3
12 b 5 3
13 c 5 3
If your your variable y is sorted as an increasing sequence as you say, then the following solution will work:
# calculate counts of each level
counts <- table(data$y)
# fill in z
data$z <- counts[match(data$y, names(counts))]
Note, however, that this method will fail if y is not ordered and, since you want to restart the count when a different level occurs. For these purposes, #psidom's solution is more robust to mis-ordered data as rle will reset the count.
This method calculates the total occurrences of a level and then feeds these total counts to the proper location using match.
Here is a quick method using dplyr, and its rather intuitive syntax:
library(dplyr)
left_join(data, data %>%
group_by(y) %>%
summarize(z = n()),
by = "y")
x y z
1 a 1 2
2 b 1 2
3 c 2 3
4 d 2 3
5 e 2 3
6 a 3 2
7 b 3 2
8 c 4 3
9 d 4 3
10 e 4 3
11 a 5 3
12 b 5 3
13 c 5 3
We can do this easily with data.table
library(data.table)
setDT(data)[, z := .N , rleid(y)]
data
# x y z
# 1: a 1 2
# 2: b 1 2
# 3: c 2 3
# 4: d 2 3
# 5: e 2 3
# 6: a 3 2
# 7: b 3 2
# 8: c 4 3
# 9: d 4 3
#10: e 4 3
#11: a 5 3
#12: b 5 3
#13: c 5 3
Or using rle from base R without any loops
inverse.rle(within.list(rle(data$y), values <- lengths))
#[1] 2 2 3 3 3 2 2 3 3 3 3 3 3
Or another base R method with ave
with(data, ave(y, cumsum(c(TRUE, y[-1]!= y[-length(y)])), FUN=length))
#[1] 2 2 3 3 3 2 2 3 3 3 3 3 3
I've googled the issue as many ways as my brain is capable of and I still can't find the answer. I'm new to R so there are some things that confuse me a little bit.
Let's say I have a data table like this:
x y z 100 200 300
1: 1 1 a 1 1 1
2: 1 1 b 2 3 4
3: 1 2 c 3 5 7
4: 1 2 d 4 7 0
5: 2 1 e 5 9 3
6: 2 1 f 6 1 6
7: 2 2 g 7 3 9
8: 2 2 h 8 5 2
This can be created with this piece of code:
DT = setDT(structure(list(c(1, 1, 1, 1, 2, 2, 2, 2),
c(1, 1, 2, 2, 1, 1, 2, 2),
c("a","b","c","d","e","f","g","h"),
c(1,2,3,4,5,6,7,8),
c(1,3,5,7,9,1,3,5),
c(1,4,7,0,3,6,9,2)),
.Names = c("x", "y", "z", 100, 200, 300), row.names = c(NA, -8L), class = "data.frame"))
However, in my actual code, the last three columns were auto-generated using another function (dcast), so the total number of columns of the data.table is not static. Also, you may notice that the names of those three last columns are numeric, which might be a problem at some point.
What I need is to create one aditional column for each "extra" column (the ones right after column "z"). I need the code to work such as this example: first, it creates column "100s", then for each row, it calculates the sum of column "100", considering only the rows with the same combination of x,y that the row in question. And so on for "200s" and "300s". Like this:
x y z 100 200 300 100s 200s 300s
1: 1 1 a 1 1 1 3 4 5
2: 1 1 b 2 3 4 3 4 5
3: 1 2 c 3 5 7 7 12 7
4: 1 2 d 4 7 0 7 12 7
5: 2 1 e 5 9 3 11 10 9
6: 2 1 f 6 1 6 11 10 9
7: 2 2 g 7 3 9 15 8 11
8: 2 2 h 8 5 2 15 8 11
I've tried with several modifications of this idea of a code:
for (i in 3:(dim(DT)[2])) {
DT <- DT[,paste(colnames(DT)[i], "s", sep=""):=sum(i),
by=c("x","y")]
}
This gives me the following result:
x y z 100 200 300 100s 200s 300s
1: 1 1 a 1 1 1 4 5 6
2: 1 1 b 2 3 4 4 5 6
3: 1 2 c 3 5 7 4 5 6
4: 1 2 d 4 7 0 4 5 6
5: 2 1 e 5 9 3 4 5 6
6: 2 1 f 6 1 6 4 5 6
7: 2 2 g 7 3 9 4 5 6
8: 2 2 h 8 5 2 4 5 6
Of course, R is not recognizing the numeric value of i as the number of column it should consider for the sum, as instead it's taking it as a raw number. I can't figure out how to adress a specific column by its position, because when it comes to sum(), that "with=FALSE" thing fails to save the day.
Any help will be appreciated.
There is no need for using a for loop in this case to get the desired result. You can update DT by reference with:
DT[, paste0(colnames(DT)[3:5],'s') := lapply(.SD, sum), by = .(x,y)]
which will give you the desired result:
> DT
x y 100 200 300 100s 200s 300s
1: 1 1 1 1 1 3 4 5
2: 1 1 2 3 4 3 4 5
3: 1 2 3 5 7 7 12 7
4: 1 2 4 7 0 7 12 7
5: 2 1 5 9 3 11 10 9
6: 2 1 6 1 6 11 10 9
7: 2 2 7 3 9 15 8 11
8: 2 2 8 5 2 15 8 11
When you don't know exacly which columns to sum, you could use one of the following methods:
# method 1:
DT[, paste0(colnames(DT)[3:ncol(DT)],'s') := lapply(.SD, sum), by = .(x,y)]
# method 2:
DT[, paste0(setdiff(colnames(DT), c('x','y')),'s') := lapply(.SD, sum), by = .(x,y)]
With the updated example, probably the best way to do is:
cols <- setdiff(colnames(DT), c('x','y','z'))
DT[, paste0(cols,'s') := lapply(.SD, sum), by = .(x,y), .SDcols = cols]
which gives:
> DT
x y z 100 200 300 100s 200s 300s
1: 1 1 a 1 1 1 3 4 5
2: 1 1 b 2 3 4 3 4 5
3: 1 2 c 3 5 7 7 12 7
4: 1 2 d 4 7 0 7 12 7
5: 2 1 e 5 9 3 11 10 9
6: 2 1 f 6 1 6 11 10 9
7: 2 2 g 7 3 9 15 8 11
8: 2 2 h 8 5 2 15 8 11