I need to divide a vector in quantiles, ie. bins with the same number of observations. I am currently looking at this two functions dplyr::ntile and ggplot2::cut_number. It looks like they do roughly the same thing. The only difference is that ntile gives back the quantile to which the observation belongs, ie. 1, 2, 3, ..., whereas cut_number returns the limits of the interval, ie. (0, 0.5], (0.5, 1], ... .
I did some experiments and it looks like I roughly get the same answers:
library(tidyverse)
df <- tibble(vec = runif(1000))
df %>% mutate(vec_cut = cut_number(vec, 10)) %>% count(vec_cut)
#> # A tibble: 10 x 2
#> vec_cut n
#> <fct> <int>
#> 1 [7.29e-05,0.0905] 100
#> 2 (0.0905,0.211] 100
#> 3 (0.211,0.325] 100
#> 4 (0.325,0.423] 100
#> 5 (0.423,0.5] 100
#> 6 (0.5,0.602] 100
#> 7 (0.602,0.699] 100
#> 8 (0.699,0.806] 100
#> 9 (0.806,0.91] 100
#> 10 (0.91,0.997] 100
df %>% mutate(vec_tile = ntile(vec, 10)) %>%
group_by(vec_tile) %>%
summarise(count = n(),
min = min(vec),
max = max(vec))
#> `summarise()` ungrouping output (override with `.groups` argument)
#> # A tibble: 10 x 4
#> vec_tile count min max
#> <int> <int> <dbl> <dbl>
#> 1 1 100 0.0000729 0.0905
#> 2 2 100 0.0905 0.210
#> 3 3 100 0.211 0.324
#> 4 4 100 0.325 0.422
#> 5 5 100 0.423 0.499
#> 6 6 100 0.501 0.602
#> 7 7 100 0.602 0.699
#> 8 8 100 0.702 0.806
#> 9 9 100 0.806 0.910
#> 10 10 100 0.911 0.997
The problem is that sometimes cut_numbers fails where ntile does not.
vec <- c(rep(0,100), seq(1:100))
table(cut_number(vec, 10))
#> Error: Insufficient data values to produce 10 bins.
#> Run `rlang::last_error()` to see where the error occurred.
table(ntile(vec,10))
#> 1 2 3 4 5 6 7 8 9 10
#> 20 20 20 20 20 20 20 20 20 20
I could use ntile, however it is nice to have the interval limits and not just the index of the quantiles. Am I doing something wrong?
Related
I have a list of few thousand data frames, and about ~95% of those data frames have the required information for me to run a specific script for them. However, in my list of data frames, there are a dozen data frames that are missing this one specific value in the column Z for me to run a script for them, so I want to filter those data frames out of the list completely. Here's a quick example of what I mean:
> head(list_of_df[[1]])
# A tibble: 6 × 4
A B Z id
<dbl> <dbl> <chr> <int>
1 27.3 0.485 "{\"type\":\"M\",\"msg\":\"VALUE0\",\}" 1
2 27.4 0.457 NA 1
3 27.5 0.430 NA 1
4 27.6 0.402 NA 1
5 27.7 0.374 "{\"type\":\"M\",\"msg\":\"VALUE1\",\}" 1
6 27.8 0.347 NA 1
The above minimal datasheet has "VALUE1" at some point in the Z column, so this data frame is OK. However if there would not be any instances of "VALUE1" in the Z column in a data frame X, then that data frame X would be filtered out. How can I do this in R?
As a bonus question, how could I filter out all the data frames from a list of data frames, that don't have the matching number of rows with "VALUE0" and "VALUE1" in the Z column..?
This should work:
list_of_df <- list_of_df[which(lapply(list_of_df, function(x) sum(grepl("VALUE1", x$Z))) > 0)]
Here's a way of doing it using purrr::keep:
(This uses simulated data where the column to check is a and the value to test for is "D")
library(tidyverse)
list_of_dfs <- map(1:20, ~tibble(a = sample(LETTERS, 10),
b = sample(1:100, 10)))
list_of_dfs %>%
keep(~ any(str_detect(.x$a, "D")))
#> [[1]]
#> # A tibble: 10 x 2
#> a b
#> <chr> <int>
#> 1 X 15
#> 2 W 36
#> 3 L 69
#> 4 D 63
#> 5 A 23
#> 6 P 72
#> 7 S 30
#> 8 Q 33
#> 9 B 92
#> 10 C 37
#>
#> [[2]]
#> # A tibble: 10 x 2
#> a b
#> <chr> <int>
#> 1 O 28
#> 2 W 85
#> 3 H 15
#> 4 D 53
#> 5 Y 77
#> 6 S 16
#> 7 C 46
#> 8 E 12
#> 9 F 11
#> 10 M 74
#>
#> ... etc.
Created on 2022-03-31 by the reprex package (v2.0.1)
I have a list of all CpG locations (base pair value) for a gene on a methylation array in one table (table a), and another table with the locations (base pair value) for 12 CpGs for the same gene not present on the array (table b). I am trying to work out for each probe in table_b, which probe in table_a is the closest in bp.
i.e. table_a
# A tibble: 88 x 2
UCSC_RefGene_Name pos
<chr> <int>
1 RXRA 137218280
2 RXRA 137243592
3 RXRA 137330570
4 RXRA 137225311
5 RXRA 137299436
6 RXRA 137277819
7 RXRA 137268074
8 RXRA 137255666
9 RXRA;RXRA 137284989
10 RXRA 137218286
# ... with 78 more rows
table_b
CpG.position Human.genome.19.coordinates
1 1 137215735
2 2 137215739
3 3 137215748
4 4 137215772
5 5 137215779
6 6 137215867
7 7 137215956
8 8 137216015
9 9 137216030
10 10 137216034
11 11 137216036
12 12 137216064
My first step was to sequentially subtract the each value in A from the first row in B -
bibs <- function(table, value, column){
position <- sym(column)
smaps <-
table %>%
summarise(
"cpg_pos" = table$CpG.position,
"new_loc" = value - {{position}})
print(smaps)
}
posns <- table_a$positions
abso <- list()
for(i in seq_along(posns)){
a <- bibs(table_b, posns[[i]], "Human.genome.19.coordinates")
abso[[i]] <- a
}
This produces a list (abso) with 88 entries (1st entry below), so seemingly its only happened for the first value in table b.
cpg_pos new_loc
1 1 2545
2 2 2541
3 3 2532
4 4 2508
5 5 2501
6 6 2413
7 7 2324
8 8 2265
9 9 2250
10 10 2246
11 11 2244
12 12 2216
I wonder if anyone can help with getting it to move sequentially through each value in B?
Thanks,
Matt
Joining is equivalent to filtering the cross-product. We can sort all combinations of rows from both tables to pick the one with the closest distance:
library(tidyverse)
# example data
genes <- tibble(gene = c("A", "A", "B"), gene_pos = c(1, 30, 50))
genes
#> # A tibble: 3 × 2
#> gene gene_pos
#> <chr> <dbl>
#> 1 A 1
#> 2 A 30
#> 3 B 50
cpgs <- tibble(cpg = seq(3), cpg_pos = c(48, 51, 31))
cpgs
#> # A tibble: 3 × 2
#> cpg cpg_pos
#> <int> <dbl>
#> 1 1 48
#> 2 2 51
#> 3 3 31
cpgs %>%
expand_grid(genes) %>%
mutate(dist = abs(gene_pos - cpg_pos)) %>%
group_by(cpg) %>%
arrange(dist) %>%
slice(1)
#> # A tibble: 3 × 5
#> # Groups: cpg [3]
#> cpg cpg_pos gene gene_pos dist
#> <int> <dbl> <chr> <dbl> <dbl>
#> 1 1 48 B 50 2
#> 2 2 51 B 50 1
#> 3 3 31 A 30 1
Created on 2022-04-14 by the reprex package (v2.0.0)
CPG number 1 is at position 48. The closest gene position is position 50 of gene B which is 2bp apart.
I have recently come across an interesting question of calculating a vector values using its penultimate value as .init argument plus an additional vector's current value. Here is the sample data set:
set.seed(13)
dt <- data.frame(id = rep(letters[1:2], each = 5), time = rep(1:5, 2), ret = rnorm(10)/100)
dt$ind <- if_else(dt$time == 1, 120, if_else(dt$time == 2, 125, as.numeric(NA)))
id time ret ind
1 a 1 0.005543269 120
2 a 2 -0.002802719 125
3 a 3 0.017751634 NA
4 a 4 0.001873201 NA
5 a 5 0.011425261 NA
6 b 1 0.004155261 120
7 b 2 0.012295066 125
8 b 3 0.002366797 NA
9 b 4 -0.003653828 NA
10 b 5 0.011051443 NA
What I would like to calculate is:
ind_{t} = ind_{t-2}*(1+ret_{t})
I tried the following code. Since .init is of no use here I tried the nullify the original .init and created a virtual .init but unfortunately it won't drag the newly created values (from third row downward) into calculation:
dt %>%
group_by(id) %>%
mutate(ind = c(120, accumulate(3:n(), .init = 125,
~ .x * 1/.x * ind[.y - 2] * (1 + ret[.y]))))
# A tibble: 10 x 4
# Groups: id [2]
id time ret ind
<chr> <int> <dbl> <dbl>
1 a 1 0.00554 120
2 a 2 -0.00280 125
3 a 3 0.0178 122.
4 a 4 0.00187 125.
5 a 5 0.0114 NA
6 b 1 0.00416 120
7 b 2 0.0123 125
8 b 3 0.00237 120.
9 b 4 -0.00365 125.
10 b 5 0.0111 NA
I was wondering if there was a tweak I could make to this code and make it work completely.
I would appreciate your help greatly in advance
Use a state vector consisting of the current value of ind and the prior value of ind. That way the prior state contains the second prior value of ind. We encode that into complex values with the real part equal to ind and the imaginary part equal to the prior value of ind. At the end we take the real part.
library(dplyr)
library(purrr)
dt %>%
group_by(id) %>%
mutate(result = c(ind[1],
Re(accumulate(.x = tail(ret, -2),
.f = ~ Im(.x) * (1 + .y) + Re(.x) * 1i,
.init = ind[2] + ind[1] * 1i)))) %>%
ungroup
giving:
# A tibble: 10 x 5
id time ret ind result
<chr> <int> <dbl> <dbl> <dbl>
1 a 1 0.00554 120 120
2 a 2 -0.00280 125 125
3 a 3 0.0178 NA 122.
4 a 4 0.00187 NA 125.
5 a 5 0.0114 NA 124.
6 b 1 0.00416 120 120
7 b 2 0.0123 125 125
8 b 3 0.00237 NA 120.
9 b 4 -0.00365 NA 125.
10 b 5 0.0111 NA 122.
Variation
This variation eliminates the complex numbers and uses a vector of 2 elements in place of each complex number with the first number corresponding to the real part in the prior solution and the second number of each pair corresponding to the imaginary part. This could be extended to cases where we need more than 2 numbers per state and where the dependence involves all of the last N values but for the question here there is the downside of the extra line of code to extract the result from the list of pairs of numbers which is more involved than using Re in the prior solution.
dt %>%
group_by(id) %>%
mutate(result = c(ind[1],
accumulate(.x = tail(ret, -2),
.f = ~ c(.x[2] * (1 + .y), .x[1]),
.init = ind[2:1])),
result = map_dbl(result, first)) %>%
ungroup
Check
We check that the results above are correct. Alternately this could be used as a straight forward solution.
calc <- function(ind, ret) {
for(i in seq(3, length(ret))) ind[i] <- ind[i-2] * (1 + ret[i])
ind
}
dt %>%
group_by(id) %>%
mutate(result = calc(ind, ret)) %>%
ungroup
giving:
# A tibble: 10 x 5
id time ret ind result
<chr> <int> <dbl> <dbl> <dbl>
1 a 1 0.00554 120 120
2 a 2 -0.00280 125 125
3 a 3 0.0178 NA 122.
4 a 4 0.00187 NA 125.
5 a 5 0.0114 NA 124.
6 b 1 0.00416 120 120
7 b 2 0.0123 125 125
8 b 3 0.00237 NA 120.
9 b 4 -0.00365 NA 125.
10 b 5 0.0111 NA 122.
I would have done it by creating dummy groups for each sequence, so that it can be done for any number of 'N'. Demonstrating it on a new elaborated data
df <- data.frame(
stringsAsFactors = FALSE,
grp = c("a","a","a","a",
"a","a","a","a","a","b","b","b","b","b",
"b","b","b","b"),
rate = c(0.082322056,
0.098491104,0.07294593,0.08741672,0.030179747,
0.061389031,0.011232314,0.08553277,0.091272669,
0.031577847,0.024039791,0.091719552,0.032540636,
0.020411727,0.094521716,0.081729178,0.066429708,
0.04985793),
ind = c(11000L,12000L,
13000L,NA,NA,NA,NA,NA,NA,10000L,13000L,12000L,
NA,NA,NA,NA,NA,NA)
)
df
#> grp rate ind
#> 1 a 0.08232206 11000
#> 2 a 0.09849110 12000
#> 3 a 0.07294593 13000
#> 4 a 0.08741672 NA
#> 5 a 0.03017975 NA
#> 6 a 0.06138903 NA
#> 7 a 0.01123231 NA
#> 8 a 0.08553277 NA
#> 9 a 0.09127267 NA
#> 10 b 0.03157785 10000
#> 11 b 0.02403979 13000
#> 12 b 0.09171955 12000
#> 13 b 0.03254064 NA
#> 14 b 0.02041173 NA
#> 15 b 0.09452172 NA
#> 16 b 0.08172918 NA
#> 17 b 0.06642971 NA
#> 18 b 0.04985793 NA
library(tidyverse)
N = 3
df %>% group_by(grp) %>%
group_by(d = row_number() %% N, .add = TRUE) %>%
mutate(ind = accumulate(rate[-1] + 1, .init = ind[1], ~ .x * .y))
#> # A tibble: 18 x 4
#> # Groups: grp, d [6]
#> grp rate ind d
#> <chr> <dbl> <dbl> <dbl>
#> 1 a 0.0823 11000 1
#> 2 a 0.0985 12000 2
#> 3 a 0.0729 13000 0
#> 4 a 0.0874 11962. 1
#> 5 a 0.0302 12362. 2
#> 6 a 0.0614 13798. 0
#> 7 a 0.0112 12096. 1
#> 8 a 0.0855 13420. 2
#> 9 a 0.0913 15057. 0
#> 10 b 0.0316 10000 1
#> 11 b 0.0240 13000 2
#> 12 b 0.0917 12000 0
#> 13 b 0.0325 10325. 1
#> 14 b 0.0204 13265. 2
#> 15 b 0.0945 13134. 0
#> 16 b 0.0817 11169. 1
#> 17 b 0.0664 14147. 2
#> 18 b 0.0499 13789. 0
Alternate answer in dplyr (using your own data modified a bit only)
set.seed(13)
dt <- data.frame(id = rep(letters[1:2], each = 5), time = rep(1:5, 2), ret = rnorm(10)/100)
dt$ind <- ifelse(dt$time == 1, 12000, ifelse(dt$time == 2, 12500, as.numeric(NA)))
library(dplyr, warn.conflicts = F)
dt %>% group_by(id) %>%
group_by(d= row_number() %% 2, .add = TRUE) %>%
mutate(ind = cumprod(1 + duplicated(id) * ret)* ind[1])
#> # A tibble: 10 x 5
#> # Groups: id, d [4]
#> id time ret ind d
#> <chr> <int> <dbl> <dbl> <dbl>
#> 1 a 1 0.00554 12000 1
#> 2 a 2 -0.00280 12500 0
#> 3 a 3 0.0178 12213. 1
#> 4 a 4 0.00187 12523. 0
#> 5 a 5 0.0114 12353. 1
#> 6 b 1 0.00416 12000 0
#> 7 b 2 0.0123 12500 1
#> 8 b 3 0.00237 12028. 0
#> 9 b 4 -0.00365 12454. 1
#> 10 b 5 0.0111 12161. 0
Goal: I would like to generate grouped percentiles for each group (hrzn)
I have the following data
# A tibble: 3,500 x 3
hrzn parameter density
<dbl> <dbl> <dbl>
1 1 0.0183 0.00914
2 1 0.0185 0.00905
3 1 0.0187 0.00897
4 1 0.0189 0.00888
5 1 0.0191 0.00880
6 1 0.0193 0.00872
7 1 0.0194 0.00864
8 1 0.0196 0.00855
9 1 0.0198 0.00847
10 1 0.0200 0.00839
The hrzn is the group, the parameter is a grid of parameter space, and the density is the density for the value in the parameter column.
I would like to generate summary the statistics percentiles 10 to 90 by 10 by hrzn. I am trying to keep this computationally efficient. I know I could sample the parameter with the density as weights, but I am curious is there is a faster way to generate the percentiles from the density without doing a sample.
The data may be obtained with the following
df <- readr::read_csv("https://raw.githubusercontent.com/alexhallam/density_data/master/data.csv")
When I load the data from your csv, each of the 5 groups have identical values for parameter and density:
df
#># A tibble: 3,500 x 3
#> hrzn parameter density
#> <int> <dbl> <dbl>
#> 1 1 0.0183 0.00914
#> 2 1 0.0185 0.00905
#> 3 1 0.0187 0.00897
#> 4 1 0.0189 0.00888
#> 5 1 0.0191 0.00880
#> 6 1 0.0193 0.00872
#> 7 1 0.0194 0.00864
#> 8 1 0.0196 0.00855
#> 9 1 0.0198 0.00847
#>10 1 0.0200 0.00839
#># ... with 3,490 more rows
sapply(1:5, function(x) all(df$parameter[df$hrzn == x] == df$parameter[df$hrzn == 1]))
# [1] TRUE TRUE TRUE TRUE TRUE
sapply(1:5, function(x) all(df$density[df$hrzn == x] == df$density[df$hrzn == 1]))
# [1] TRUE TRUE TRUE TRUE TRUE
I'm not sure if this is a mistake or not, but clearly if you're worried about computation, anything you want to do on all the groups can be done 5 times faster by only doing it on a single group.
Anyway, to get the 10th and 90th centiles for each hrzn, you just need to see which parameter is adjacent to 0.1 and 0.9 on the cumulative distribution function. Let's generalize to working it out for all the groups in case there's an issue with the data or you want to repeat it with different data:
library(dplyr)
df %>%
mutate(hrzn = factor(hrzn)) %>%
group_by(hrzn) %>%
summarise(centile_10 = parameter[which(cumsum(density) > .1)[1]],
centile_90 = parameter[which(cumsum(density) > .9)[1]] )
#># A tibble: 5 x 3
#> hrzn centile_10 centile_90
#> <fct> <dbl> <dbl>
#>1 1 0.0204 0.200
#>2 2 0.0204 0.200
#>3 3 0.0204 0.200
#>4 4 0.0204 0.200
#>5 5 0.0204 0.200
Of course, they're all the same for the reasons mentioned above.
If you're worried about computation time (even though the above only takes a few milliseconds), and you don't mind opaque code, you could take advantage of the ordering to cut the cumsum of your entire density column between 0 and 5 in steps of 0.1, to get all the 10th centiles, like this:
summary <- df[which((diff(as.numeric(cut(cumsum(df$density), seq(0,5,.1))) - 1) != 0)) + 1,]
summary <- summary[-(1:5)*10,]
summary$centile <- rep(1:9*10, 5)
summary
#> # A tibble: 45 x 4
#> hrzn parameter density centile
#> <int> <dbl> <dbl> <dbl>
#> 1 1 0.0204 0.00824 10
#> 2 1 0.0233 0.00729 20
#> 3 1 0.0271 0.00634 30
#> 4 1 0.0321 0.00542 40
#> 5 1 0.0392 0.00453 50
#> 6 1 0.0498 0.00366 60
#> 7 1 0.0679 0.00281 70
#> 8 1 0.103 0.00199 80
#> 9 1 0.200 0.00114 90
#> 10 2 0.0204 0.00824 10
#> # ... with 35 more rows
Perhaps I have misunderstood you and you are actually working in a 5-dimensional parameter space and want to know the parameter values at the 10th and 90th centiles of 5d density. In that case, you can take advantage of the fact that all groups are the same to calculate the 10th and 90th centiles for the 5-d density by simply taking the 5th root of these two centiles:
df %>%
mutate(hrzn = factor(hrzn)) %>%
group_by(hrzn) %>%
summarise(centile_10 = parameter[which(cumsum(density) > .1^.2)[1]],
centile_90 = parameter[which(cumsum(density) > .9^.2)[1]] )
#> # A tibble: 5 x 3
#> hrzn centile_10 centile_90
#> <fct> <dbl> <dbl>
#> 1 1 0.0545 0.664
#> 2 2 0.0545 0.664
#> 3 3 0.0545 0.664
#> 4 4 0.0545 0.664
#> 5 5 0.0545 0.664
I want to get the confidence intervals for proportions within my tibble. Is there a way of doing this?
library(tidyverse)
library(Hmisc)
library(broom)
df <- tibble(id = c(1, 2, 3, 4, 5, 6),
count = c(4, 1, 22, 4545, 33, 23),
n = c(22, 65, 34, 6323, 35, 45))
Which looks like this:
# A tibble: 6 x 3
id count n
<dbl> <dbl> <dbl>
1 1 4 22
2 2 1 65
3 3 22 34
4 4 4545 6323
5 5 33 35
6 6 23 45
Using binconf from Hmisc and tidy from broom the solution could be from any package:
The intervals for the first row:
tidy(binconf(4, 22))
# A tibble: 1 x 4
.rownames PointEst Lower Upper
<chr> <dbl> <dbl> <dbl>
1 "" 0.182 0.0731 0.385
I have tried using map in purrr but get errors:
map(df, tidy(binconf(count, n)))
Error in x[i] : object of type 'closure' is not subsettable
I could just calculate them using dplyr but I get values below zero (e.g. row 2) or above one (e.g row 5), which I don't want. e.g.
df %>%
mutate(prop = count / n) %>%
mutate(se = (sqrt(prop * (1-prop)/n))) %>%
mutate(lower = prop - (se*1.96)) %>%
mutate(upper = prop + (se*1.96))
# A tibble: 6 x 7
id count n prop se lower upper
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 4 22 0.182 0.0822 0.0206 0.343
2 2 1 65 0.0154 0.0153 -0.0145 0.0453
3 3 22 34 0.647 0.0820 0.486 0.808
4 4 4545 6323 0.719 0.00565 0.708 0.730
5 5 33 35 0.943 0.0392 0.866 1.02
6 6 23 45 0.511 0.0745 0.365 0.657
Is there a good way of doing this? I did have a look at the confint_tidy() function, but could not get that to work. Any ideas?
It may not be tidy but
> as.tibble(cbind(df, binconf(df$count, df$n)))
# A tibble: 6 x 6
id count n PointEst Lower Upper
<dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 1 4 22 0.182 0.0731 0.385
2 2 1 65 0.0154 0.000789 0.0821
3 3 22 34 0.647 0.479 0.785
4 4 4545 6323 0.719 0.708 0.730
5 5 33 35 0.943 0.814 0.984
6 6 23 45 0.511 0.370 0.650
seems to work