Problem: I have 2 datasets with no matching identifiers (like ID) and need to find the closest match in df1$time to df2$tstart. df1 (with time column) has 660,000 rows with time stamps approximately every 0.00125 s.
Whatever the closest match is to df2$tstart, I would like a new column made (df1$trial_start) that says "yes", otherwise "no".
I've tried findInterval, but it only seems to match in ascending order, and doesn't check values in both directions. In the below code, it looks good for most of the outputs, but there are some indices where the value after the returned index is closer to $tstart
#my actual code:
index_closest <- findInterval(iti_summaries_2183[["24"]]$tstart, poke_1s$time)
poke_1s$trial_start <- ifelse(seq_len(nrow(poke_1s)) %in% index_closest, "yes", "no")
I've also tried which.min, which doesn't work since the lists lengths don't match.
Additionally, I've fought with roll = "nearest" like here but the functions return values and I'm not sure how to create a new column and assign y/n.
Code to replicate problem:
n <- 773
df1 <- structure(list(initiate = sample(c(0,1), replace=TRUE, size=n),
left = sample(c(0,1), replace=TRUE, size=n),
right = sample(c(0,1), replace=TRUE, size=n),
time = seq(from = 2267.2, to = 2363.75, by = 0.125)))
df1 <- data.frame(df1)
df2 <- structure(list(trial = c(156:162),
control = c(0, 0, 0, 0, 3, 0, 3),
t_start = c(2267.231583, 2289.036355, 2298.046849, 2318.933635, 2328.334036, 2347.870449, 2363.748095),
t_end = c(2268.76760, 2290.83370, 2299.38547, 2320.71400, 2329.93985, 2349.15464, 2365.12455)),
class = c("spec_tbl_df",
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -7L), spec = structure(list(
cols = list(trial = structure(list(), class = c("collector_double",
"collector")), control = structure(list(), class = c("collector_double",
"collector")), t_start = structure(list(), class = c("collector_double",
"collector")), t_end = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), skip = 1L), class = "col_spec"))
If I understand your question correctly :
library(data.table)
setDT(df1)
setDT(df2)
df1[df2,.(initiate,left,right,x.time,trial,control,t_start,t_end,
trial_start=fifelse(x.time>t_start&x.time<t_end,'Y','N')),
on=.(time=t_start),roll='nearest']
initiate left right x.time trial control t_start t_end trial_start
<num> <num> <num> <num> <int> <num> <num> <num> <char>
1: 0 0 1 2267.200 156 0 2267.232 2268.768 N
2: 0 0 1 2289.075 157 0 2289.036 2290.834 Y
3: 0 0 1 2298.075 158 0 2298.047 2299.385 Y
4: 1 1 1 2318.950 159 0 2318.934 2320.714 Y
5: 1 1 1 2328.325 160 3 2328.334 2329.940 N
6: 0 0 1 2347.825 161 0 2347.870 2349.155 N
7: 1 1 0 2363.700 162 3 2363.748 2365.125 N
I need a chart of accounts to stay in order when new accounts are added or dropped in future years. This is because in Accounting the accounts are sorted by type (for example Asset, Liability Equity) but it is not explicit in the dataset. This is an example of the code that is putting new "Accounts" from Year2 and Year3 at the bottom.
XYZCompany_Consolidated <- XYZCompany_Year1 %>%
full_join(XYZCompany_Year2 by = "Account") %>%
full_join(XYZCompany_Year3, by = "Account")
Example: This picture is just to give a simplified example. The highlight in orange is where the new accounts are going and to the right is the code i'm using, and the green is what I'm trying to achieve
Perhaps I'm overthinking this problem but I find it hard to solve. Let's define some data first:
df_year1 <- structure(list(Account = c("Cash", "Accounts", "Loan1", "Auto",
"JaneDoe"), Year_1 = c(100, 1000, 20, 300, 500)), class = c("spec_tbl_df",
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -5L), spec = structure(list(
cols = list(Account = structure(list(), class = c("collector_character",
"collector")), Year_1 = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), skip = 1L), class = "col_spec"))
df_year2 <- structure(list(Account = c("Cash", "Accounts", "Loan1", "Auto",
"Laptop", "JaneDoe"), Year_2 = c(80, 1200, 50, 300, 500, 0)), class = c("spec_tbl_df",
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -6L), spec = structure(list(
cols = list(Account = structure(list(), class = c("collector_character",
"collector")), Year_2 = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), skip = 1L), class = "col_spec"))
df_year3 <- structure(list(Account = c("Cash", "Accounts", "Loan1", "Auto",
"Rent", "JaneDoe"), Year_3 = c(80, 1200, 50, 300, 1000, 0)), class = c("spec_tbl_df",
"tbl_df", "tbl", "data.frame"), row.names = c(NA, -6L), spec = structure(list(
cols = list(Account = structure(list(), class = c("collector_character",
"collector")), Year_3 = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), skip = 1L), class = "col_spec"))
Those are similar to the data shown in the OP's picture, e.g. df_year1 looks like
# A tibble: 5 x 2
Account Year_1
<chr> <dbl>
1 Cash 100
2 Accounts 1000
3 Loan1 20
4 Auto 300
5 JaneDoe 500
Next we transform those data a little bit, namely
library(dplyr)
library(tidyr)
df_y1 <- df_year1 %>%
mutate(Year = 1,
no = row_number()) %>%
rename(value = Year_1)
which returns
# A tibble: 5 x 4
Account value Year no
<chr> <dbl> <dbl> <int>
1 Cash 100 1 1
2 Accounts 1000 1 2
3 Loan1 20 1 3
4 Auto 300 1 4
5 JaneDoe 500 1 5
The new column no stores the account's original position, column Year stores the chart's year. All three data.frames are processed like this, so we get df_y1, df_y2, df_y3.
Finally we bind them together
bind_rows(df_y1, df_y2, df_y3) %>%
mutate(num_years = max(Year)) %>%
group_by(Account) %>%
mutate(rank = sum((num_years - n() + 1) * no), .keep = "unused") %>%
pivot_wider(names_from = Year) %>%
arrange(rank) %>%
select(-rank) %>%
ungroup()
and calculate a rank for each account. The accounts are ordered by this rank. As a result, we get
# A tibble: 7 x 4
Account Year_1 Year_2 Year_3
<chr> <dbl> <dbl> <dbl>
1 Cash 100 80 80
2 Accounts 1000 1200 1200
3 Loan1 20 50 50
4 Auto 300 300 300
5 Laptop NA 500 NA
6 Rent NA NA 1000
7 JaneDoe 500 0 0
Note
I believe, there are better approaches, but at least this works for the example data.
I'm not sure about the calculated rank's stability. Take care.
I am trying to get the first value of the next group in r to estimate a ratio. I have created a group based on the type column in my the df. Then estimated some influence factors using the sample position within the group. Finally, I am trying to estimate a ratio like this: RRF=response/(F1*first(response)+(F2*??????)) where the F1*first(response) is the cal in the group but I don't know how to call the first value of the next group to finish the ratio. Can someone help with this? This is my code and my data:
library(dplyr)
conc_zero_test <- zero_test %>%
gather(gas, response, -time,-type)%>%
group_by(group = cumsum(type == "current_std"),gas)%>%
mutate(X1= row_number()-1, #estimates the position of the sample within the group -1 removes std
F1=1-(X1/n()), #relative factor influence of the cal in the current group
F2=1-F1, #relative factor influence of the cal in the next group
RRF=response/(F1*first(response)+(F2*????))
structure(list(time = structure(c(1564468200, 1564475400, 1564484400,
1564486200, 1564493400, 1564497000, 1564498800, 1564506000, 1564509600,
1564511400, 1564518600, 1564522200, 1564524000, 1564527600, 1564531200
), class = c("POSIXct", "POSIXt"), tzone = "UTC"), type = c("current_std",
"n2", "n2", "current_std", "n2", "-", "current_std", "-", "n2",
"current_std", "n2", "-", "current_std", "-", "-"), ben = c(2293951.5,
12703.1, 6392.7, 1762512.6, 10748.4, 25468.3, 1597679, 24400.4,
6019.4, 1510760.2, 10329.1, 29292.6, 1495942.8, 61227.5, 25379.5
), xyl = c(210975.6, 4482, 2910.8, 127612.4, 3792.6, 10295.7,
113439.1, 10628.8, 2064.3, 107134.3, 3764.1, 10380.6, 107353.6,
23639.1, 10317.4), cym = c(546894.5, 12202.6, 8400.8, 302091.6,
11072.2, 16349.2, 291637.5, 18891.8, 6500.7, 276997.5, 10821.2,
18672, 274149.4, 61379.2, 19254.7), isop = c(397288.2, 0, 0,
239779.9, 0, 1364.8, 199081.5, 1511.2, 0, 179364, 0, 1318.4,
174450.7, 7137.5, 9567), macr = c(221195.8, 0, 0, 138806.3, 0,
0, 116644, 0, 0, 108893.3, 0, 0, 105689, 4325.4, 0), pin = c(50795.3,
0, 0, 28436, 0, 1020.1, 26482.9, 925.2, 0, 27394.1, 0, 989.7,
24344.6, 1414.7, 736.3), tmb = c(9314.5, 0, 0, 5798, 0, 0, 5136.4,
2252.5, 0, 4542.9, 0, 0, 4398.4, 3794.4, 2186.3), tol = c(880567.1,
7430.6, 4225.5, 569616.2, 6091.8, 65642.6, 495780.5, 52129.9,
3226, 456079.6, 5874, 34725.9, 453944.8, 56594.4, 66148.1), mvk = c(169036.8,
0, 0, 108738, 0, 0, 56712.5, 0, 0, 79148.9, 0, 0, 64065, 0, 0
), euc = c(12815.2, 0, 0, 8012.6, 0, 0, 5411.8, 0, 0, 5839.9,
0, 491.7, 5450.7, 1990.8, 500.7)), class = c("spec_tbl_df", "tbl_df",
"tbl", "data.frame"), row.names = c(NA, -15L), spec = structure(list(
cols = list(time = structure(list(format = ""), class = c("collector_datetime",
"collector")), type = structure(list(), class = c("collector_character",
"collector")), ben = structure(list(), class = c("collector_double",
"collector")), xyl = structure(list(), class = c("collector_double",
"collector")), cym = structure(list(), class = c("collector_double",
"collector")), isop = structure(list(), class = c("collector_double",
"collector")), macr = structure(list(), class = c("collector_double",
"collector")), pin = structure(list(), class = c("collector_double",
"collector")), tmb = structure(list(), class = c("collector_double",
"collector")), tol = structure(list(), class = c("collector_double",
"collector")), mvk = structure(list(), class = c("collector_double",
"collector")), euc = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), skip = 2), class = "col_spec"))
Example of expected output
time type gas response group X1 F1 F2 RRF
<dttm> <chr> <chr> <dbl> <int> <dbl> <dbl> <dbl> <dbl>
1 2019-07-30 06:30:00 current_std ben 2293952. 1 0 1 0 1
2 2019-07-30 08:30:00 n2 ben 12703. 1 1 0.667 0.333 0.006005
3 2019-07-30 11:00:00 n2 ben 6393. 1 2 0.333 0.667 0.003962
I would use a self-join to get the first response of the next group:
library(tidyverse)
# the OPs example data (long!)
zero_test <-
structure(
list(
time = structure(
c(
1564468200,
1564475400,
1564484400,
1564486200,
1564493400,
1564497000,
1564498800,
1564506000,
1564509600,
1564511400,
1564518600,
1564522200,
1564524000,
1564527600,
1564531200
),
class = c("POSIXct", "POSIXt"),
tzone = "UTC"
),
type = c(
"current_std",
"n2",
"n2",
"current_std",
"n2",
"-",
"current_std",
"-",
"n2",
"current_std",
"n2",
"-",
"current_std",
"-",
"-"
),
ben = c(
2293951.5,
12703.1,
6392.7,
1762512.6,
10748.4,
25468.3,
1597679,
24400.4,
6019.4,
1510760.2,
10329.1,
29292.6,
1495942.8,
61227.5,
25379.5
),
xyl = c(
210975.6,
4482,
2910.8,
127612.4,
3792.6,
10295.7,
113439.1,
10628.8,
2064.3,
107134.3,
3764.1,
10380.6,
107353.6,
23639.1,
10317.4
),
cym = c(
546894.5,
12202.6,
8400.8,
302091.6,
11072.2,
16349.2,
291637.5,
18891.8,
6500.7,
276997.5,
10821.2,
18672,
274149.4,
61379.2,
19254.7
),
isop = c(
397288.2,
0,
0,
239779.9,
0,
1364.8,
199081.5,
1511.2,
0,
179364,
0,
1318.4,
174450.7,
7137.5,
9567
),
macr = c(
221195.8,
0,
0,
138806.3,
0,
0,
116644,
0,
0,
108893.3,
0,
0,
105689,
4325.4,
0
),
pin = c(
50795.3,
0,
0,
28436,
0,
1020.1,
26482.9,
925.2,
0,
27394.1,
0,
989.7,
24344.6,
1414.7,
736.3
),
tmb = c(
9314.5,
0,
0,
5798,
0,
0,
5136.4,
2252.5,
0,
4542.9,
0,
0,
4398.4,
3794.4,
2186.3
),
tol = c(
880567.1,
7430.6,
4225.5,
569616.2,
6091.8,
65642.6,
495780.5,
52129.9,
3226,
456079.6,
5874,
34725.9,
453944.8,
56594.4,
66148.1
),
mvk = c(169036.8,
0, 0, 108738, 0, 0, 56712.5, 0, 0, 79148.9, 0, 0, 64065, 0, 0),
euc = c(
12815.2,
0,
0,
8012.6,
0,
0,
5411.8,
0,
0,
5839.9,
0,
491.7,
5450.7,
1990.8,
500.7
)
),
class = c("spec_tbl_df", "tbl_df",
"tbl", "data.frame"),
row.names = c(NA,-15L),
spec = structure(list(
cols = list(
time = structure(list(format = ""), class = c("collector_datetime",
"collector")),
type = structure(list(), class = c("collector_character",
"collector")),
ben = structure(list(), class = c("collector_double",
"collector")),
xyl = structure(list(), class = c("collector_double",
"collector")),
cym = structure(list(), class = c("collector_double",
"collector")),
isop = structure(list(), class = c("collector_double",
"collector")),
macr = structure(list(), class = c("collector_double",
"collector")),
pin = structure(list(), class = c("collector_double",
"collector")),
tmb = structure(list(), class = c("collector_double",
"collector")),
tol = structure(list(), class = c("collector_double",
"collector")),
mvk = structure(list(), class = c("collector_double",
"collector")),
euc = structure(list(), class = c("collector_double",
"collector"))
),
default = structure(list(), class = c("collector_guess",
"collector")),
skip = 2
), class = "col_spec")
)
temp1 <- zero_test %>%
gather(gas, response, -time,-type) %>%
group_by(group = cumsum(type == "current_std"), gas) %>%
mutate(X1= row_number()-1, #estimates the position of the sample within the group -1 removes std
F1=1-(X1/n()), #relative factor influence of the cal in the current group
F2=1-F1,
first_response = first(response)) %>%
ungroup
conc_zero_test <- temp1 %>%
left_join(y = {temp1 %>%
mutate(group = group - 1) %>%
select(gas, group, first_response_next = first_response) %>%
distinct},
by = c("gas", "group")) %>%
mutate(RRF = response / ((F1 * first_response) + (F2 * first_response_next)))
conc_zero_test
#> # A tibble: 150 x 11
#> time type gas response group X1 F1 F2
#> <dttm> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 2019-07-30 06:30:00 curr… ben 2293952. 1 0 1 0
#> 2 2019-07-30 08:30:00 n2 ben 12703. 1 1 0.667 0.333
#> 3 2019-07-30 11:00:00 n2 ben 6393. 1 2 0.333 0.667
#> 4 2019-07-30 11:30:00 curr… ben 1762513. 2 0 1 0
#> 5 2019-07-30 13:30:00 n2 ben 10748. 2 1 0.667 0.333
#> 6 2019-07-30 14:30:00 - ben 25468. 2 2 0.333 0.667
#> 7 2019-07-30 15:00:00 curr… ben 1597679 3 0 1 0
#> 8 2019-07-30 17:00:00 - ben 24400. 3 1 0.667 0.333
#> 9 2019-07-30 18:00:00 n2 ben 6019. 3 2 0.333 0.667
#> 10 2019-07-30 18:30:00 curr… ben 1510760. 4 0 1 0
#> # … with 140 more rows, and 3 more variables: first_response <dbl>,
#> # first_response_next <dbl>, RRF <dbl>
Created on 2020-08-16 by the reprex package (v0.3.0)
I am at a loss, I feel as if I am finding the answer... but I have been unable to do so. I really don't know where to start
I have the results from ranked voting:
It shows how many votes (100 voters) each candidate (5 candidates) got for each place (reproducible data is at the bottom):
Name 1st 2nd 3rd 4th 5th
Can1 50 0 15 25 10
Can2 15 25 0 10 50
Can3 25 50 10 0 15
Can4 0 10 50 15 25
Can5 10 15 0 0 0
I am trying to recreate ballots from the results, 100 ballots like this (also, some ballots have not been filled completely):
Ballot1: Can1, Can3, Can4, Can5, Can2
Ballot2: Can1, Can3, Can5
Ballot3: Can3, Can5, Can2, Can1, Can4
...
Ballot100: Can2, Can5, Can1, Can4
I need to do this with 60 candidates and more than 1000 votes.
voting.results <- structure(list(X1 = c("Can1", "Can2", "Can3", "Can4", "Can5"),
`1place` = c(50L, 15L, 25L, 0L, 10L), `2place` = c(0L, 25L,
50L, 10L, 15L), `3place` = c(15L, 0L, 10L, 50L, 0L), `4place` = c(25L,
10L, 0L, 15L, 0L), `5place` = c(10L, 50L, 15L, 25L, 0L)), .Names = c("X1",
"1place", "2place", "3place", "4place", "5place"), class = "data.frame", row.names = c(NA,
-5L), spec = structure(list(cols = structure(list(X1 = structure(list(), class = c("collector_character",
"collector")), `1place` = structure(list(), class = c("collector_integer",
"collector")), `2place` = structure(list(), class = c("collector_integer",
"collector")), `3place` = structure(list(), class = c("collector_integer",
"collector")), `4place` = structure(list(), class = c("collector_integer",
"collector")), `5place` = structure(list(), class = c("collector_integer",
"collector"))), .Names = c("X1", "1place", "2place", "3place",
"4place", "5place")), default = structure(list(), class = c("collector_guess",
"collector"))), .Names = c("cols", "default"), class = "col_spec"))
at the beginning would be nice to have this dataset with each candidates and his/her all performances. What was done below it's just repeating each row (candidate, place) by the time it's occurred in voting.results.
df1 is a number of specific places by candidate.
library(magrittr);library(dplyr)
df1 <-
voting.results %>%
reshape2::melt() %>%
mutate( variable = as.integer(gsub("place","",variable) )) %>%
rename(place=variable,can=X1)
head(df1)
# can place value
# 1 Can1 1 50
# 2 Can2 1 15
# 3 Can3 1 25
# 4 Can4 1 0
# 5 Can5 1 10
# 6 Can1 2 0
And df2 is a dataset with row per each performance.
df2 <-
df1[ rep(row.names(df1), df1$value) , ] %>%
mutate(id = 1:n()) %>%
select(-value) %>%
arrange(place)
head(df2)
# can place id
# 1 Can1 1 1
# 2 Can1 1 2
# 3 Can1 1 3
# 4 Can1 1 4
# 5 Can1 1 5
# 6 Can1 1 6
We know that all events have it's winners, so we can initiate each separate event with first candidate (Assuming there is no ex-qequo). And then in every for( e in 1:length(events) ) add candidates at other places. Added candidates are substracted from initial dataset.
If some individuals from initial dataset are not assigned (nrow(temp)>1), then process is repeated until it's finish with success.
i <- 0
temp <- data.frame(1)
while(nrow(temp)>0){
i <- i + 1
temp <- df2[ sample(1:nrow(df2)),]
events <- temp %>% filter(place==1) %>% split(1:nrow(.))
for( e in 1:length(events) ){
for( p in sort( unique(temp$place) ) ){
inAlready <- events[[e]]
toInput <-
temp %>%
filter( !can %in% inAlready$can & place == p) %>%
.[1,]
events[[e]] <- rbind( inAlready , toInput )
}
events[[e]]$event <- e
idToExclude <- lapply( events , function(x) x$id) %>% unlist
temp %<>% filter(!id %in% idToExclude)
}
}
all <-
bind_rows(events) %>%
arrange(event, place) %>%
filter(!is.na(id))
I don't know if it's perfect solution, and how many iterations are necessary, but I hope this will help you find perfect solution. Anyway, probably there is more than one final solutions, so the perfect reproducibility could be impossible. I'm curious if there is some operational-research-like method to solve this problem.
Enjoy and good luck!