I wanted to assign same serial number for all same Submission_Ids under one Batch_number. Could some one please help me figure this out?
Submission_Id <- c(619295,619295,619295,619295,619296,619296,619296,619296,619296,556921,556921,559254,647327,647327,647327,646040,646040,646040,646040,646040,64604)
Batch_No <- (633,633,633,633,633,633,633,633,633,633,633,633,634,634,634,650,650,650,650,650,650)
Expected result
Sl.No <- c(1,1,1,1,2,2,2,2,2,3,3,4,1,1,1,1,1,1,1,1,1)
One way to do it is creating run-length IDs with data.table::rleid(Submission_Id) grouped_by(Batch_No). We can use this inside 'dplyr'. To show this I created a tibble() with both given vectors Batch_Id and Submission_Id.
library(dplyr)
library(data.table)
dat <- tibble(Submission_Id = Submission_Id,
Batch_No = Batch_No)
dat %>%
group_by(Batch_No) %>%
mutate(S1.No = data.table::rleid(Submission_Id))
#> # A tibble: 21 x 3
#> # Groups: Batch_No [3]
#> Submission_Id Batch_No S1.No
#> <dbl> <dbl> <int>
#> 1 619295 633 1
#> 2 619295 633 1
#> 3 619295 633 1
#> 4 619295 633 1
#> 5 619296 633 2
#> 6 619296 633 2
#> 7 619296 633 2
#> 8 619296 633 2
#> 9 619296 633 2
#> 10 556921 633 3
#> # ... with 11 more rows
The original data
Submission_Id <- c(619295,619295,619295,619295,619296,619296,619296,619296,619296,556921,556921,559254,647327,647327,647327,646040,646040,646040,646040,646040,64604)
Batch_No <- c(633,633,633,633,633,633,633,633,633,633,633,633,634,634,634,650,650,650,650,650,650)
Created on 2022-12-16 by the reprex package (v2.0.1)
Related
I am trying to obtain the largest 10 investors in a country but obtain confusing result using arrange in dplyr versus order in base R.
head(fdi_partner)
give the following results
# A tibble: 6 x 3
`Main counterparts` `Number of projects` `Total registered capital (Mill. USD)(*)`
<chr> <chr> <chr>
1 TOTAL 1818 38854.3
2 Singapore 231 11358.66
3 Korea Rep.of 377 7679.9
4 Japan 204 4325.79
5 Netherlands 24 4209.64
6 China, PR 216 3001.79
and
fdi_partner %>%
rename("Registered capital" = "Total registered capital (Mill. USD)(*)") %>%
mutate_at(c("Number of projects", "Registered capital"), as.numeric) %>%
arrange("Number of projects") %>%
head()
give almost the same result
# A tibble: 6 x 3
`Main counterparts` `Number of projects` `Registered capital`
<chr> <dbl> <dbl>
1 TOTAL 1818 38854.
2 Singapore 231 11359.
3 Korea Rep.of 377 7680.
4 Japan 204 4326.
5 Netherlands 24 4210.
6 China, PR 216 3002.
while the following code is working fine with base R
head(fdi_partner)
fdi_numeric <- fdi_partner %>%
rename("Registered capital" = "Total registered capital (Mill. USD)(*)") %>%
mutate_at(c("Number of projects", "Registered capital"), as.numeric)
head(fdi_numeric[order(fdi_numeric$"Number of projects", decreasing = TRUE), ], n=11)
which gives
# A tibble: 11 x 3
`Main counterparts` `Number of projects` `Registered capital`
<chr> <dbl> <dbl>
1 TOTAL 1818 38854.
2 Korea Rep.of 377 7680.
3 Singapore 231 11359.
4 China, PR 216 3002.
5 Japan 204 4326.
6 Hong Kong SAR (China) 132 2365.
7 United States 83 783.
8 Taiwan 66 1464.
9 United Kingdom 50 331.
10 F.R Germany 37 131.
11 Thailand 36 370.
Can anybody help explain what's wrong with me?
dplyr (and more generally tidyverse packages) accept only unquoted variable names. If your variable name has a space in it, you must wrap it in backticks:
library(dplyr)
test <- data.frame(`My variable` = c(3, 1, 2), var2 = c(1, 1, 1), check.names = FALSE)
test
#> My variable var2
#> 1 3 1
#> 2 1 1
#> 3 2 1
# Your code (doesn't work)
test %>%
arrange("My variable")
#> My variable var2
#> 1 3 1
#> 2 1 1
#> 3 2 1
# Solution
test %>%
arrange(`My variable`)
#> My variable var2
#> 1 1 1
#> 2 2 1
#> 3 3 1
Created on 2023-01-05 with reprex v2.0.2
I have a data frame with the columns sampleID, method, parameter and value.
set.seed(123)
mydata <- data.frame(sample_ID = rep(1:100, each=4),
method = rep(LETTERS[1:2], 100),
parameter = rep(c("M1","M2"),times=c(2,2)),
value = round(runif(100, min = 100, max = 5000)),
stringsAsFactors = FALSE)
This data frame is organized in long format and I would like to convert it to wide format like this: The sample_ID should be the identifier of the row - now the columns method + parameter should be combined with the corresponding value, f.e.
Sample_ID 1 has the value
1509 for method A and parameter M1
3963 for method B and parameter M1
2104 for method A and parameter M2
4427 for method B and parameter M2
Now I would like to convert these 4 rows to a single row like this:
sample_ID = 1, A_M1 = 1509, B_M1 = 3963, A_M2 = 2104, B_M2 = 4427
The next row would be consist of those variables with sample_ID = 2, ...
I'm sorry but I was not able to do this with spread() or melt().
Thank you in advance!
Making use of tidyr::pivot_wider you could do:
tidyr::pivot_wider(mydata, names_from = c("method", "parameter"), values_from = value)
#> # A tibble: 100 × 5
#> sample_ID A_M1 B_M1 A_M2 B_M2
#> <int> <dbl> <dbl> <dbl> <dbl>
#> 1 1 1509 3963 2104 4427
#> 2 2 4708 323 2688 4473
#> 3 3 2802 2337 4788 2321
#> 4 4 3420 2906 604 4509
#> 5 5 1306 306 1707 4777
#> 6 6 4459 3495 3238 4972
#> 7 7 3313 3572 2766 3011
#> 8 8 1517 821 4819 4521
#> 9 9 3484 3998 221 2441
#> 10 10 3816 1160 1659 1235
#> # … with 90 more rows
Using dcast() from data.table package
dcast(mydata, sample_ID~...)
If you convert your data.frame to a data.table first using setDT() you can express this using the proper data.table notation
mydata[, dcast(.SD, sample_ID~...)]
The content in the column appears as follows $1,521+ 2 bds. I want to extract 1521 and put it in a new column. I know this can be done in alteryx using regex can I do it R?
How about the following?:
library(tidyverse)
x <- '$1,521+ 2 bds'
parse_number(x)
For example:
library(tidyverse)
#generate some data
tbl <- tibble(string = str_c('$', as.character(seq(1521, 1541, 1)), '+', ' 2bds'))
new_col <-
tbl$string %>%
str_split('\\+',simplify = TRUE) %>%
`[`(, 1) %>%
str_sub(2, -1) #get rid of '$' at the start
mutate(tbl, number = new_col)
#> # A tibble: 21 x 2
#> string number
#> <chr> <chr>
#> 1 $1521+ 2bds 1521
#> 2 $1522+ 2bds 1522
#> 3 $1523+ 2bds 1523
#> 4 $1524+ 2bds 1524
#> 5 $1525+ 2bds 1525
#> 6 $1526+ 2bds 1526
#> 7 $1527+ 2bds 1527
#> 8 $1528+ 2bds 1528
#> 9 $1529+ 2bds 1529
#> 10 $1530+ 2bds 1530
#> # … with 11 more rows
Created on 2021-06-12 by the reprex package (v2.0.0)
We can use sub from base R
as.numeric( sub("\\$(\\d+),(\\d+).*", "\\1\\2", x))
#[1] 1521
data
x <- '$1,521+ 2 bds'
Is it possible in some way to use a fit object, specifically the regression object I get form a plm() model, to flag observations, in the data used for the regression, if they were in fact used in the regression. I realize this could be done my looking for complete observations in my original data, but I am curious if there's a way to use the fit/reg object to flag the data.
Let me illustrate my issue with a minimal working example,
First some packages needed,
# install.packages(c("stargazer", "plm", "tidyverse"), dependencies = TRUE)
library(plm); library(stargazer); library(tidyverse)
Second some data, this example is drawing heavily on Baltagi (2013), table 3.1, found in ?plm,
data("Grunfeld", package = "plm")
dta <- Grunfeld
now I create some semi-random missing values in my data object, dta
dta[c(3:13),3] <- NA; dta[c(22:28),4] <- NA; dta[c(30:33),5] <- NA
final step in the data preparation is to create a data frame with an index attribute that describes its individual and time dimensions, using tidyverse,
dta.p <- dta %>% group_by(firm, year)
Now to the regression
plm.reg <- plm(inv ~ value + capital, data = dta.p, model = "pooling")
the results, using stargazer,
stargazer(plm.reg, type="text") # stargazer(dta, type="text")
#> ============================================
#> Dependent variable:
#> ---------------------------
#> inv
#> ----------------------------------------
#> value 0.114***
#> (0.008)
#>
#> capital 0.237***
#> (0.028)
#>
#> Constant -47.962***
#> (9.252)
#>
#> ----------------------------------------
#> Observations 178
#> R2 0.799
#> Adjusted R2 0.797
#> F Statistic 348.176*** (df = 2; 175)
#> ===========================================
#> Note: *p<0.1; **p<0.05; ***p<0.01
Say I know my data has 200 observations, and I want to find the 178 that was used in the regression.
I am speculating if there's some vector in the plm.reg I can (easily) use to crate a flag i my original data, dta, if this observation was used/not used, i.e. the semi-random missing values I created above. Maybe some broom like tool.
I imagine something like,
dta <- dta %>% valid_reg_obs(plm.reg)
The desired outcome would look something like this, the new element is the vector plm.reg at the end, i.e.,
dta %>% as_tibble()
#> # A tibble: 200 x 6
#> firm year inv value capital plm.reg
#> * <int> <int> <dbl> <dbl> <dbl> <lgl>
#> 1 1 1935 318 3078 2.80 T
#> 2 1 1936 392 4662 52.6 T
#> 3 1 1937 NA 5387 157 F
#> 4 1 1938 NA 2792 209 F
#> 5 1 1939 NA 4313 203 F
#> 6 1 1940 NA 4644 207 F
#> 7 1 1941 NA 4551 255 F
#> 8 1 1942 NA 3244 304 F
#> 9 1 1943 NA 4054 264 F
#> 10 1 1944 NA 4379 202 F
#> # ... with 190 more rows
Update, I tried to use broom's augment(), but unforunatly it gave me the error message I had hoped would create some flag,
# install.packages(c("broom"), dependencies = TRUE)
library(broom)
augment(plm.reg, dta)
#> Error in data.frame(..., check.names = FALSE) :
#> arguments imply differing number of rows: 200, 178
The vector is plm.reg$residuals. Not sure of a nice broom solution, but this seems to work:
library(tidyverse)
dta.p %>%
as.data.frame %>%
rowid_to_column %>%
mutate(plm.reg = rowid %in% names(plm.reg$residuals))
for people who use the class pdata.frame() to create an index attribute that describes its individual and time dimensions, you can us the following code, this is from another Baltagi in the ?plm,
# == Baltagi (2013), pp. 204-205
data("Produc", package = "plm")
pProduc <- pdata.frame(Produc, index = c("state", "year", "region"))
form <- log(gsp) ~ log(pc) + log(emp) + log(hwy) + log(water) + log(util) + unemp
Baltagi_reg_204_5 <- plm(form, data = pProduc, model = "random", effect = "nested")
pProduc %>% mutate(reg.re = rownames(pProduc) %in% names(Baltagi_reg_204_5$residuals)) %>%
as_tibble() %>% select(state, year, region, reg.re)
#> # A tibble: 816 x 4
#> state year region reg.re
#> <fct> <fct> <fct> <lgl>
#> 1 CONNECTICUT 1970 1 T
#> 2 CONNECTICUT 1971 1 T
#> 3 CONNECTICUT 1972 1 T
#> 4 CONNECTICUT 1973 1 T
#> 5 CONNECTICUT 1974 1 T
#> 6 CONNECTICUT 1975 1 T
#> 7 CONNECTICUT 1976 1 T
#> 8 CONNECTICUT 1977 1 T
#> 9 CONNECTICUT 1978 1 T
#> 10 CONNECTICUT 1979 1 T
#> # ... with 806 more rows
finally, if you are running the first Baltagi without index attributes, i.e. unmodified example from the help file, the code should be,
Grunfeld %>% rowid_to_column %>%
mutate(plm.reg = rowid %in% names(p$residuals)) %>% as_tibble()
** Sample data added after comment**
What I have:
pmts <- data.frame(stringsAsFactors=FALSE,
name = c("johndoe", "johndoe", "janedoe", "foo", "foo", "foo"),
pmt_amount = c(550L, 550L, 995L, 375L, 375L, 375L),
pmt_date = c("9/1/16", "11/1/16", "12/15/16", "1/5/17", "3/5/17", "5/5/17")
)
#> name pmt_amount pmt_date
#> 1 johndoe 550 9/1/16
#> 2 johndoe 550 11/1/16
#> 3 janedoe 995 12/15/16
#> 4 foo 375 1/5/17
#> 5 foo 375 3/5/17
#> 6 foo 375 5/5/17
What I am looking to achieve:
read.table(header = T, text =
"name pmt_amount first_pmt second_pmt third_pmt
johndoe 550 9/1/16 11/1/16 NA
janedoe 995 12/15/16 NA NA
foo 375 1/5/17 3/5/17 5/5/17"
)
#> name pmt_amount first_pmt second_pmt third_pmt
#> 1 johndoe 550 9/1/16 11/1/16 <NA>
#> 2 janedoe 995 12/15/16 <NA> <NA>
#> 3 foo 375 1/5/17 3/5/17 5/5/17
** End of update**
I have a large dataset with payment information for different products. Some of these products have a pay-in-full option as well as a two-pay and three-pay option. I need to create fields that would be First_Payment, Second_Payment, and Third_Payment and would populate NA in the respective fields if there was only one or two payments.
I've tried a couple options and the best workaround I have thus far is this:
pmts %>%
group_by(Email, Name, Amount, Form.Title) %>%
summarise(First_Payment = min(Payment.Date),
Second_Payment = median(Payment.Date),
Last_Payment = max(Payment.Date)) -> pmts
This obviously is not ideal as is making up a payment date for the 2-pay plans and I would have to instruct the end-user to ignore this field and just look at the 1st and 3rd fields.
I also tried to summarise with partial sorts like this:
n <- length(pmts$Payment.Date)
sort(pmts$Payment.Date,partial=n-1)[n-1]
However, if there wasn't three payments for the person, it would take the n-1 date from the entire data set and apply to all other fields.
Ideally, I would have it so if it was a pay-in-full the the First_Payment field would have the date and the 2nd/3rd fields would say NA. The 2-pay would have 1st and 2nd dates and the 3rd field would say NA. And finally the 3 pay would have all 3 dates.
The end users here are not super data savvy so I'm trying to make this as easy to interpret as possible. Any suggestions would be tremendously appreciated. Thank you!
Using data.table this is a simple one-liner
library(data.table) #v1.9.8+
dcast(setDT(pmts), name + pmt_amount ~ rowid(pmt_amount))
# Using 'pmt_date' as value column. Use 'value.var' to override
# name pmt_amount 1 2 3
# 1: foo 375 1/5/17 3/5/17 5/5/17
# 2: janedoe 995 12/15/16 NA NA
# 3: johndoe 550 9/1/16 11/1/16 NA
dcast converts from long to wide and it accepts expressions. rowid is just adding a row counter per pmt_amount.
You can use tidyr for this.
library(dplyr)
library(tidyr)
pmts <- tibble(
name = c("johndoe", "johndoe", "janedoe", "foo", "foo", "foo"),
pmt_amount = c(550L, 550L, 995L, 375L, 375L, 375L),
pmt_date = lubridate::mdy(c("9/1/16", "11/1/16", "12/15/16", "1/5/17", "3/5/17", "5/5/17"))
)
pmts
#> # A tibble: 6 x 3
#> name pmt_amount pmt_date
#> <chr> <int> <date>
#> 1 johndoe 550 2016-09-01
#> 2 johndoe 550 2016-11-01
#> 3 janedoe 995 2016-12-15
#> 4 foo 375 2017-01-05
#> 5 foo 375 2017-03-05
#> 6 foo 375 2017-05-05
pmts_long <- pmts %>%
group_by(name) %>%
arrange(name, pmt_date) %>%
mutate(pmt = row_number()) %>%
ungroup() %>%
complete(name, nesting(pmt)) %>%
fill(pmt_amount, .direction = "down")
pmts_long
#> # A tibble: 9 x 4
#> name pmt pmt_amount pmt_date
#> <chr> <int> <int> <date>
#> 1 foo 1 375 2017-01-05
#> 2 foo 2 375 2017-03-05
#> 3 foo 3 375 2017-05-05
#> 4 janedoe 1 995 2016-12-15
#> 5 janedoe 2 995 NA
#> 6 janedoe 3 995 NA
#> 7 johndoe 1 550 2016-09-01
#> 8 johndoe 2 550 2016-11-01
#> 9 johndoe 3 550 NA
pmts_wide <- pmts_long %>%
gather("key", "val", -name, -pmt_amount, -pmt) %>%
unite(pmt_number, key, pmt) %>%
spread(pmt_number, val)
pmts_wide
#> # A tibble: 3 x 5
#> name pmt_amount pmt_date_1 pmt_date_2 pmt_date_3
#> * <chr> <int> <date> <date> <date>
#> 1 foo 375 2017-01-05 2017-03-05 2017-05-05
#> 2 janedoe 995 2016-12-15 NA NA
#> 3 johndoe 550 2016-09-01 2016-11-01 NA