Suppose i have a dataframe with 100 rows and 100 columns.
For each row, if any 2 columns have the same value, then this row should be removed.
For example, if column 1 and 2 are equal, then this row should be removed.
Another example, if column 10 and column 47 are equal, then this row should be removed as well.
Example:
test <- data.frame(x1 = c('a', 'a', 'c', 'd'),
x2 = c('a', 'x', 'f', 'h'),
x3 = c('s', 'a', 'f', 'g'),
x4 = c('a', 'x', 'u', 'a'))
test
x1 x2 x3 x4
1 a a s a
2 a x a x
3 c f f u
4 d h g a
Only the 4th row should be kept.
How to do this in a quick and concise way? Not using for loops....
Use apply to look for duplicates in each row. (Note that this internally converts your data to a matrix for the comparison. If you are doing a lot of row-wise operations I would recommend either keeping it as a matrix or converting it to a long format as in Jack Brookes's answer.)
# sample data
set.seed(47)
dd = data.frame(matrix(sample(1:5000, size = 100^2, replace = TRUE), nrow = 100))
# remove rows with duplicate entries
result = dd[apply(dd, MARGIN = 1, FUN = function(x) !any(duplicated(x))), ]
Tested on this 20x20 dataframe
library(tidyverse)
N <- 20
df <- matrix(as.integer(runif(N^2, 1, 500)), nrow = N, ncol = N) %>%
as.tibble()
df
# # A tibble: 20 x 20
# V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
# <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
# 1 350 278 256 484 486 249 35 308 248 66 493 130 149 2 374 51 370 423 165 388
# 2 368 448 441 62 304 373 38 375 406 463 412 95 174 365 170 113 459 369 62 21
# 3 250 459 416 128 372 67 281 450 48 122 308 56 121 497 498 220 34 4 126 411
# 4 171 306 390 13 395 160 256 258 76 131 471 487 190 492 21 237 380 129 5 30
# 5 402 421 6 401 50 292 470 319 283 178 234 46 176 178 288 499 7 221 123 268
# 6 415 342 132 379 150 35 323 225 246 496 460 478 205 255 460 62 78 207 82 118
# 7 207 52 420 216 9 366 390 382 304 63 427 425 350 112 488 400 328 239 148 40
# 8 392 455 156 386 478 3 359 184 420 138 29 434 31 279 87 233 455 21 181 437
# 9 349 460 498 278 104 93 253 287 124 351 60 333 321 116 19 156 372 168 95 169
# 10 386 73 362 127 313 93 427 81 188 366 418 115 353 412 483 147 295 53 82 188
# 11 272 480 168 306 359 75 436 228 187 279 410 388 62 227 415 374 366 313 187 49
# 12 177 382 233 146 338 76 390 232 336 448 175 79 202 230 317 296 410 90 102 465
# 13 108 433 59 151 8 138 464 458 183 316 481 153 403 193 71 136 27 454 62 439
# 14 421 72 106 442 338 440 476 357 74 108 94 407 453 262 355 356 27 217 243 455
# 15 325 449 151 473 241 11 154 52 77 489 137 279 420 120 165 289 70 128 384 53
# 16 126 189 43 354 233 168 48 285 175 348 404 254 168 126 95 65 493 493 187 228
# 17 26 143 112 107 350 198 353 439 192 158 151 23 326 4 304 162 84 412 499 170
# 18 88 156 222 227 452 233 397 203 478 73 483 241 151 38 176 77 244 396 9 393
# 19 361 486 423 310 153 235 274 204 399 493 422 374 399 10 215 468 322 38 395 390
# 20 417 124 21 220 123 399 354 182 233 24 397 263 182 211 360 419 202 240 363 187
Removing rows with any duplicates
df %>%
group_by(id = row_number()) %>%
gather(col, value, -id) %>%
filter(!any(duplicated(value))) %>%
spread(col, value)
# # A tibble: 11 x 21
# # Groups: id [11]
# id V1 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V2 V20 V3 V4 V5 V6 V7 V8 V9
# <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <int>
# 1 1 350 66 493 130 149 2 374 51 370 423 165 278 388 256 484 486 249 35 308 248
# 2 3 250 122 308 56 121 497 498 220 34 4 126 459 411 416 128 372 67 281 450 48
# 3 4 171 131 471 487 190 492 21 237 380 129 5 306 30 390 13 395 160 256 258 76
# 4 7 207 63 427 425 350 112 488 400 328 239 148 52 40 420 216 9 366 390 382 304
# 5 9 349 351 60 333 321 116 19 156 372 168 95 460 169 498 278 104 93 253 287 124
# 6 12 177 448 175 79 202 230 317 296 410 90 102 382 465 233 146 338 76 390 232 336
# 7 13 108 316 481 153 403 193 71 136 27 454 62 433 439 59 151 8 138 464 458 183
# 8 14 421 108 94 407 453 262 355 356 27 217 243 72 455 106 442 338 440 476 357 74
# 9 15 325 489 137 279 420 120 165 289 70 128 384 449 53 151 473 241 11 154 52 77
# 10 17 26 158 151 23 326 4 304 162 84 412 499 143 170 112 107 350 198 353 439 192
# 11 18 88 73 483 241 151 38 176 77 244 396 9 156 393 222 227 452 233 397 203 478
You can try a series of filters from dplyr. I cooked up some sample data here. If your variables are named then you can use something like the first example. Otherwise the second should work
library(tidyverse)
#> Warning: package 'dplyr' was built under R version 3.5.1
data <- data_frame(
A = c(1,2,3,4,5,6),
B= c(1,3,5,7,9,11),
C = c(2,2,6,8,10,12)
)
data %>%
filter(A != B) %>% # This removed the first row
filter(A != C) # This removed the second row
#> # A tibble: 4 x 3
#> A B C
#> <dbl> <dbl> <dbl>
#> 1 3 5 6
#> 2 4 7 8
#> 3 5 9 10
#> 4 6 11 12
data %>%
filter(.[1] != .[2]) %>%
filter(.[1] != .[3])
#> # A tibble: 4 x 3
#> A B C
#> <dbl> <dbl> <dbl>
#> 1 3 5 6
#> 2 4 7 8
#> 3 5 9 10
#> 4 6 11 12
Related
Sample data
set.seed(16)
aaa <- 1:1000
aaa[round(runif(100,1,1000))] <- NA
aaa.df <- as.data.frame(matrix(aaa, ncol=5))
I want the aaa.df to be split into multiple groups based on which column(s) contains NA value(s), so for example, if 10th, 16th, 200th rows has NA value in the same column, I want these rows to be in one group and so on. It should also work when a. there is no NA values in a row and b. there is multiple NA values in a row.
I also want to keep the original row number when grouping.
Edit: To make it clearer this is the expected output (Obtained using Taufi's answer, but I am still looking for a more elegant way)
[[1]]
# A tibble: 119 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 1 201 401 601 801 1
2 2 202 402 602 802 2
3 3 203 403 603 803 3
4 4 204 404 604 804 4
5 5 205 405 605 805 5
6 6 206 406 606 806 6
7 7 207 407 607 807 7
8 8 208 408 608 808 8
9 9 209 409 609 809 9
10 10 210 410 610 810 10
# ... with 109 more rows
[[2]]
# A tibble: 14 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 20 220 420 620 NA 20
2 32 232 432 632 NA 32
3 47 247 447 647 NA 47
4 70 270 470 670 NA 70
5 85 285 485 685 NA 85
6 92 292 492 692 NA 92
7 129 329 529 729 NA 129
8 132 332 532 732 NA 132
9 137 337 537 737 NA 137
10 151 351 551 751 NA 151
11 152 352 552 752 NA 152
12 168 368 568 768 NA 168
13 178 378 578 778 NA 178
14 181 381 581 781 NA 181
[[3]]
# A tibble: 15 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 11 211 411 NA 811 11
2 37 237 437 NA 837 37
3 62 262 462 NA 862 62
4 82 282 482 NA 882 82
5 83 283 483 NA 883 83
6 89 289 489 NA 889 89
7 107 307 507 NA 907 107
8 115 315 515 NA 915 115
9 116 316 516 NA 916 116
10 117 317 517 NA 917 117
11 118 318 518 NA 918 118
12 165 365 565 NA 965 165
13 176 376 576 NA 976 176
14 189 389 589 NA 989 189
15 200 400 600 NA 1000 200
[[4]]
# A tibble: 1 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 12 212 412 NA NA 12
[[5]]
# A tibble: 16 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 17 217 NA 617 817 17
2 28 228 NA 628 828 28
3 31 231 NA 631 831 31
4 48 248 NA 648 848 48
5 58 258 NA 658 858 58
6 72 272 NA 672 872 72
7 80 280 NA 680 880 80
8 126 326 NA 726 926 126
9 144 344 NA 744 944 144
10 145 345 NA 745 945 145
11 149 349 NA 749 949 149
12 153 353 NA 753 953 153
13 186 386 NA 786 986 186
14 190 390 NA 790 990 190
15 192 392 NA 792 992 192
16 196 396 NA 796 996 196
and so on..
In addition to my previous more brute-force kind of answer, I came up with the following way more elegant one-liner that avoids any unnecessary joins or intermediate assignment steps. Since you already accepted my previous answer, I let that be as it stands and add the conceptually different one-liner below. The idea is to split() the data.frame based on pasted column numbers from which() that indicate the presence of NA.
split(aaa.df,
apply(aaa.df, 1,
function(x) paste(which(is.na(x)), collapse = ",")))
Output
$`1`
V1 V2 V3 V4 V5
77 NA 277 477 677 877
93 NA 293 493 693 893
97 NA 297 497 697 897
109 NA 309 509 709 909
119 NA 319 519 719 919
140 NA 340 540 740 940
154 NA 354 554 754 954
158 NA 358 558 758 958
171 NA 371 571 771 971
172 NA 372 572 772 972
$`1,2,3`
V1 V2 V3 V4 V5
51 NA NA NA 651 851
$`1,3,5`
V1 V2 V3 V4 V5
75 NA 275 NA 675 NA
$`1,4`
V1 V2 V3 V4 V5
194 NA 394 594 NA 994
$`1,4,5`
V1 V2 V3 V4 V5
49 NA 249 449 NA NA
...
and so on ...
A quick, but not very elegant solution would be as follows. Note that the original row number later is in V6.
aaa.df %<>% mutate(Rownum = 1:nrow(aaa.df))
Aux.df <- cbind(is.na(aaa.df[, 1:(ncol(aaa.df) - 1)]), 1:nrow(aaa.df)) %>%
as.data.frame %>%
group_by(V1, V2, V3, V4, V5) %>%
group_split
Sol <- lapply(Aux.df, function(x) inner_join(x, aaa.df, by = c("V6"="Rownum")) %>%
select(V1.y, V2.y, V3.y, V4.y, V5.y, V6))
Output
> Sol
[[1]]
# A tibble: 119 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 1 201 401 601 801 1
2 2 202 402 602 802 2
3 3 203 403 603 803 3
4 4 204 404 604 804 4
5 5 205 405 605 805 5
6 6 206 406 606 806 6
7 7 207 407 607 807 7
8 8 208 408 608 808 8
9 9 209 409 609 809 9
10 10 210 410 610 810 10
# ... with 109 more rows
[[2]]
# A tibble: 14 x 6
V1.y V2.y V3.y V4.y V5.y V6
<int> <int> <int> <int> <int> <int>
1 20 220 420 620 NA 20
2 32 232 432 632 NA 32
3 47 247 447 647 NA 47
4 70 270 470 670 NA 70
5 85 285 485 685 NA 85
6 92 292 492 692 NA 92
7 129 329 529 729 NA 129
8 132 332 532 732 NA 132
9 137 337 537 737 NA 137
10 151 351 551 751 NA 151
11 152 352 552 752 NA 152
12 168 368 568 768 NA 168
13 178 378 578 778 NA 178
14 181 381 581 781 NA 181
....
and so on ...
The function ludridate::yday returns the day of the year as an integer:
> lubridate::yday("2020-07-01")
[1] 183
I would like to be able to calculate the day of the year assuming a different yearly start date. For example, I would like to start all years on July 1st (07-01), such that I could call:
> lubridate::yday("2020-07-01", start = "2020-07-01")
[1] 1
I could call :
> lubridate::yday("2020-07-01") - lubridate::yday("2020-06-30")
[1] 1
But not only this would fail to account for leap years, it would be difficult to account for a date with a 2021 year (or any date that crosses the January 1st threshold for any given year):
> lubridate::yday("2021-01-01") - lubridate::yday("2020-06-30")
[1] -181
After working a little bit with this on my own, this is what I have created:
valiDATE <- function(date) {
stopifnot(`date must take the form of "MM-DD"` = stringr::str_detect(date, "^\\d{2}-\\d{2}$"))
}
days <- function(x, end = "06-30") {
valiDATE(end)
calcdiff <- function(x) {
endx <- glue::glue("{lubridate::year(x)}-{end}")
if(lubridate::yday(x) > lubridate::yday(endx)) {
diff <- ceiling(difftime(x, endx, units = "days"))
} else {
endx <- glue::glue("{lubridate::year(x)-1}-{end}")
diff <- ceiling(difftime(x, endx, units = "days"))
}
unclass(diff)
}
purrr::map_dbl(x, calcdiff)
}
day_vec <- seq(as.Date("2020-07-01"), as.Date("2021-06-30"), by = "days")
days(day_vec)
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
[38] 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
[75] 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
[112] 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148
[149] 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185
[186] 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222
[223] 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259
[260] 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296
[297] 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333
[334] 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365
I would still like to see other solutions. Thanks!
Adding or subtracting months from your date to the desired start of the year can be helpful in this case.
For your example vector of dates day_vec, you can subtract six months from all the dates if you want to start your year on July 1.
day_vec <- seq(as.Date("2020-07-01"), as.Date("2021-06-30"), by = "days")
day_vec2 <- day_vec %m-% months(6) #Substracting because the new year will start 6 months later
yday(day_vec2) #Result is similar to what you desired.
The point to keep in mind is whether your new beginning of the year is before or after the conventional beginning. If your year starts early, you should add months and vice-versa.
Assume I have following Inputs:
Date <- seq.Date(as.Date("2000-01-01"),as.Date("2006-01-01"), by = "quarter")
mat <- matrix(1:730,73,10)
mat <- data.frame(mat)
mat$Time <- c(seq.Date(as.Date("2000-01-01"),as.Date("2002-12-01"), by= "month"),as.Date("2003-01-03") ,seq.Date(as.Date("2003-02-01"),as.Date("2004-12-01"),by ="month"),as.Date("2005-01-02"),seq(as.Date("2005-02-01"),as.Date("2006-01-01"), by ="month"))
mat
And now I would like to get the rows in the matrix which are the same as the date vector. However, some of the dates in the Date vector dont exist. So iwould like to get the closest date. Therefore I tried this:
for(i in 1:length(Date)){
if(Date[i] == mat$Time){
Date[i] <- Date[i]
}else{
Date_Row <- which(abs(mat$Time - Date[i]) == min(abs(mat$Time -Date[i])))
Date[i] <- mat[Date_Row,]
}
}
Date
But it doesn't work. How can I fix this? Thanks!
We can extract the row names and subset the data frame by assigning year and quarter values to the input data, then merging with the reference data that has one observation per quarter.
aFile <- " rowName X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
2000-01-01 1 40 79 118 157 196 235 274 313 352
2000-02-01 2 41 80 119 158 197 236 275 314 353
2000-03-01 3 42 81 120 159 198 237 276 315 354
2000-04-01 4 43 82 121 160 199 238 277 316 355
2000-05-01 5 44 83 122 161 200 239 278 317 356
2000-06-01 6 45 84 123 162 201 240 279 318 357
2000-07-01 7 46 85 124 163 202 241 280 319 358
2000-08-01 8 47 86 125 164 203 242 281 320 359
2000-09-01 9 48 87 126 165 204 243 282 321 360
2000-10-01 10 49 88 127 166 205 244 283 322 361
2000-11-01 11 50 89 128 167 206 245 284 323 362
2000-12-01 12 51 90 129 168 207 246 285 324 363
2001-01-01 13 52 91 130 169 208 247 286 325 364
2002-11-01 35 74 113 152 191 230 269 308 347 386
2002-12-01 36 75 114 153 192 231 270 309 348 387
2003-01-03 37 76 115 154 193 232 271 310 349 388"
df <- read.table(text = aFile,header = TRUE, row.names = "rowName")
referenceDate <- seq.Date(as.Date("2000-01-01"),as.Date("2006-01-01"),
by = "quarter")
library(lubridate)
quarterData <- data.frame(referenceDate,year = year(referenceDate),
qtr = quarter(referenceDate) )
library(dplyr)
df %>% mutate(date = ymd(rownames(df)),
year = year(date),
qtr = quarter(date)) %>%
left_join(.,quarterData)
...and the output:
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 date year qtr referenceDate
1 1 40 79 118 157 196 235 274 313 352 2000-01-01 2000 1 2000-01-01
2 2 41 80 119 158 197 236 275 314 353 2000-02-01 2000 1 2000-01-01
3 3 42 81 120 159 198 237 276 315 354 2000-03-01 2000 1 2000-01-01
4 4 43 82 121 160 199 238 277 316 355 2000-04-01 2000 2 2000-04-01
5 5 44 83 122 161 200 239 278 317 356 2000-05-01 2000 2 2000-04-01
6 6 45 84 123 162 201 240 279 318 357 2000-06-01 2000 2 2000-04-01
7 7 46 85 124 163 202 241 280 319 358 2000-07-01 2000 3 2000-07-01
8 8 47 86 125 164 203 242 281 320 359 2000-08-01 2000 3 2000-07-01
9 9 48 87 126 165 204 243 282 321 360 2000-09-01 2000 3 2000-07-01
10 10 49 88 127 166 205 244 283 322 361 2000-10-01 2000 4 2000-10-01
11 11 50 89 128 167 206 245 284 323 362 2000-11-01 2000 4 2000-10-01
12 12 51 90 129 168 207 246 285 324 363 2000-12-01 2000 4 2000-10-01
13 13 52 91 130 169 208 247 286 325 364 2001-01-01 2001 1 2001-01-01
14 35 74 113 152 191 230 269 308 347 386 2002-11-01 2002 4 2002-10-01
15 36 75 114 153 192 231 270 309 348 387 2002-12-01 2002 4 2002-10-01
16 37 76 115 154 193 232 271 310 349 388 2003-01-03 2003 1 2003-01-01
>
Filter to dates near start of quarter
The reference dates in the OP are at the start of each quarter. Solutions for subsetting the joined data rely on this assumption.
Now that we've joined the data, if we want to subset to only the items early in the quarter, we can filter() based on the difference between date and referenceDate to keep those rows that are within the first 5 days of the quarter.
df %>% mutate(date = ymd(rownames(df)),
year = year(date),
qtr = quarter(date)) %>%
left_join(.,quarterData) %>%
filter(.,(date - referenceDate) < 5)
...and the output:
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 date year qtr referenceDate
1 1 40 79 118 157 196 235 274 313 352 2000-01-01 2000 1 2000-01-01
2 4 43 82 121 160 199 238 277 316 355 2000-04-01 2000 2 2000-04-01
3 7 46 85 124 163 202 241 280 319 358 2000-07-01 2000 3 2000-07-01
4 10 49 88 127 166 205 244 283 322 361 2000-10-01 2000 4 2000-10-01
5 13 52 91 130 169 208 247 286 325 364 2001-01-01 2001 1 2001-01-01
6 37 76 115 154 193 232 271 310 349 388 2003-01-03 2003 1 2003-01-01
>
Filtering to a date beyond the first few days of quarter
If the first day in a quarter falls outside the criteria above, or if the input data includes multiple days that meet the filter criteria, another approach is to create a unique sequential number representing sorted dates within a year and quarter, and selecting the first item in the sequence.
# filter first obs in quarter
df %>% mutate(date = ymd(rownames(df)),
year = year(date),
qtr = quarter(date)) %>%
left_join(.,quarterData) %>%
arrange(.,year,qtr,date) %>%
group_by(year,qtr) %>%
mutate(quarterSequence = seq_along(qtr)) %>%
filter(quarterSequence == 1)
...and the output:
# A tibble: 7 x 15
# Groups: year, qtr [7]
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 date year
<int> <int> <int> <int> <int> <int> <int> <int> <int> <int> <date> <dbl>
1 1 40 79 118 157 196 235 274 313 352 2000-01-01 2000
2 4 43 82 121 160 199 238 277 316 355 2000-04-01 2000
3 7 46 85 124 163 202 241 280 319 358 2000-07-01 2000
4 10 49 88 127 166 205 244 283 322 361 2000-10-01 2000
5 13 52 91 130 169 208 247 286 325 364 2001-01-01 2001
6 35 74 113 152 191 230 269 308 347 386 2002-11-01 2002
7 37 76 115 154 193 232 271 310 349 388 2003-01-03 2003
# … with 3 more variables: qtr <int>, referenceDate <date>, quarterSequence <int>
>
A simpler approach: use the original data to create reference dates
We can solve the problem posed in the original post without joining one set of dates to another. How? We use lubridate functions to create the first day of the quarter for each row by parsing the year and quarter values from the dates provided in the row names of the original data frame.
# read same data file as top of this answer
df <- read.table(text = aFile,header = TRUE, row.names = "rowName")
library(lubridate)
library(dplyr)
df %>%
mutate(date = ymd(rownames(.)),
referenceDate = ymd(sprintf("%4d-%02d-%02d",year(date),
(quarter(date)-1)*3+1,1))) %>%
filter(.,(date - referenceDate) < 5)
...and the output:
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 date referenceDate
1 1 40 79 118 157 196 235 274 313 352 2000-01-01 2000-01-01
2 4 43 82 121 160 199 238 277 316 355 2000-04-01 2000-04-01
3 7 46 85 124 163 202 241 280 319 358 2000-07-01 2000-07-01
4 10 49 88 127 166 205 244 283 322 361 2000-10-01 2000-10-01
5 13 52 91 130 169 208 247 286 325 364 2001-01-01 2001-01-01
6 37 76 115 154 193 232 271 310 349 388 2003-01-03 2003-01-01
I copy and pasted the top few rows of your data into an excel spreadsheet, then exported it to a csv to read into R as the variable Book1
I used your same code but changed the variable for clarity
Datetofind <- seq.Date(as.Date("2000-01-01"),as.Date("2006-01-01"), by = "quarter")
I got the dataset into a tibble to use lubridate and tidyverse the code below got the column into a Date format
Book1$Date <- ymd(Book1$Date)
Now I just used dplyr to filter the dates in your original datasets and return only the rows that match the quarters.
Book1 %>%
filter(Date %in% Datetofind)
That got me the data below
Date X1 X2 X3 X4 X5 X6 X7 X8 X9 X10
<date> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
2000-01-01 1 40 79 118 157 196 235 274 313 352
2000-04-01 4 43 82 121 160 199 238 277 316 355
2000-07-01 7 46 85 124 163 202 241 280 319 358
2000-10-01 10 49 88 127 166 205 244 283 322 361
2001-01-01 13 52 91 130 169 208 247 286 325 364
I am trying to creating a training dataframe for fitting my model. The dataframe I am working with is a nested dataframe. Using createDataPartition , I have created a list of indexes. But I am having trouble subsetting the dataframe with said list.
Here is what the object partitionindex created by caret::createDataParition looks like:
partitionindex
[[1]]
[[1]]$Resample1
[1] 4 5 6 8 9 10 11 12 14 15 17 18 20 21 23 28 30 32 34 38 39 41 42 46
[25] 47 48 50 52 53 56 57 58 59 60 64 66 67 70 73 75 76 77 78 82 85 87 90 95
[49] 97 99 105 106 110 113 114 116 117 118 119 120 123 124 126 128 129 130 132 134 135 137 139 141
[73] 142 143 144 145 146 148 149 151 153 154 155 157 158 164 165 167 170 174 176 178 182 183 184 186
[97] 189 190 191 193 194 197 198 200 201 202 203 206 210 211 212 213 214 216 219 221 222 223 226 232
[121] 236 237 241 243 247 248 251 254 255 256 258 262 263 264 269 270 271 274 276 277 280 281 284 291
[145] 292 293 295 296 297 299 300 301 302 303 304 309 314 317 318 319 320 323 324 327 328 329 339 341
[169] 342 343 344 345 349 350 351 353 354 355 356 360 361 363 364 365 367 370 371 375 379 380
[[2]]
[[2]]$Resample1
[1] 1 2 4 5 7 8 9 10 14 17 19 22 24 26 28 29 31 32 34 36 37 42 44 45
[25] 47 48 49 51 52 53 56 58 65 66 67 68 72 74 75 77 78 81 83 86 95 96 98 100
[49] 102 104 105 106 110 113 114 115 118 119 122 123 124 125 128 129 130 132 135 137 142 144 145 147
[73] 149 150 151 152 158 160 161 163 165 168 169 170 171 175 176 180 183 186 187 188 191 194 196 199
[97] 203 205 206 207 208 209 210 211 213 215 218 220 221 222 224 225 227 228 231 233 240 241 242 243
[121] 247 248 250 251 254 255 256 257 258 262 263 264 267 268 269 270 272 273 277 278 282 285 286 288
[145] 289 290 292 293 294 295 296 300 301 302 304 305 307 308 312 314 315 316 317 321 323 328 329 332
[169] 333 335 336 339 341 343 344 345 347 348 349 354 355 359 360 362 363 366 369 374 375 376 377
[[3]]
[[3]]$Resample1
[1] 5 8 10 12 17 22 25 26 27 30 32 33 34 36 38 39 42 44 45 46 47 51 52 57
[25] 58 59 62 64 66 70 71 73 75 78 81 82 83 84 86 89 90 95 96 97 98 100 103 104
[49] 105 108 109 111 112 113 114 117 119 120 121 123 124 127 130 131 132 133 137 139 140 141 144 148
[73] 149 150 151 153 154 155 156 157 159 160 163 164 167 168 170 172 173 176 178 179 181 182 184 186
[97] 187 188 189 190 191 207 208 212 214 215 219 220 222 223 227 230 233 234 238 248 250 251 252 253
[121] 256 258 260 261 262 264 265 266 267 270 271 272 275 278 281 285 288 289 291 293 295 297 298 302
[145] 303 305 306 308 312 314 315 318 319 320 321 323 325 326 329 332 333 334 335 336 338 342 343 345
[169] 347 348 349 350 351 352 360 361 363 364 365 366 368 369 370 371 372 374 375 376 377 378
[[4]]
[[4]]$Resample1
[1] 1 2 3 4 5 6 7 8 10 12 14 15 18 19 20 22 23 25 26 27 28 30 31 34
[25] 37 38 40 44 45 46 47 49 50 51 52 59 62 64 66 68 70 71 72 73 75 76 79 80
[49] 81 83 84 86 88 89 91 92 94 95 96 97 99 100 102 105 108 109 112 119 125 126 129 130
[73] 132 134 137 139 140 141 145 150 153 155 156 158 159 162 163 170 178 179 181 182 184 185 187 188
[97] 190 191 192 194 196 197 199 201 205 206 207 218 219 220 223 229 230 231 232 237 238 240 241 242
[121] 244 245 247 248 249 251 252 253 257 258 260 261 263 264 265 266 270 271 273 275 276 283 285 289
[145] 290 291 294 298 299 300 302 303 304 306 307
And the nested dataframe:
> nested_df
# A tibble: 4 x 2
# Groups: League [4]
League data
<chr> <list<df[,133]>>
1 F1 [380 x 133]
2 E0 [380 x 133]
3 SP1 [380 x 133]
4 D1 [308 x 133]
I tried something like this but to no avail:
nested_df%>%
mutate(train = data[map(data,~.x[partitionindex,])])
Error in x[i] : invalid subscript type 'list'
Is there a solution involving purrr::map or lappy?
I think this could work, with purrr::pmap
nested_df %>%
ungroup() %>% # make sure the table is not grouped
mutate(i = row_number()) %>%
mutate(train = pmap(
.,
function(data, i, ...) {
data[partitionindex[[i]]$Resample1,]
}
)) %>%
select(-i)
I have downloaded the historical stock prices of a list of 218 stocks. I want check whether it is populated with the the most recent date or not. I have written a function to that effect, by name check.date
function(snlq){
j <- 1;
for(i in 1:length(snlq)){
ind <- index(snlq[[i]])
if(identical(ind[length(ind)],"2018-05-04") == FALSE){
s[j] <- i
j <- j+1
}
}
return(s);
}
snlq is list of stocks with length 218 and of class list
But when I run it, I get the following output:
check.date(snlq)
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
[33] 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
[65] 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
[97] 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
[129] 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160
[161] 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192
[193] 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 356 358 359 360 361 362
[225] 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394
[257] 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426
[289] 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458
[321] 459 460 461 462 463 464 465 466 467 468 469 470
How can the output be of length more than 218? Also I have checked that snlq[[1]] is up to date; then why is 1 in the output?
This might seem like a simple for loop problem, but is perplexing me.
Very many thanks for your time and effort...
It seems the problem is that s is not created in scope in which it is updated and used. #Dave2e has correctly pointed out in above comment. The most logical error seems to me is that s has been created in global space that's why your function is not giving error, otherwise your function would have not run.
There are many ways to fix the problem. One of the option can be as:
check.date <- function(snlq){
j <- 1;
ss <- integer() #declare before use in function scope
for(i in 1:length(snlq)){
ind <- index(snlq[[i]])
if(identical(ind[length(ind)],"2018-05-04") == FALSE){
s = c(s,j) #Kind of adding an element to vector s
j <- j+1
}
}
return(s);
}
I cannot check this result without a reproducible example, but I think this will simplify your function greatly.
check.data <- function(input, today) {
result <- sapply(input, function(x) {
ind <- index(x)
!identical(ind[length(ind)], today)
})
which(result)
}