cleaning the data R - r

I've got the final data DS such as :
|user_id
40 33
70 50
93 67
106 77
136 91
144 97
160 105
176 113
195 128
207 132
211 134
229 142
280 159
338 187
407 232
425 248
442 259
446 261
469 277
470 278
588 353
590 355
594 358
598 362
609 369
615 375
626 381
633 386
652 399
657 402
735 452
751 464
758 470
760 471
769 478
774 480
806 493
821 501
825 505
856 526
876 536
886 540
890 542
894 543
903 549
919 556
921 558
932 562
The fist column is a what left of line numbers I suppose, after many data manipulations,
and I'd like to drop them, nice, efficient way, and replace it with normal order numbers , 1,2,3,4,5 etc.
I did try to use :
aggr.cid <-aggregate(cbind(DS$user_id), by=list(CustID = DS$user_id),
function(x) x[1])
But instead of getting 1 line I'm getting two, with content of "user_id"
I can remove the second one and all will looks as I need but it is a doggy way....


Those are the row names. You can reset them with
rownames(DS) <- NULL

Related

Calculating number combinations through R

I have following data. i want to first find out the most occurring digit on every place value. Obviously one place can have 10 possibilities from 0 to 9. Than i want an option where by i can choose 5 top occurrences or 6 or 7 or 8 top occurrences for e.g. if i choose 5 then the program should take the top 5 occurrences or if i choose 8 then program should leave out the least 2 occurring digits and take all others.
Data example:
076060
693022
585821
980575
438068
766214
051726
060417
822591
015507
635576
180231
212238
417651
631269
720767
348344
532148
748085
474026
380897
512421
749492
423616
950330
930079
097759
638901
319356
683308
818127
880675
256095
639187
339904
945437
799571
466063
428853
397799
782034
462486
739342
879023
419264
793319
603131
315791
351701
151747
365656
982700
348093
793392
946875
912108
070001
780515
222468
345439
234846
227112
757243
341747
480781
906624
868265
388572
947873
898895
452518
738580
217342
849951
437382
247068
743776
562584
636948
049434
139296
688436
443629
I want option of choosing 5, 6,7 or 8 top occurrences and 2 or 3 or 4number combination
Expected results, 2 number combination basis top 8 occurrences and so on.
01
02
03
04
05
06
08
09
21
22
23
24
25
26
28
29
31
32
33
34
35
36
38
39
41
42
43
44
45
46
48
49
61
62
63
64
65
66
68
69
71
72
73
74
75
76
78
79
81
82
83
84
85
86
88
89
91
92
93
94
95
96
98
99
Expected results, 3 number combination basis top 8 occurrences and so on.
010
012
013
015
016
017
018
019
020
022
023
025
026
027
028
029
030
032
033
035
036
037
038
039
040
042
043
045
046
047
048
049
050
052
053
055
056
057
058
059
060
062
063
065
066
067
068
069
080
082
083
085
086
087
088
089
090
092
093
095
096
097
098
099
210
212
213
215
216
217
218
219
220
222
223
225
226
227
228
229
230
232
233
235
236
237
238
239
240
242
243
245
246
247
248
249
250
252
253
255
256
257
258
259
260
262
263
265
266
267
268
269
280
282
283
285
286
287
288
289
290
292
293
295
296
297
298
299
310
312
313
315
316
317
318
319
320
322
323
325
326
327
328
329
330
332
333
335
336
337
338
339
340
342
343
345
346
347
348
349
350
352
353
355
356
357
358
359
360
362
363
365
366
367
368
369
380
382
383
385
386
387
388
389
390
392
393
395
396
397
398
399
410
412
413
415
416
417
418
419
420
422
423
425
426
427
428
429
430
432
433
435
436
437
438
439
440
442
443
445
446
447
448
449
450
452
453
455
456
457
458
459
460
462
463
465
466
467
468
469
480
482
483
485
486
487
488
489
490
492
493
495
496
497
498
499
610
612
613
615
616
617
618
619
620
622
623
625
626
627
628
629
630
632
633
635
636
637
638
639
640
642
643
645
646
647
648
649
650
652
653
655
656
657
658
659
660
662
663
665
666
667
668
669
680
682
683
685
686
687
688
689
690
692
693
695
696
697
698
699
710
712
713
715
716
717
718
719
720
722
723
725
726
727
728
729
730
732
733
735
736
737
738
739
740
742
743
745
746
747
748
749
750
752
753
755
756
757
758
759
760
762
763
765
766
767
768
769
780
782
783
785
786
787
788
789
790
792
793
795
796
797
798
799
810
812
813
815
816
817
818
819
820
822
823
825
826
827
828
829
830
832
833
835
836
837
838
839
840
842
843
845
846
847
848
849
850
852
853
855
856
857
858
859
860
862
863
865
866
867
868
869
880
882
883
885
886
887
888
889
890
892
893
895
896
897
898
899
910
912
913
915
916
917
918
919
920
922
923
925
926
927
928
929
930
932
933
935
936
937
938
939
940
942
943
945
946
947
948
949
950
952
953
955
956
957
958
959
960
962
963
965
966
967
968
969
980
982
983
985
986
987
988
989
990
992
993
995
996
997
998
999
code i have tried
getwd()
setwd("C:/Users/aziq/Desktop")
library(xlsx)
x <- read.xlsx("numbers.xlsx","Sheet1")
generate_combinations <- function(x, pos, n) {
#select first pos characters from each string
#split each character and create a matrix
mat <- do.call(rbind, strsplit(substr(x, 1, pos), ''))
#Find top n occurrence in each column of matrix
tmp <- apply(mat, 2, function(x) tail(names(sort(table(x))), n))
#Create all combinations of top occurrences.
do.call(expand.grid, asplit(tmp, 2))
}
generate_combinations(x, 2, 8)
nrow(generate_combinations(x, 2, 8))
Error it is showing
Error in asplit(tmp, 2) : dim(x) must have a positive length
Dput results:
> dput(x)
structure(list(X076060 = c("693022", "585821", "980575", "438068",
"766214", "051726", "060417", "822591", "015507", "635576", "180231",
"212238", "417651", "631269", "720767", "348344", "532148", "748085",
"474026", "380897", "512421", "749492", "423616", "950330", "930079",
"097759", "638901", "319356", "683308", "818127", "880675", "256095",
"639187", "339904", "945437", "799571", "466063", "428853", "397799",
"782034", "462486", "739342", "879023", "419264", "793319", "603131",
"315791", "351701", "151747", "365656", "982700", "348093", "793392",
"946875", "912108", "070001", "780515", "222468", "345439", "234846",
"227112", "757243", "341747", "480781", "906624", "868265", "388572",
"947873", "898895", "452518", "738580", "217342", "849951", "437382",
"247068", "743776", "562584", "636948", "049434", "139296", "688436",
"443629")), class = "data.frame", row.names = c(NA, -82L))

We can write a function :
generate_combinations <- function(x, pos, n) {
if(pos == 1) {
return(data.frame(Var1 = names(sort(table(substr(x, 1, pos)),
= decreasing = TRUE)[1:n])))
}
#select first pos characters from each string
#split each character and create a matrix
mat <- do.call(rbind, strsplit(substr(x, 1, pos), ''))
#Find top n occurrence in each column of matrix
tmp <- apply(mat, 2, function(x) tail(names(sort(table(x))), n))
#Create all combinations of top occurrences.
do.call(expand.grid, asplit(tmp, 2))
}
generate_combinations(x, 2, 8)
# Var1 Var2
#1 0 2
#2 2 2
#3 8 2
#4 6 2
#5 9 2
#6 3 2
#7 4 2
#8 7 2
#9 0 5
#10 2 5
#...
#...
nrow(generate_combinations(x, 2, 8))
#[1] 64
nrow(generate_combinations(x, 3, 8))
#[1] 512
data
x <- c("076060", "693022", "585821", "980575", "438068", "766214",
"051726", "060417", "822591", "015507", "635576", "180231", "212238",
"417651", "631269", "720767", "348344", "532148", "748085", "474026",
"380897", "512421", "749492", "423616", "950330", "930079", "097759",
"638901", "319356", "683308", "818127", "880675", "256095", "639187",
"339904", "945437", "799571", "466063", "428853", "397799", "782034",
"462486", "739342", "879023", "419264", "793319", "603131", "315791",
"351701", "151747", "365656", "982700", "348093", "793392", "946875",
"912108", "070001", "780515", "222468", "345439", "234846", "227112",
"757243", "341747", "480781", "906624", "868265", "388572", "947873",
"898895", "452518", "738580", "217342", "849951", "437382", "247068",
"743776", "562584", "636948", "049434", "139296", "688436", "443629")

Efficiently Finding Sequences Between Vectors of Start and End Numbers in R [duplicate]

This question already has answers here:
Expand ranges defined by "from" and "to" columns
(10 answers)
Closed 2 years ago.
I have two vectors in R, say, start_values and end_values, which contain numbered elements of increasing value. For example:
start_values <- c(88, 241, 394, 545)
end_values <- c(147, 300, 453, 604)
I'm trying to find an efficient (hopefully without writing a loop) that will allow me to obtain a single vector of numbers with sequences of numbers that range from the first element in start_values to the first element in end_values, then from the second element in start_values to the second element in end_values, etc. So in the end, I'd like a vector called sequence_range that looks like this:
sequence_range <- c(seq(88, 147), seq(241, 300), seq(394, 453), seq(545, 604))
which should have output that looks like:
> sequence_range
[1] 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
[30] 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
[59] 146 147 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267
[88] 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296
[117] 297 298 299 300 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
[146] 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447
[175] 448 449 450 451 452 453 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567
[204] 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596
[233] 597 598 599 600 601 602 603 604
I'd appreciate any ideas for efficient techniques to accomplish this so that it's generalizeable to any two vectors of start and end values.
Thanks.

You may use : in an apply on cbinded vectors.
as.vector(apply(cbind(start_values, end_values), 1, function(x) x[1]:x[2]))
# [1] 88 89 90 91 92 93 94 95 96 97 98 99 100 101
# [15] 102 103 104 105 106 107 108 109 110 111 112 113 114 115
# [29] 116 117 118 119 120 121 122 123 124 125 126 127 128 129
# [43] 130 131 132 133 134 135 136 137 138 139 140 141 142 143
# [57] 144 145 146 147 241 242 243 244 245 246 247 248 249 250
# [71] 251 252 253 254 255 256 257 258 259 260 261 262 263 264
# [85] 265 266 267 268 269 270 271 272 273 274 275 276 277 278
# [99] 279 280 281 282 283 284 285 286 287 288 289 290 291 292
# [113] 293 294 295 296 297 298 299 300 394 395 396 397 398 399
# [127] 400 401 402 403 404 405 406 407 408 409 410 411 412 413
# [141] 414 415 416 417 418 419 420 421 422 423 424 425 426 427
# [155] 428 429 430 431 432 433 434 435 436 437 438 439 440 441
# [169] 442 443 444 445 446 447 448 449 450 451 452 453 545 546
# [183] 547 548 549 550 551 552 553 554 555 556 557 558 559 560
# [197] 561 562 563 564 565 566 567 568 569 570 571 572 573 574
# [211] 575 576 577 578 579 580 581 582 583 584 585 586 587 588
# [225] 589 590 591 592 593 594 595 596 597 598 599 600 601 602
# [239] 603 604

mapply works nicely:
> as.vector(mapply(seq,start_values,end_values))
[1] 88 89 90 91 92 93 94 95 96 97 98 99 100 101
[15] 102 103 104 105 106 107 108 109 110 111 112 113 114 115
[29] 116 117 118 119 120 121 122 123 124 125 126 127 128 129
[43] 130 131 132 133 134 135 136 137 138 139 140 141 142 143
[57] 144 145 146 147 241 242 243 244 245 246 247 248 249 250
[71] 251 252 253 254 255 256 257 258 259 260 261 262 263 264
[85] 265 266 267 268 269 270 271 272 273 274 275 276 277 278
[99] 279 280 281 282 283 284 285 286 287 288 289 290 291 292
[113] 293 294 295 296 297 298 299 300 394 395 396 397 398 399
[127] 400 401 402 403 404 405 406 407 408 409 410 411 412 413
[141] 414 415 416 417 418 419 420 421 422 423 424 425 426 427
[155] 428 429 430 431 432 433 434 435 436 437 438 439 440 441
[169] 442 443 444 445 446 447 448 449 450 451 452 453 545 546
[183] 547 548 549 550 551 552 553 554 555 556 557 558 559 560
[197] 561 562 563 564 565 566 567 568 569 570 571 572 573 574
[211] 575 576 577 578 579 580 581 582 583 584 585 586 587 588
[225] 589 590 591 592 593 594 595 596 597 598 599 600 601 602
[239] 603 604

r tm extract doc ids after sort operation

I have mindfreeze around this issue. I am extracting most frequent words from a tm::dtm, like so:
> s1<-sort(rowSums(as.matrix(dtm10[,])), decreasing=TRUE)
I get:
290 429 318 125 128 425 431 153 52 385 144 491 126 423 111 130 492 163 176 391
916 875 860 851 844 823 822 766 759 743 701 700 686 673 670 669 663 658 652 623`
But the doc ids and rowSums are in a tuple.
> s1[2]
429
875
where 429 is doc id and rowsum is 875. I have no further use of the rowSums, how do I get a list of the sorted doc ids? I am looking for a vector output like:
290 429 318 125 128 425 431 153 52 385 144 491 126 423 111 130 492 163 176 391
Many thanks.

s1 is a named vector with names as 290, 429, 318 etc and values like 916,875, 860 and so on.
To extract only the names of s1 , we can use,
names(s1)
which would give :
#290 429 318 125 128 425 431 153 52 385 144 491 126 423 111 130 492 163 176 391

looping a sequence of numbers

I need to generate the following sequence of numbers:
from 101 to 124 then
from 201 to 224
and so on. I need to repeat this pattern 7 times, up to 724.
I know I can simply use
c(101:124,201:224, ...)
but I suspect there is an easier way. Maybe a loop?

We can try seq with sapply
c(sapply(seq(101, 700, by = 100), function(i) i:(i+23)))
Or we can use rep
(101:124) +rep(0:6, each = 24)*100
#[1] 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
#[19] 119 120 121 122 123 124 201 202 203 204 205 206 207 208 209 210 211 212
#[37] 213 214 215 216 217 218 219 220 221 222 223 224 301 302 303 304 305 306
#[55] 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324
#[73] 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
#[91] 419 420 421 422 423 424 501 502 503 504 505 506 507 508 509 510 511 512
#[109] 513 514 515 516 517 518 519 520 521 522 523 524 601 602 603 604 605 606
#[127] 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624
#[145] 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718
#[163] 719 720 721 722 723 724

Using common 'for' loop:
> vect=c(); for(i in seq(from=101,to=701, by=100)) vect = c(vect, seq(from=i,to=i+23))
> vect
[1] 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 201 202 203 204 205 206 207
[32] 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 301 302 303 304 305 306 307 308 309 310 311 312 313 314
[63] 315 316 317 318 319 320 321 322 323 324 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421
[94] 422 423 424 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 601 602 603 604
[125] 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 701 702 703 704 705 706 707 708 709 710 711
[156] 712 713 714 715 716 717 718 719 720 721 722 723 724

We can use either the outer or CJ(cross join), both of which list all the combinations of elements from the two vector arguments. By specifying the reduce function as "+", where for outer, there is a parameter placeholder while for CJ you will have to explicitly use the Reduce function, they should produce the results as wanted. Just to be noted, the outer function will result in a matrix, so we use as.vector to convert it to a 1d array.
as.vector(outer(1:24, (1:7)*100, "+"))
or
Reduce("+", data.table::CJ(1:24, (1:7)*100))

In R, how can one detect consecutive data points within a column of data which have a value >y only when they appear more than x times consecutively?

I am a beginner R user and am trying to find a way to detect x consecutive values within a column of data which have a value >=y. Example: find all instances where 4 or more consecutive data points have a value >=2
The run length encoding rle() command looks promising for identifying these consecutive values:
rle(dataset>=2)
but I cannot find a way to further set the condition for the lengths to be >=4 and the values to be "TRUE".
Any suggestions?

res <- rle(dataset>=2)
which( res$lengths>=4 & res$values==TRUE] )
That will identify them in the compacted representation of the rle result and you will then need to expand that result and pick out starting points for the sequences.

You can simply transform the vector, and use rle on that:
res = rle(runif(1000, 0, 4) >= 2)
res
Run Length Encoding
lengths: int [1:491] 2 2 2 2 3 1 3 2 7 1 ...
values : logi [1:491] TRUE FALSE TRUE FALSE TRUE FALSE ...
To get the indices where in the vector the runs are located, you can use this trick:
res$values = res$lengths > 4
inv_res = inverse.rle(res)
seq_along(inv_res)[inv_res]
[1] 3 4 5 6 7 8 9 10 11 12 13 37 38 39 40 41 42 74
[19] 75 76 77 78 79 80 81 82 83 84 85 108 109 110 111 112 142 143
[37] 144 145 146 147 148 221 222 223 224 225 226 235 236 237 238 239 240 241
[55] 278 279 280 281 282 305 306 307 308 309 310 311 312 313 314 315 316 317
[73] 318 319 342 343 344 345 346 347 414 415 416 417 418 419 430 431 432 433
[91] 434 435 449 450 451 452 453 472 473 474 475 476 477 478 523 524 525 526
[109] 527 545 546 547 548 549 561 562 563 564 565 566 567 568 569 607 608 609
[127] 610 611 612 613 625 626 627 628 629 630 646 647 648 649 650 651 652 699
[145] 700 701 702 703 765 766 767 768 769 770 771 772 773 789 790 791 792 793
[163] 794 795 800 801 802 803 804 810 811 812 813 814 850 851 852 853 854 855
[181] 869 870 871 872 873 879 880 881 882 883 904 905 906 907 908 909 919 920
[199] 921 922 923 949 950 951 952 953 954 955 956 957 958 959 960 961

Resources