I have collected data from sensors and need to split the data into 1 day data frames automatically.
This is how my data looks like:
2 2021-10-20 20:17:14 151 -135.9 8.304 -339.5 8.175 23.13232 78.95514 97.10153
3 2021-10-20 20:27:15 152 -136.9 8.302 -340.6 8.175 23.89337 86.71063 98.07861
4 2021-10-20 20:37:15 153 -138.2 8.302 -340.5 8.177 23.00682 80.71004 96.15726
5 2021-10-20 20:47:16 154 -138.8 8.302 -341.0 8.176 23.76786 83.38557 98.30032
I used tibbletime and dplyr to get a tbl_time object.
So far I was able to create a Subset for day1 and manually it would be easy to do it for day 2 and 3 etc. In the end I will have like a year of data, so this will be a pain in the a to do manually.
Here's the line of code I used:
day1<- filter_time(bio2_table, time_formula = '2021-10-20' ~ '2021-10-20')
I'm a r noob but I still want to believe there is a way that r does all the work itself.
Thanks!
What data type is your date object?
The first step is to get the date format right, then you can group_split() to get a list of data frames, each of which represents a specific date.
If interested in this approach, you could look at the setNames() function to name each data frame.
data %>%
mutate(date = as.Date(format(date, "%y-%m-%d")))%>%
group_split(date)
This returns a data frame with vectors as your grouped data:
t(aggregate( dat[,-2], by=list(dat$V2), list ))
[,1] [,2] [,3]
Group.1 "2021-10-20" "2021-10-21" "2021-10-22"
V1 Integer,4 6 7
V3 Character,4 "20:47:16" "20:47:16"
V4 Integer,4 155 156
V5 Numeric,4 -135.8 -131.8
V6 Numeric,4 8.306 10.302
V7 Numeric,4 -339 -347
V8 Numeric,4 8.166 8.187
V9 Numeric,4 25.76789 15.76786
V10 Numeric,4 91.38557 87.38557
V11 Numeric,4 102.3003 111.3003
# OR
aggregate( dat[,-2], by=list(dat$V2), list )
Group.1 V1 V3
1 2021-10-20 2, 3, 4, 5 20:17:14, 20:27:15, 20:37:15, 20:47:16
2 2021-10-21 6 20:47:16
3 2021-10-22 7 20:47:16
V4 V5 V6
1 151, 152, 153, 154 -135.9, -136.9, -138.2, -138.8 8.304, 8.302, 8.302, 8.302
2 155 -135.8 8.306
3 156 -131.8 10.302
V7 V8
1 -339.5, -340.6, -340.5, -341.0 8.175, 8.175, 8.177, 8.176
2 -339 8.166
3 -347 8.187
V9 V10
1 23.13232, 23.89337, 23.00682, 23.76786 78.95514, 86.71063, 80.71004, 83.38557
2 25.76789 91.38557
3 15.76786 87.38557
V11
1 97.10153, 98.07861, 96.15726, 98.30032
2 102.3003
3 111.3003
Data
dat <- structure(list(V1 = 2:7, V2 = c("2021-10-20", "2021-10-20", "2021-10-20",
"2021-10-20", "2021-10-21", "2021-10-22"), V3 = c("20:17:14",
"20:27:15", "20:37:15", "20:47:16", "20:47:16", "20:47:16"),
V4 = 151:156, V5 = c(-135.9, -136.9, -138.2, -138.8, -135.8,
-131.8), V6 = c(8.304, 8.302, 8.302, 8.302, 8.306, 10.302
), V7 = c(-339.5, -340.6, -340.5, -341, -339, -347), V8 = c(8.175,
8.175, 8.177, 8.176, 8.166, 8.187), V9 = c(23.13232, 23.89337,
23.00682, 23.76786, 25.76789, 15.76786), V10 = c(78.95514,
86.71063, 80.71004, 83.38557, 91.38557, 87.38557), V11 = c(97.10153,
98.07861, 96.15726, 98.30032, 102.30032, 111.30032)), class = "data.frame", row.names = c(NA,
-6L))
dat
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10
1 2 2021-10-20 20:17:14 151 -135.9 8.304 -339.5 8.175 23.13232 78.95514
2 3 2021-10-20 20:27:15 152 -136.9 8.302 -340.6 8.175 23.89337 86.71063
3 4 2021-10-20 20:37:15 153 -138.2 8.302 -340.5 8.177 23.00682 80.71004
4 5 2021-10-20 20:47:16 154 -138.8 8.302 -341.0 8.176 23.76786 83.38557
5 6 2021-10-21 20:47:16 155 -135.8 8.306 -339.0 8.166 25.76789 91.38557
6 7 2021-10-22 20:47:16 156 -131.8 10.302 -347.0 8.187 15.76786 87.38557
V11
1 97.10153
2 98.07861
3 96.15726
4 98.30032
5 102.30032
6 111.30032
I am given a large data-table that needs to be aggregated according to the first column:
The problem is the following:
For several columns, one just has to form the sum for each category (given in column 1)
For other columns, one has to calculate the mean
There is a 1-1 correspondence between the entries in the first and second columns. Such that the entries of the second column should be kept.
The following is a possible example of such a data-table. Let's assume that columns 3-9 need to be summed up and columns 10-12 need to be averaged.
library(data.table)
set.seed(1)
a<-matrix(c("cat1","text1","cat2","text2","cat3","text3"),nrow=3,byrow=TRUE)
M<-do.call(rbind, replicate(1000, a, simplify=FALSE)) # where m is your matrix
M<-cbind(M,matrix(sample(c(1:100),3000*10,replace=TRUE ),ncol=10))
M <- as.data.table(M)
The result should be a table of the form
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
1: cat1 text1 27 81 78 95 27 22 12 76 18 76
2: cat2 text2 38 48 70 100 11 97 8 53 56 33
3: cat3 text3 58 18 66 24 14 73 18 27 92 70
but with entries the corresponding sums respective averages.
M[, names(M)[-c(1,2)] := lapply(.SD, as.numeric),
.SDcols = names(M)[-c(1,2)]][,
c(lapply(.SD[, ((3:9)-2), with=FALSE], sum),
lapply(.SD[, ((10:12)-2), with=FALSE], mean)),
by = eval(names(M)[c(1,2)])]
#> V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
#> 1: cat1 text1 51978 49854 48476 49451 49620 49870 50248 50.193 51.516 49.694
#> 2: cat2 text2 50607 50097 50572 50507 48960 51419 48905 49.700 49.631 48.863
#> 3: cat3 text3 51033 50060 49742 50345 51532 51299 50957 50.192 50.227 50.689
Unable to find the solution to my problem.
I am trying to read a text file using r. File contains a single row and separated by number of characters.
000341656.0000000000000000004.6000000000000000009.0000000000000000050.9566787004000000052.0000000000000000072.8621215573000000007.0000000000000000050.0361010830000000047.2490974729000000054.5560183531000000006.0000000000000000049.9711191336000000047.0397111913000000043.1488475260000000023.0000000000000000046.6281588448000000040.1516245487000000038.4653540241000000002.0000000000000000046.2129963899000000041.9963898917000000037.3850068798000000030.0000000000000000046.0144404332000000040.0324909747000000027.0930952140000000003.0000000000000000043.3971119134000000032.4801444043000000010.4757238771
First value is of 20 digit floating.9 digit followed by 10 decimal digit.
The file contains between 22 and 30 values, each 20 digits long (decimal digit set to '.')
What i am unable to figure out how to get rid of this extra 0.
Any lead of help is highly appreciable.
You can read data in a fixed-width format with read.fwf:
> read.fwf("./d.txt", widths=rep(20,30))
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
1 341656 4.6 9 50.95668 52 72.86212 7 50.0361 47.2491 54.55602 6 49.97112
2 341656 4.6 9 50.95668 52 72.86212 7 50.0361 47.2491 54.55602 6 49.97112
V13 V14 V15 V16 V17 V18 V19 V20 V21 V22
1 47.03971 43.14885 23 46.62816 40.15162 38.46535 2 46.213 41.99639 37.38501
2 47.03971 43.14885 23 46.62816 40.15162 38.46535 2 46.213 41.99639 37.38501
V23 V24 V25 V26 V27 V28 V29 V30
1 30 46.01444 40.03249 27.0931 3 43.39711 32.48014 10.47572
2 30 46.01444 40.03249 27.0931 3 43.39711 32.48014 10.47572
You need to know how many fields and how big they are. You didnt say how many lines in the file but I copied your line in twice (hence the duplication).
library(stringi)
stri_match_all_regex(
"000341656.0000000000000000004.6000000000000000009.0000000000000000050.9566787004000000052.0000000000000000072.8621215573000000007.0000000000000000050.0361010830000000047.2490974729000000054.5560183531000000006.0000000000000000049.9711191336000000047.0397111913000000043.1488475260000000023.0000000000000000046.6281588448000000040.1516245487000000038.4653540241000000002.0000000000000000046.2129963899000000041.9963898917000000037.3850068798000000030.0000000000000000046.0144404332000000040.0324909747000000027.0930952140000000003.0000000000000000043.3971119134000000032.4801444043000000010.4757238771",
".{20}"
) %>%
unlist() %>%
as.numeric()
## [1] 341656.00000 4.60000 9.00000 50.95668 52.00000
## [6] 72.86212 7.00000 50.03610 47.24910 54.55602
## [11] 6.00000 49.97112 47.03971 43.14885 23.00000
## [16] 46.62816 40.15162 38.46535 2.00000 46.21300
## [21] 41.99639 37.38501 30.00000 46.01444 40.03249
## [26] 27.09310 3.00000 43.39711 32.48014 10.47572
Also:
as.numeric(readChar("~/Data/20.txt", rep(20, file.size("~/Data/20.txt")/20)))
I wanna skip the first three columns. Couldn't quite understand the posts about colClasses because I'm new to R.
YDL025C YDL025C 1 -0.1725 -0.5375 -0.4970 -0.3818 -0.5270 -0.4260 -0.6929 -0.4020 -0.3263 -0.3373 -0.3532 -0.2771 -0.2732 -0.3307 -0.4660 -0.4314 -0.3135
YKL032C YKL032C 1 -0.2364 0.0794 0.1678 0.2389 0.3847 0.2625 0.1889 0.2681 0.0363 -0.1992 -0.0521 -0.0307 0.0584 0.2817 0.2239 -0.0253 0.0751
If you have to use read.table and you want to filter on the way in, you can use col.classes as follows. You have 20 columns. Say the first 2 are character, rest are numeric and you want to drop 4,5,6. You construct a vector of length 20 detailing that information. The NULL will not pull in those columns.
x<- read.table(file="datat.txt",
colClasses = c(rep("character", 2),
rep("numeric", 1),
rep("NULL", 3),
rep("numeric", 14)),
header = FALSE)
x
V1 V2 V3 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
1 YDL025C YDL025C 1 -0.3818 -0.5270 -0.4260 -0.6929 -0.4020 -0.3263 -0.3373 -0.3532 -0.2771 -0.2732 -0.3307 -0.4660 -0.4314 -0.3135
2 YKL032C YKL032C 1 0.2389 0.3847 0.2625 0.1889 0.2681 0.0363 -0.1992 -0.0521 -0.0307 0.0584 0.2817 0.2239 -0.0253 0.0751
As commented above, easier to remove the columns after reading in. For example:
mydf <- read.table("mydf.txt")
Then,
mydf[, 4:ncol(mydf)]
will remove the first 3 columns.
I would like to generate an string output into a list if some values are met. I have a table that looks like this:
grp V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17
1: 1 go.1 142 144 132 134 0 31 11 F D T hy al qe 34 6 3
2: 2 go.1 313 315 303 305 0 31 11 q z t hr ye er 29 20 41
3: 3 go.1 316 318 306 308 0 31 11 f w y hu er es 64 43 19
4: 4 go.1 319 321 309 311 0 31 11 r a y ie uu qr 26 22 20
5: 5 go.1 322 324 312 314 0 31 11 g w y hp yu re 44 7 0
I'm using this function to generate a desired output:
library(IRanges); library(data.table)
rangeFinder = function(x){
x.ir = reduce(IRanges(x$V2, x$V3))
max.idx = which.max(width(x.ir))
ans = data.table(out = x[1,1],
start = start(x.ir)[max.idx],
end = end(x.ir)[max.idx])
return(ans)}
rangeFinder(x.out)
out start end
1: 1 313 324
I would also like to generate a list with the letters (from column V9-V11) in the between the start and end output from rangeFinder.
For example, the output should look like this.
out
[[go.1]]
[1] "qztfwyraygwy"
rangeFinder is looking at values in column V2 and V3 and printing the longest match of numbers. Notice how "FDT" is not included in the list output even though rangeFinder produced an output from 313-324 (and not from 142-324). How can I get the desired output?
reduce has an argument with.revmap to add a "metadata" column (accessible with mcols()) to the object. This associates with each reduced range the indexes of the original range that map to the reduced range, as an IntegerList class, basically a list where all elements are guaranteed to be integer vectors. So these are the rows you're interested in
ir <- with(x, IRanges(V2, V3))
r <- reduce(ir, with.revmap=TRUE)
i <- unlist(mcols(r)[which.max(width(r)), "revmap"])
and the data character string can be munged with something like
j <- paste0("V", 9:11)
paste0(as.matrix(x[i, j, drop=FALSE]), collapse="")
It's better to ask your questions about IRanges on the Bioconductor mailing list; no subscription required.
with.revmap is a convenience argument added relatively recently; I think
h = findOverlaps(ir, r)
i = queryHits(h)[subjectHits(h) == which.max(width(r))]
is a replacement.