I have a dataframe where the columns represent monthly data and the rows different simulations. the data I am working with accumulates over time so I want to take the difference between the months to get the true value for that month. There are not headers for my data frame
For example:
View(df)=
1 3 4 6 19 23 24 25 26 ...
1 2 3 4 5 6 7 8 9 ...
0 0 2 3 5 7 14 14 14 ...
My plan was to use the diff() function or something like it, but I am having trouble using it on a dataframe.
I have tried:
df1<-diff(df, lag = 1, differences = 1)
but only get zeros.
I am grateful for any advice.
see ?apply. If it's a data frame
apply(df,2,diff)
should work. Also since a dataframe is a list of vectors sapply(df,diff) should work.
Related
I have a data frame that looks like this
data link: https://1drv.ms/t/s!ArOzUuixE-mg6W7zY2Xvgu80dCsL?e=BuP6xM
letters counts
1 AAAAAA 21
2 AAAAAAAA 9
3 AAAAAAAACAAGGA 1
4 AAAAAAAAGAGT 1
5 AAAAAAACA 24
6 AAAAAAACACAAG 1
7 AAAAAAACAGGG 41
8 AAAAAAACAGTCAATCCTA 2
9 AAAAAAAG 48
10 AAAAAAAGCTGT 2
I have millions of rows like this. I have tried the package "smbinning"
but I am not sure how it can be applied to this type of data.
Do you know any other package or how the smbinning might work.
Thank for your time
I have a dataset of Ages for the customer and I wanted to make a frequency distribution by 9 years of a gap of age.
Ages=c(83,51,66,61,82,65,54,56,92,60,65,87,68,64,51,
70,75,66,74,68,44,55,78,69,98,67,82,77,79,62,38,88,76,99,
84,47,60,42,66,74,91,71,83,80,68,65,51,56,73,55)
My desired outcome would be similar to below-shared table, variable names can be differed(as you wish)
Could I use binCounts code into it ? if yes could you help me out using the code as not sure of bx and idxs in this code?
binCounts(x, idxs = NULL, bx, right = FALSE) ??
Age Count
38-46 3
47-55 7
56-64 7
65-73 14
74-82 10
83-91 6
92-100 3
Much Appreciated!
I don't know about the binCounts or even the package it is in but i have a bare r function:
data.frame(table(cut(Ages,0:7*9+37)))
Var1 Freq
1 (37,46] 3
2 (46,55] 7
3 (55,64] 7
4 (64,73] 14
5 (73,82] 10
6 (82,91] 6
7 (91,100] 3
To exactly duplicate your results:
lowerlimit=c(37,46,55,64,73,82,91,101)
Labels=paste(head(lowerlimit,-1)+1,lowerlimit[-1],sep="-")#I add one to have 38 47 etc
group=cut(Ages,lowerlimit,Labels)#Determine which group the ages belong to
tab=table(group)#Form a frequency table
as.data.frame(tab)# transform the table into a dataframe
group Freq
1 38-46 3
2 47-55 7
3 56-64 7
4 65-73 14
5 74-82 10
6 83-91 6
7 92-100 3
All this can be combined as:
data.frame(table(cut(Ages,s<-0:7*9+37,paste(head(s+1,-1),s[-1],sep="-"))))
I've got a mess of data and am trying to efficiently wrangle it into shape. Here's a simplified short sample of the general format of my data.frame right now. The main difference is that I have a few more data labels like Label1 for my sampling units - each has a set of data similar to the data.frame I'm including but in my situation they are all in the same data.frame. I don't think that will complicate the reformatting so I've just included the single sampling unit of mock data here. StatsType levels Ave, Max, and Min are effectively nested within MeasureType.
tastycheez<-data.frame(
Day=rep((1:3),9),
StatsType=rep(c(rep("Ave",3),rep("Max",3),rep("Min",3)),3),
MeasureType=rep(c("Temp","H2O","Tastiness"),each=9),
Data_values=1:27,
Label1=rep("SamplingU1",27))
Ultimately, I would like a data frame where for each sampling unit and each Day there are columns holding the Data_values for my categories, like this:
Day Label1 Ave.Temp Ave.H2O Ave.Tastiness Max.Temp ...
1 SamplingU1 1 10 19 4 ...
2 SamplingU1 2 11 20 5 ...
I think some combination of functions from reshape,dplyr,tidyr, and/or data.table could do the job but I can't figure out how to code it. Here's what I've tried:
First, I spread the tastycheez (yum!), and that got me partway:
test<-spread(tastycheez,StatsType,Data_values)
Now I'm trying to spread it again or to cast, but with no luck:
test2<-spread(test,MeasureType,(Ave,Max,Min))
test2 <- recast(Day ~ MeasureType+c(Ave,Max,Min), data=test)
(I also tried melting the tastycheez but the results were a sticky, gooey mess and my tongue got burnt. that doesn't seem to be the right function for this.)
If you hate my puns please excuse them, I really can't figure this out!
Here are a couple related questions:
Combining two subgroups of data in the same dataframe
How can I spread repeated measures of multiple variables into wide format?
reshape2 You could use dcast from reshape2:
library(reshape2)
dcast(tastycheez,
Day + Label1 ~ paste(StatsType, MeasureType, sep="."),
value.var = "Data_values")
which gives
Day Label1 Ave.H2O Ave.Tastiness Ave.Temp Max.H2O Max.Tastiness Max.Temp Min.H2O Min.Tastiness Min.Temp
1 1 SamplingU1 10 19 1 13 22 4 16 25 7
2 2 SamplingU1 11 20 2 14 23 5 17 26 8
3 3 SamplingU1 12 21 3 15 24 6 18 27 9
tidyr Stealing #DavidArenburg's comment, here's the tidyr way:
library(tidyr)
tastycheez %>%
unite(temp, StatsType, MeasureType, sep = ".") %>%
spread(temp, Data_values)
which gives
Day Label1 Ave.H2O Ave.Tastiness Ave.Temp Max.H2O Max.Tastiness Max.Temp Min.H2O Min.Tastiness Min.Temp
1 1 SamplingU1 10 19 1 13 22 4 16 25 7
2 2 SamplingU1 11 20 2 14 23 5 17 26 8
3 3 SamplingU1 12 21 3 15 24 6 18 27 9
I have a data frame with two columns. The first is a numerical value, the other is a string describing a time. The time format looks like yyyy-mm-dd--hh-mm-ss-?????? (e.g. 2015-03-04--12-11-35-669696), I don't know what the last 6 digits mean. E.g.
y time
1 4.548 2014-08-11--09-07-44-202586
2 4.548 2014-08-11--09-07-54-442586
3 4.548 2014-08-11--09-08-04-522586
4 4.478 2014-08-11--09-08-14-762586
5 4.431 2014-08-11--09-08-24-522586
6 4.446 2014-08-11--09-08-34-922586
7 4.492 2014-08-11--09-08-44-522586
8 4.508 2014-08-11--09-08-54-442586
9 4.486 2014-08-11--09-09-04-202586
10 4.497 2014-08-11--09-09-14-442586
11 4.461 2014-08-11--09-09-24-202586
I want to plot them with
ggplot(df, aes(x=time, y=y)) + geom_line()
But I have the problem, that ggplot doesn't know how to deal with data of class character and in particular with my given time format.
I tried to use AsciiToInt from the pakage {sfsmisc} to convert the strings to numerical values, but it repeats a list of integers for each string (one number for each character, of course).
I can also sort my time strings with mixedsort from the pakage {gtools}, but I don't how to apply it for the plot (also keeping in mind the distance).
Another problem is that I don't want every time string appear as tick at the x-axis, due to I have around 20k rows. Maybe I can solve that problem like in this question, but I cannot check that as long as the first problem occurs.
Can you help me, ploting such data with the time as a numeric-like value on the x-axis?
I loaded your data as a .txt file called time dat. First I convert your data into POSIXct type. To make a cleaner graph for test purposes I omit the seconds field, if you want to add them in just use the commented out line.
library(ggplot2)
timedat<-read.csv("~/Work/Timedat.csv")
timedat
str(timedat)
> str(timedat)
'data.frame': 11 obs. of 3 variables:
$ X : int 1 2 3 4 5 6 7 8 9 10 ...
$ y : num 4.55 4.55 4.55 4.48 4.43 ...
$ time: Factor w/ 11 levels "2014-08-11--09-07-44-202586",..: 1 2 3 4 5 6 7 8 9 10 ...
#timedat$time<-as.POSIXct(as.character(timedat$time),format = "%Y-%m-%d--%H-%M-%S")
timedat$time<-as.POSIXct(as.character(timedat$time),format = "%Y-%m-%d--%H-%M")
qplot(data=timedat,y=y,x=time)+theme_bw()
> timedat
X y time
1 1 4.548 2014-08-11--09-07-44-202586
2 2 4.548 2014-08-11--09-07-54-442586
3 3 4.548 2014-08-11--09-08-04-522586
4 4 4.478 2014-08-11--09-08-14-762586
5 5 4.431 2014-08-11--09-08-24-522586
6 6 4.446 2014-08-11--09-08-34-922586
7 7 4.492 2014-08-11--09-08-44-522586
8 8 4.508 2014-08-11--09-08-54-442586
9 9 4.486 2014-08-11--09-09-04-202586
10 10 4.497 2014-08-11--09-09-14-442586
11 11 4.461 2014-08-11--09-09-24-202586
This produces the following plot with the dates nicely ordered.
I am working with a large dataset and I am trying to first identify clusters of values that meet specific threshold values. My aim then is to only keep clusters of a minimum length. Below is some example data and my progress thus far:
Test = c("A","A","A","A","A","A","A","A","A","A","B","B","B","B","B","B","B","B","B","B")
Sequence = c(1,2,3,4,5,6,7,8,9,10,1,2,3,4,5,6,7,8,9,10)
Value = c(3,2,3,4,3,4,4,5,5,2,2,4,5,6,4,4,6,2,3,2)
Data <- data.frame(Test, Sequence, Value)
Using package evd, I have identified clusters of values >3
C1 <- clusters(Data$Value, u = 3, r = 1, cmax = F, plot = T)
Which produces
C1
$cluster1
4
4
$cluster2
6 7 8 9
4 4 5 5
$cluster3
12 13 14 15 16 17
4 5 6 4 4 6
My problem is twofold:
1) I don't know how to relate this back to the original dataframe (for example to Test A & B)
2) How can I only keep clusters with a minimum size of 3 (thus excluding Cluster 1)
I have looked into various filtering options etc. however they do not cluster data according to a desired threshold, with no options for the minimum size of the cluster either.
Any help is much appreciated.
Q1: relate back to original dataframe: Have a look at Carl Witthoft's answer. He wrote a variant of rle() (seqle() because it allows one to look for integer sequences rather than repetitions): detect intervals of the consequent integer sequences
Q2: only keep clusters of certain length:
C1[sapply(C1, length) > 3]
yields the 2 clusters that are long enough:
$cluster2
6 7 8 9
4 4 5 5
$cluster3
12 13 14 15 16 17
4 5 6 4 4 6