I have a data matrix of order 2000 x 20, and I want to choose a specific order of entire rows from the matrix, for example, 1st,7th, 8th, 14th, 15th, 21th, 22th, is such a sequence of rows until the last 2000th rows.
[1, 7, 8, 14, 15, 21, 22, ...]
Manually it's very difficult to select such a sequence, is there is an alternative to do the same task in R? are the for looping is helpful in solving such a problem.
Using the updated question data, something like:
cumsum(rep(c(1,6), 2000/7))
# [1] 1 7 8 14 15
# ...
#[566] 1981 1982 1988 1989 1995
Since your pattern is +1/+6 up until 2000, you can repeat the two values c(1,6) as many times as sum(c(1,6)) goes in to 2000, and then take a cumulative sum.
Try this
mat[sort(c(k <- seq(6, 2000, by = 7), k + 1)),]
You can define your sequence first, using e.g. sequence, and then subset using [].
n = 2000 / 7
s = sequence(nvec = c(1, rep(2,n)), from = c(1, 7*1:n))
# s
# [1] 1 7 8 14 15 21 22 28 29 35 36 42 43 49 50 56 57 63 64 70 71 ...
yourMatrix[s, ]
sequence creates a sequence of sequences of length nvec and of starting point from.
Related
I have the following data frame for a student with homework and exam scores.
> student1
UID Homework_1 Homework_2 Homework_3 Homework_4 Homework_5 Homework_6 Homework_7 Homework_8
10 582493224 59 99 88 10 66 90 50 80
Homework_9 Homework_10 Exam_1 Exam_2 Exam_3 Section
10 16 NA 41 61 11 A
The Homework_10 score is missing, and I need to create a function to impute the NA value with mean or median.
The function messy_impute should have the following arguments:
data : data frame or tibble to be imputed.
center : whether to impute using mean or median.
margin : whether to use row or column to input value (1- use row 2-use column).
For example,
messy_impute(student1,mean,1) should print out
> student1
UID Homework_1 Homework_2 Homework_3 Homework_4 Homework_5 Homework_6 Homework_7 Homework_8
10 582493224 59 99 88 10 66 90 50 80
Homework_9 Homework_10 Exam_1 Exam_2 Exam_3 Section
10 16 **62** 41 61 11 A
since the mean of the rest of the homework is 62.
And, if the mean of the columns (other students) in section A for homework 10 is 50, then
messy_impute(student1,mean,2) should print out
> student1
UID Homework_1 Homework_2 Homework_3 Homework_4 Homework_5 Homework_6 Homework_7 Homework_8
10 582493224 59 99 88 10 66 90 50 80
Homework_9 Homework_10 Exam_1 Exam_2 Exam_3 Section
10 16 **50** 41 61 11 A
since the mean of columns in section A is 50.
Please note the if the margin is 2, then the calculation should be done with the same section.
I'm really stuck on this defining the function.
Base R solution:
# Define function to Impute a row-wise mean (assumes one observation per student):
row_wise_mean_impute <- function(df){
grade_df <- df[,names(df) != "studid"]
return(cbind(df[,c("studid"), drop = FALSE],
replace(grade_df, is.na(grade_df), apply(grade_df, 1, mean, na.rm = TRUE))))
}
# Apply function:
row_wise_mean_impute(student1)
Data:
x <- c(rnorm(85, 50, 3), rnorm(15, 50, 15))
student1 <- cbind(studid = 1010101, data.frame(t(x)))
student1[, 10] <- NA_real_
I am working with R version i386 3.1.1 and RStudio 0.99.442.
I have large datasets of tree species that I've collected from 7 plots, each of which are divided into 5 subplots (i.e. 35 distinct subplots). I am trying to get R to run through my dataset and print the species which are present within each plot.
I thought I could use "aggregate" to apply the "levels" function to the Species data column and have it return the Species present for each Plot and Subplot, however it returns the levels of the entire data frame (for 12 species, total) rather than the 3 or 4 species that are actually present in the Subplot.
To provide a reproducible example of what I'm trying to do, we can use the "warpbreaks" dataset that comes with R.
I convert the 'breaks' variable in warpbreaks to a factor variable to recreate the problem; It thus exemplifies my 'species' variable, whereas 'warpbreaks$wool' would represent 'plot', and 'warpbreaks$tension' would represent 'subplot'.
require(stats)
warpbreaks$breaks = as.factor(warpbreaks$breaks)
aggregate(breaks ~ wool + tension, data = warpbreaks, FUN="levels")
If we look at the warpbreaks data, then for "Plot" A (wool) and "Subplot" L (tension) - the desired script would print the species "26, 30, 54, 25, etc."
breaks wool tension
1 26 A L
2 30 A L
3 54 A L
4 25 A L
5 70 A L
6 52 A L
7 51 A L
8 26 A L
9 67 A L
10 18 A M
11 21 A M
12 29 A M
...
Instead, R returns something of this sort, where it is printing ALL of the levels of the factor variable for ALL of the plots:
wool tension breaks.1 breaks.2 breaks.3 breaks.4 breaks.5 breaks...
1 A L 10 12 13 14 15 ...
2 B L 10 12 13 14 15 ...
3 A M 10 12 13 14 15 ...
4 B M 10 12 13 14 15 ...
5 A H 10 12 13 14 15 ...
6 B H 10 12 13 14 15 ...
How do I get it to print only the factors that are present within that Plot/Subplot combination? Am I totally off in my use of "aggregate"? I'd imagine this is a relatively easy task for an experience R user...
First time stackoverflow post - would appreciate any help or nudges towards the right code!
Many kind thanks.
Try FUN=unique rather than FUN=levels. levels will return every level of the factor, as you have surmised already. unique(...) will only return the unique levels.
y <- aggregate(breaks ~ wool + tension, data = warpbreaks, FUN=unique)
wool tension breaks
1 A L 14, 18, 29, 13, 31, 28, 27, 30
2 B L 15, 4, 17, 9, 19, 23, 10, 26
3 A M 8, 11, 17, 7, 2, 20, 18, 21
4 B M 24, 14, 9, 6, 22, 16, 11, 17
5 A H 21, 11, 12, 8, 1, 25, 16, 5, 14
6 B H 10, 11, 12, 7, 3, 5, 6, 16
NOTE the breaks column is a little weird, as in each row of that column instead of having one value (which makes sense for a dataframe), you have a vector of values. i.e. each cell of that breaks column is NOT a string; it's a vector!
> class(y$wool)
[1] "factor"
> class(y$breaks) # list !!
[1] "list"
> y$breaks[[1]] # first row in breaks
[1] 26 30 54 25 70 52 51 67
Levels: 10 12 13 14 15 16 17 18 19 20 21 24 25 26 27 28 29 30 31 35 36 39 41 42 43 44 51 52 54 67 70
Note that to access the first element of the breaks column, instead of doing y$breaks[1] (like you would with the wool or tension column) you need to do y$breaks[[1]] because of this.
Data frames are not really meant to work like this; a single cell in a dataframe is supposed to have a single value, and most functions will expect a dataframe to conform to this, so just keep this in mind when doing future processing.
If you wanted to convert to a string, use (e.g.) FUN=function (x) paste(unique(x), collapse=', '); then y$breaks will be a column of strings and behaves as normal.
I have two vectors of uneven lengths.
> starts
[1] 1 4 7 11 13 15 18 20 37 41 53 61
> ends
[1] 3 6 10 17 19 35 52 60 63
Each corresponding part in starts and ends are supposed to form a boundary, e.g. (1, 3) for the first, (4, 6) for second, etc. However you will notice that starts has 10 elements, and ends has just 9. What happened is for some anomaly, there may be consecutive starts, e.g. 4th to 6th elements of starts (11, 13, 15) are all smaller than the 4th element of ends (17).
Edit: please note also corresponding ends are not always 1 higher than starts, sample above edited to reflect so i.e. after ends 35, the next starts is 37.
My question is, how to find all these extranuous unpaired starts? My aim is to lengthen ends to be same length as starts, and pair all extranuous starts with a corresponding NA in ends. The actual vector lengths are in thousands, with mismatches in hundreds. I can imagine a nested for loop to address this, but am wondering if there is a more efficient solution.
Edit: the expected result would be (starts unchanged, displayed for comparison):
> starts
[1] 1 4 7 11 13 15 18 20 37 41 53 61
> ends
[1] 3 6 10 NA NA 17 19 35 NA 52 60 63
or equivalent, not particular about format.
> starts = c(1, 4, 7, 11, 15, 19, 23, 27)
> ends = c(3, 5, 14, 22, 25)
> e = ends[findInterval(starts, ends)+1]
> e
[1] 3 5 14 14 22 22 25 NA
> e[duplicated(e, fromLast=T)]=NA
> e
[1] 3 5 NA 14 NA 22 25 NA
findInterval seems to work
Assuming both starts and ends are sorted and that it's only in ends where the values are missing, you might be able to do something as straightforward as:
ends[c(match(starts, ends + 1)[-1], length(ends))]
# [1] 3 6 10 NA 17 19 36 52 60 63
I have a data frame with a string of values, with certain anomalous readings I want to identify. I would like to make a third column in my data frame marking certain readings as "anomaly", and the rest as "normal". Looking over a plot of my data, by eye it seems pretty obvious when I get these odd dips, but I am having trouble figuring out how to get R to recognize the odd readings since the baseline average changes over time. The best I can come up with is three rules to use to classify something as "anomaly".
1: Starting with the second value, if the second value is within a close range of the first value, then mark as "N" for normal in the third column. And so on through the rest of the data set.
2: If the second value represents a large increase or decrease from the first value, mark as "A" for anomaly in the third column.
3: If a value is marked as "A", the following value will be marked as "A" as well if it is within a small range the previous anomalous value. If the following value represents a large increase or decrease from the previous anomalous value, it is to be marked as "N".
This was my best logic I could come up with, but looking at the data below if you can come up with a better idea I'm all for it.
So given a dummy data set:
SampleNum<-1:50
Value <- c(1, 2, 2, 2, 23, 22, 2, 3, 2, -23, -23, 4, 4, 5, 5, 25, 24,
6, 7, 6, 35, 38, 20, 21, 22, -22, 2, 2, 6, 7, 7, 6, 30, 31,
6, 6, 6, 5, 22, 22, 4, 5, 4, 5, 30, 39, 18, 18, 19, 18)
DF<-data.frame(SampleNum,Value)
This is how I might see the final data, with a third column identifying which values are anomalous.
SampleNum Value Name
1 1 N
2 2 N
3 2 N
4 2 N
5 23 A
6 22 A
7 2 N
8 3 N
9 2 N
10 -23 A
11 -23 A
12 4 N
13 4 N
14 5 N
15 5 N
16 25 A
17 24 A
18 6 N
19 7 N
20 6 N
21 35 A
22 38 A
23 20 N
24 21 N
25 22 N
26 -22 A
27 2 N
28 2 N
29 6 N
30 7 N
31 7 N
32 6 N
33 30 A
34 31 A
35 6 N
36 6 N
37 6 N
38 5 N
39 22 A
40 22 A
41 4 N
42 5 N
43 4 N
44 5 N
45 30 A
46 39 A
47 18 N
48 18 N
49 19 N
50 18 N
You need to distinguish anomalies from mixtures of different distributions. This is usually NOT a statistical question but rahter soemthing that comes from domain-specific knowledge. If you plot the desnity estimates from you data you get:
png(); plot( density(DF$Value)) ; dev.off()
So how are we supposed to know that the two values below zero are not real? They are 4% of your sample so applying a rule: "anomalies == items being outside the 99% confidence interval" would not define them as "anomalies. Are these activity measurements of some sort where the instrument should have given a positive value? The much larger bump peaking at 20 is surely not an anomaly by any reasonable definition.
You should do some searching on the topic of statistical prcess control. There are R packages with SPC oriented functions in them.
I have a data set of "X" (values from 0 to 80) and "Y" (values from 0 to 80). I would like to create a new column "Table". I have 36 tables in mind: In groups of 6... They should be grouped according to:
Tables 1-6:ALL Y 11-20... Table 7-12:Y 21-30, Table 13-18:Y 31-40, Table 19-24:Y 41-50, Table 25-30:Y 51-60, Table 31-36:Y 61-70
Table 1: X 21-30 and Tables 7, 13, 19, 25, 31
Table 2: X 31-40 and Tables 8, 14, 20, 26, 32
Table 3: X 41-50 and Tables 9, 15, 21, 27, 33
Table 4: X 51-60 and Tables 10, 16, 22, 28, 34
Table 5: X 61-70 and Tables 11, 17, 23, 29, 35
Table 6: X 71-80 and Tables 12, 18, 24, 30, 36
End Result:
X Y Table
45 13 3
66 59 29
21 70 31
17 66 NA (there is no table for X lower than 21)
Should I be using the If Else function to group the data from the "X" and "Y" into my new "Table", ranging from 1 to 36 or something else? Any help will be appreciated! Thank you!
head(data)
value avg.temp X Y
1 0 6.69 45 13
2 0 6.01 48 14
3 0 7.35 39 15
4 0 5.86 45 15
5 0 6.43 42 16
6 0 5.68 48 16
I think you could use something like this. If your data frame is called df :
df$Table <- NA
df$Table[df$X>=21 & df$X<=30 & df$Y>=11 & df$Y<=20] <- 1
df$Table[df$X>=31 & df$X<=40 & df$Y>=11 & df$Y<=20] <- 2
...
Use math and indexes:
# demo data
x <- data.frame(X = c(45,66,21,17,0,1,21,80,45),Y = c(13,59,70,66,80,11,0,1,27))
# if each GROUP of Y tables was numbered 1-6, aka indexing
x$ytableindex <- ((x$Y-1) - (x$Y-1) %% 10) / 10
# NA if too low
x$ytableindex[x$ytableindex < 1] <- NA
# find lowest table based on Y index
x$ytable <- (0:5*6+1)[x$ytableindex]
# find difference from lowest Y table to arrive at correct table using X
x$xdiff <- floor((x$X - 1) / 10 - 2)
# NA if too low
x$xdiff[x$xdiff < 0] <- NA
# use difference to calculate the correct table, NA's stay NA
x$Table <- x$ytable + x$xdiff