I'm trying to get a handle on the ubiquitous which function. Until I started reading questions/answers on SO I never found the need for it. And I still don't.
As I understand it, which takes a Boolean vector and returns a weakly shorter vector containing the indices of the elements which were true:
> seq(10)
[1] 1 2 3 4 5 6 7 8 9 10
> x <- seq(10)
> tf <- (x == 6 | x == 8)
> tf
[1] FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE FALSE FALSE
> w <- which(tf)
> w
[1] 6 8
So why would I ever use which instead of just using the Boolean vector directly? I could maybe see some memory issues with huge vectors, since length(w) << length(tf), but that's hardly compelling. And there are some options in the help file which don't add much to my understanding of possible uses of this function. The examples in the help file aren't of much help either.
Edit for clarity-- I understand that the which returns the indices. My question is about two things: 1) why you would ever need to use the indices instead of just using the boolean selector vector? and 2) what interesting behaviors of which might make it preferred to just using a vectorized Boolean comparison?
Okay, here is something where it proved useful last night:
In a given vector of values what is the index of the 3rd non-NA value?
> x <- c(1,NA,2,NA,3)
> which(!is.na(x))[3]
[1] 5
A little different from DWin's use, although I'd say his is compelling too!
The title of the man page ?which provides a motivation. The title is:
Which indices are TRUE?
Which I interpret as being the function one might use if you want to know which elements of a logical vector are TRUE. This is inherently different to just using the logical vector itself. That would select the elements that are TRUE, not tell you which of them was TRUE.
Common use cases were to get the position of the maximum or minimum values in a vector:
> set.seed(2)
> x <- runif(10)
> which(x == max(x))
[1] 5
> which(x == min(x))
[1] 7
Those were so commonly used that which.max() and which.min() were created:
> which.max(x)
[1] 5
> which.min(x)
[1] 7
However, note that the specific forms are not exact replacements for the generic form. See ?which.min for details. One example is below:
> x <- c(4,1,1)
> which.min(x)
[1] 2
> which(x==min(x))
[1] 2 3
Two very compelling reasons not to forget which:
1) When you use "[" to extract from a dataframe, any calculation in the row position that results in NA will get a junk row returned. Using which removes the NA's. You can use subset or %in%, which do not create the same problem.
> dfrm <- data.frame( a=sample(c(1:3, NA), 20, replace=TRUE), b=1:20)
> dfrm[dfrm$a >0, ]
a b
1 1 1
2 3 2
NA NA NA
NA.1 NA NA
NA.2 NA NA
6 1 6
NA.3 NA NA
8 3 8
# Snipped remaining rows
2) When you need the array indicators.
which could be useful (by the means of saving both computer and human resources) e.g. if you have to filter the elements of a data frame/matrix by a given variable/column and update other variables/columns based on that. Example:
df <- mtcars
Instead of:
df$gear[df$hp > 150] <- mean(df$gear[df$hp > 150])
You could do:
p <- which(df$hp > 150)
df$gear[p] <- mean(df$gear[p])
Extra case would be if you have to filter a filtered elements what could not be done with a simple & or |, e.g. when you have to update some parts of a data frame based on other data tables. This way it is required to store (at least temporary) the indexes of the filtered element.
Another issue what cames to my mind if you have to loop thought a part of a data frame/matrix or have to do other kind of transformations requiring to know the indexes of several cases. Example:
urban <- which(USArrests$UrbanPop > 80)
> USArrests[urban, ] - USArrests[urban-1, ]
Murder Assault UrbanPop Rape
California 0.2 86 41 21.1
Hawaii -12.1 -165 23 -5.6
Illinois 7.8 129 29 9.8
Massachusetts -6.9 -151 18 -11.5
Nevada 7.9 150 19 29.5
New Jersey 5.3 102 33 9.3
New York -0.3 -31 16 -6.0
Rhode Island -2.9 68 15 -6.6
Sorry for the dummy examples, I know it makes not much sense to compare the most urbanized states of USA by the states prior to those in the alphabet, but I hope this makes sense :)
Checking out which.min and which.max gives some clue also, as you do not have to type a lot, example:
> row.names(mtcars)[which.max(mtcars$hp)]
[1] "Maserati Bora"
Well, I found one possible reason. At first I thought it might be the ,useNames option, but it turns out that simple boolean selection does that too.
However, if your object of interest is a matrix, you can use the ,arr.ind option to return the result as (row,column) ordered pairs:
> x <- matrix(seq(10),ncol=2)
> x
[,1] [,2]
[1,] 1 6
[2,] 2 7
[3,] 3 8
[4,] 4 9
[5,] 5 10
> which((x == 6 | x == 8),arr.ind=TRUE)
row col
[1,] 1 2
[2,] 3 2
> which((x == 6 | x == 8))
[1] 6 8
That's a handy trick to know about, but hardly seems to justify its constant use.
Surprised no one has answered this: how about memory efficiency?
If you have a long vector of very sparse TRUE's, then keeping track of only the indices of the TRUE values will probably be much more compact.
I use it quiet often in data exploration. For example if I have a dataset of kids data and see from summary that the max age is 23 (and should be 18), I might go:
sum(dat$age>18)
If that was 67, and I wanted to look closer I might use:
dat[which(dat$age>18)[1:10], ]
Also useful if you're making a presentation and want to pull out a snippet of data to demonstrate a certain oddity or what not.
Related
Is there an easy function, outside of attach/detach, that will break apart a dataframe or data table into its individual vectors with the names of the vectors as the names of the columns in the dataframe.
For example, suppose I have a data frame
x <- data.frame(a=c(1,2,3), b=c(4,5,6), d=c(7,8,9))
Then using the function it would return 3 vectors: a, b, and d. Seems like there should be function to do this but I cannot find it.
One option would be list2env
list2env(x,.GlobalEnv)
a
#[1] 1 2 3
b
#[1] 4 5 6
d
#[1] 7 8 9
Yes, there is a (very old, very standard) function called attach() that does that:
> x <- data.frame(a=c(1,2,3), b=c(4,5,6), d=c(7,8,9))
> attach(x)
> a
[1] 1 2 3
> b
[1] 4 5 6
> d
[1] 7 8 9
>
However, the general consensus is to Don't Do That (TM) as it can a) clutter the environment into which you attach() (usually the global one) and can b) silently overwrite an existing variable (though it warns by default unless you override a switch, see ?attach). There is a counterpart detach() to remove them too. The (aptly named) Section "Good Practice" in the help page for attach has more on all this, including a hint to use on.exit() with detach() where you use attach().
But if you need it, you can use it. Just be aware of Them Dragons.
While two answers have already been posted, I would like to suggest a solution that does not write to the global environment, which is generally considered to be a bad thing (see Dirk´s comments on attach).
First of all, R can only return single objects. It is not possible to return three, separate vectors. However, we can easily return a list of vectors.
df <- data.frame(a=c(1,2,3), b=c(4,5,6), d=c(7,8,9))
# Returns a named list
as.list(df)
#> $a
#> [1] 1 2 3
#>
#> $b
#> [1] 4 5 6
#>
#> $d
#> [1] 7 8 9
I have data table which looks like:
require(data.table)
df <- data.table(Day = seq(as.Date('2014-01-01'), as.Date('2014-12-31'), by = 'days'), Number = 1:365)
I want to subset my data table such that it returns just values of the first 110 rows which are higher than 10. When I use
df2 <- subset(df[1:110,], df$Number[1:110] > 10)
everything works well. However, if I subset using
df2 <- subset(df[1:110,], df[1:110,2] > 10)
R returns the following error:
Error in `[.data.table`(x, r, vars, with = FALSE) :
i is invalid type (matrix). Perhaps in future a 2 column matrix could return a list of elements of DT (in the spirit of A[B] in FAQ 2.14). Please report to data.table issue tracker if you'd like this, or add your comments to FR #657.
Should the way of subsetting not be the same? The problem is that I want to use this subset in an apply command and therefore, the names of the data table change. Hence, I cannot use the column name with the $-operator to refer to the second column and want use the index number but it does not work. I could rename the data table columns or read out the names of the column and use the $-operator but my apply function runs over lots of entries and I want to minimize the workload of the apply function.
So how do I make the subsetting with the index number work and why do I get the mentioned error in the first place? I would like to understand what my mistake is. Thanks!
First let's understand why it doesn't work in your case. When you are doing
df[1:110,2] > 10
# Number
# [1,] FALSE
# [2,] FALSE
# [3,] FALSE
# [4,] FALSE
# [5,] FALSE
# [6,] FALSE
# [7,] FALSE
#....
it returns a 1 column matrix which is used for subsetting.
class(df[1:110,2] > 10)
#[1] "matrix"
which works fine on dataframe
df1 <- data.frame(df)
subset(df1[1:110,], df1[1:110,2] > 10)
# Day Number
#11 2014-01-11 11
#12 2014-01-12 12
#13 2014-01-13 13
#14 2014-01-14 14
#15 2014-01-15 15
#....
but not on data.table. Unfortunately subsetting doesn't work that way in data.table. You could convert it into a vector instead of matrix and then use it for subsetting
subset(df[1:110,], df[1:110][[2]] > 10)
# Day Number
# 1: 2014-01-11 11
# 2: 2014-01-12 12
# 3: 2014-01-13 13
# 4: 2014-01-14 14
# 5: 2014-01-15 15
#...
The difference would be more clear when you see the results of
df[matrix(TRUE), ]
vs
df1[matrix(TRUE), ]
PS - in the first case doing
subset(df[1:110,], Number > 10)
would also have worked.
In R, is there a way to reference a vector from within the vector?
Say I have vectors with long names:
my.vector.with.a.long.name <- 1:10
Rather than this:
my.vector.with.a.long.name[my.vector.with.a.long.name > 5]
Something like this would be nice:
> my.vector.with.a.long.name[~ > 5]
[1] 6 7 8 9 10
Or alternatively indexing by a function would be convenient:
> my.vector.with.a.long.name[is.even]
[1] 2 4 6 8 10
Is there a package that already supports this?
You can use pipes which allow self-referencing with .:
library(pipeR)
my.vector.with.a.long.name %>>% `[`(.>5)
[1] 6 7 8 9 10
my.vector.with.a.long.name %>>% `[`(.%%2==0)
[1] 2 4 6 8 10
The Filter function helps with this
my.vector.with.a.long.name <- 1:10
Filter(function(x) x%%2==0, my.vector.with.a.long.name)
or
is.even <- function(x) x%%2==0
Filter(is.even, my.vector.with.a.long.name)
You can easily create another object with a shorter name:
my.vector.with.a.long.name <- 1:10
mm = my.vector.with.a.long.name
mm
[1] 1 2 3 4 5 6 7 8 9 10
mm[mm<5]
[1] 1 2 3 4
mm[mm>5]
[1] 6 7 8 9 10
Why use other packages and complex code?
So, you're basically asking if you can use something other than the variable's name to refer to it. The short answer is no. That is the whole idea behind variable names. If you want a shorter name, name it something shorter.
The longer answer is it depends. You're really just using logical indexing in its long form. To make it shorter/refer to it more than once without having to type that enormous name, just save it in a vector like so:
gt5 <- my.vector.with.a.long.name > 5
[1] FALSE FALSE FALSE FALSE FALSE TRUE...
my.vector.with.a.long.name[gt5]
[1] 6 7 8 9 10
You can do the same thing with a function as long as it returns the indexes or a logical vector.
The dplyr package allows you to do some cool chaining things, where you use the %.% operator to take the LHS of the operator and input into the first argument of the RHS function call.
It's cool to use in the dplyr package by saying things like:
data %.% group_by(group.var) %.% summarize(Mean=mean(ID))
instead of:
summarize(group_by(data, group.var), Mean=mean(ID)).
suppose I have a numeric vector like:
x <- c(1.0, 2.5, 3.0)
and data.frame:
df<-data.frame(key=c(0.5,1.0,1.5,2.0,2.5,3.0),
value=c(-1.187,0.095,-0.142,-0.818,-0.734,0.511))
df
key value
1 0.5 -1.187
2 1.0 0.095
3 1.5 -0.142
4 2.0 -0.818
5 2.5 -0.734
6 3.0 0.511
I want to extract all the rows in df$key that have the same values equal to x, with result like:
df.x$value
[1] 0.095 -0.734 0.511
Is there an efficient way to do this please? I've tried data.frame, hash package and data.table, all with no success. Thanks for help!
Thanks guys. I actually tried similar thing but got df$key and x reversed. Is it possible to do this with the hash() function (in the 'hash' package)? I see hash can do things like:
h <- hash( keys=letters, values=1:26 )
h$a # 1
h$foo <- "bar"
h[ "foo" ]
h[[ "foo" ]]
z <- letters[3:5]
h[z]
<hash> containing 3 key-value pair(s).
c : 3
d : 4
e : 5
But seems like it doesn't take an array in its key chain, such as:
h[[z]]
Error in h[[z]] : wrong arguments for subsetting an environment
but I need the values only as in a vector rather than a hash. Otherwise, it would be perfect so that we can get rid of data.frame by using some 'real' hash concept.
Try,
df[df$key %in% x,"value"] # resp
df[df$key %in% x,]
Using an OR | condition you may modify it in such a way that your vector may occur in either of your columns. General tip: also have a look at which.
Have you tried testing the valued of df$key that are in x and extracting the value in the value column? I only say this out loud because StackOverflow doesnt like oneline answers:
> x
[1] 1.0 2.5 3.0
> df
key value
1 0.5 -0.7398436
2 1.0 0.6324852
3 1.5 1.8699257
4 2.0 1.0038996
5 2.5 1.2432679
6 3.0 -0.6850663
> df[df$key %in% x,'value']
[1] 0.6324852 1.2432679 -0.6850663
>
BIG WARNING - comparisons with floating point numbers with == can be a bad idea - read R FAQ 7.31 for more info.
I'm having some trouble with an rle command that is designed to find the point at which participants reach 8 contiguous ones in a row.
For example, if:
x <- c(0,1,0,1,1,1,1,1,1,1,1,1)
i want to return a value of 11.
Thanks to DWin to I've been using this piece of code:
which( rle(x2)$values==1 & rle(x2)$lengths >= 8)
sum(rle(x)$lengths[ 1:(min(which(rle(x)$lengths >= 8))-1) ]) + 8
I've been using this code successfully to process my data. However, i noticed that it made a mistake when processing one of my data files.
For example, if
x <- c(1,1,1,1,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,1,1,1,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)
the code returns 19, which is the point at which eight contiguous zeros in a row is reached. i'm not sure what is going wrong or how it fix it.
thanks in advance for your help.
Will
You need to paste the first line of code in its entirety into the second:
sum(rle(x)$lengths[ 1:(min(which( rle(x2)$values==1 & rle(x2)$lengths >= 8))-1) ]) + 8
[1] 39
However, here is another approach, using the function filter. This yields the same result in what I consider to be much more readable code:
which(filter(x2, rep(1/8, 8), sides=1) == 1)[1]
[1] 39
The filter function when used in this way essentially computes a moving average over a block of 8 values in the vector. I then return the position of the first value where the moving average equals 1.
In the basic programming course I teach, I advise students to give proper names to subresults, and to inspect these subresults:
lengthOfrepeatsOfAnything<-rle(x)$lengths
#4 2 5 11 2 2 3 2 17
whichRepeatsAreOfOnes<-rle(x)$values==1
#1 3 5 7 9
repeatsOfOnesLength<-lengthOfrepeatsOfAnything * whichRepeatsAreOfOnes #TRUE = 1, FALSE=0
#4 0 5 0 2 0 3 0 17
whichRepeatOfOneAreLongerThanEight<-which(repeatsOfOnesLength >= 8)
#9
result<-NA
if(length(whichRepeatOfOneAreLongerThanEight)>0){
firstRepeatOfOneAreLongerThanEight<-whichRepeatOfOneAreLongerThanEight[1]
#9
if(firstRepeatOfOneAreLongerThanEight==1){
result<-8
}
else{
repeatsBeforeFirstEightOnes<-1:(firstRepeatOfOneAreLongerThanEight-1)
#1 2 3 4 5 6 7 8
lengthsOfRepeatsBeforeFirstEightOnes<-lengthOfrepeatsOfAnything[repeatsBeforeFirstEightOnes]
#4 2 5 11 2 2 3 2
result<-sum(lengthsOfRepeatsBeforeFirstEightOnes) + 8
}
}
I know it doesn't look as dandy as a oneline solution, but it helps to make things clear and to pick up errors... Besides: what if you look back at this code in 4 months? Which one will be easier to understand again?
My advice would be to break the code up into simpler pieces. As suggested by #Nick, you want to write code which can be easily debugged and modular coding allows you to do that.
# find runs of 0s and 1s
run_01 = rle(x)
# find run of 1's with length >=8
run_1 = with(run_01, which(values == 1 & lengths >=8))
# find starting position of run_1
start_pos = sum(run_01$lengths[1:(run_1 - 1)])
# add 8 to it
end_pos = start_pos + 8