Given data like this
C1<-c(3,-999.000,4,4,5)
C2<-c(3,7,3,4,5)
C3<-c(5,4,3,6,-999.000)
DF<-data.frame(ID=c("A","B","C","D","E"),C1=C1,C2=C2,C3=C3)
How do I go about removing the -999.000 data in all of the columns
I know this works per column
DF2<-DF[!(DF$C1==-999.000 | DF$C2==-999.000 | DF$C3==-999.000),]
But I'd like to avoid referencing each column. I am thinking there is an easy way to reference all of the columns in a particular data frame aka:
DF3<-DF[!(DF[,]==-999.000),]
or
DF3<-DF[!(DF[,(2:4)]==-999.000),]
but obviously these do not work
And out of curiosity, bonus points if you can me why I need that last comma before the ending square bracket as in:
==-999.000),]
The following may work
DF[!apply(DF==-999,1,sum),]
or if you can have multiple -999 on a row
DF[!(apply(DF==-999,1,sum)>0),]
or
DF[!apply(DF==-999,1,any),]
Based on your code, I'll assume that you want to remove all rows that contain -999.
DF2 <- DF[rowSums(DF == -999) == 0, ]
As for your bonus question: A data frame is a list of vectors, all of which have the same length. If we think of the vectors as columns, then a data frame can be thought of as a matrix where the columns might have different types (numeric, character, etc). R allows you to refer to elements of a data frame much the same way you refer to elements of a matrix; by using row and column indices. So DF[i, j] refers to the ith element in the jth vector of DF, which you can think of as the ith row and jth column. So if you want to retain only some of the rows of the data frame and all columns, you can use a matrix-like notation: DF[row.indices, ].
To address your "bonus" question, if we go to the documentation for ?Extract.data.frame we will find:
Data frames can be indexed in several modes. When [ and [[ are used
with a single index (x[i] or x[[i]]), they index the data frame as if
it were a list. In this usage a drop argument is ignored, with a
warning.
and also:
When [ and [[ are used with two indices (x[i, j] and x[[i, j]]) they
act like indexing a matrix: [[ can only be used to select one element.
Note that for each selected column, xj say, typically (if it is not
matrix-like), the resulting column will be xj[i], and hence rely on
the corresponding [ method, see the examples section.
So you need the comma to ensure that R knows you are referring to a row, not a column.
I don't understand if your target is to remove all the rows that contain at least one NA, if this is what you are looking for, then this could be a possible answer:
DF[DF==-999] <- NA
na.omit(DF)
ID C1 C2 C3
1 A 3 3 5
3 C 4 3 3
4 D 4 4 6
Related
I have a code I'm working with which has the following line,
data2 <- apply(data1[,-c(1:(index-1))],2,log)
I understand that this creates a new data frame, from the data1, taking column-wise values log-transformed and some columns are eliminated, but I don't understand how the columns are removed. what does 1:(index-1) do exactly?
The ":" operator creates an integer sequence. Because (1:(index-1) ) is numeric and being used in the second position for the extraction operator"[" applied to a dataframe, it is is referring to column numbers. The person writing the code didn't need the c-function. It could have been more economically written:
data1[,-(1:(index-1))]
# but the outer "("...")"'s are needed so it starts at 1 rather than -1
So it removes the first index-1 columns from the object passed to apply. (As MrFlick points out, index must have been defined before this gets passed to R. There's not default value or interpretation for index in R.
Suppose the index is 5, then index -1 returns 4 so the sequence will be from 1 to 4 i.e. and then we use - implies loop over the columns other than the first 4 columns as MARGIN = 2
How do we identify all those row entries in a particular column that contain a specific set of keywords?
For example, I have the following dataframe:
test <- data.frame(nom = 1:5, name = c("ser bla", "onlybla", "inspectiongfa serdafds", "inspection", "serbla blainspection"))
My keywords of interest are "ser" & "inspection"
What I'm looking for is to enlist all the values of the second column (i.e. name) in which both the keywords are present together.
So basically, my output should enlist the name values of rows 3 and 4 viz. "inspectiongfa serdafds" & "serbla blainspection"
What I have tried is the following:
I first generate a truth table to enlist the presence of each of the keywords for each row in the dataframe as follows:
as.data.frame(sapply(c("ser", "inspection"), grepl, test$name))
Once I get this, all I have to do is identify all those row entries where the values are a pair of TRUE TRUE. Hence, they'll correspond to the cases where the keywords of interest are present. Here it's the same rows 3 & 4.
But, I'm not able to figure out how to identify such row entries with the TRUE TRUE pair and whether this whole process is a bit of an overkill and it can be done in a much efficient manner.
Any help would be appreciated. Thanks!
You're almost there :)
Here's a solution extending what you have done:
# store your logic test outcomes
conditions_df <- as.data.frame(sapply(c("ser", "inspection"), grepl, test$name))
# False=0 & True=1. Can use rowSums to get the total and find ones that =2 ie True+True
# which gives you the indices of the TRUE outcomes ie the rows we need to filter test
locate_rows <- which(rowSums(conditions_df) == 2)
test$name[locate_rows]
[1] "inspectiongfa serdafds"
[2] "serbla blainspection"
Im trying to make a large blank data.table with a header row in order to add values in specific places once it is set up. I have been able to duplicate the first row and then clear every other row or every row, but what I'd like to do is clear every row after the header row. Some columns are numeric input and some are character input.
[input3]:
headers: header1 header2 header3..... header 60+
Values: NA NA NA ... NA
Duplicate row:
input3 <- input2[rep(1:nrow(input2), each = 2), ]
Clear every row:
input3[1:nrow(input3) %% 1 == 0, ] <- NA
But if I try to rewrite that as duplicating blank rows starting at row 2 (to preserve the header) I get this error:
input3[2:nrow(input3) %% 1 == 0, ] <- NA
"Error in [.data.table(x, i, which = TRUE) : i evaluates to a logical vector length 9 but there are 10 rows. Recycling of logical i is no longer allowed as it hides more bugs than is worth the rare convenience. Explicitly use rep(...,length=.N) if you really need to recycle."
I need to be able to dynamically add rows while keeping the header as this is going to be a gigantic table I will export to another program.
Edit: this is different from this link in that I'm adding additional rows not specified originally in the data. Not just wiping rows.
Instead use
input3[c(FALSE,2:nrow(input3) %% 1 == 0,] <- NA
By using 2:nrow, you were explicitly giving a shortened vector. When that thing is a logical vector, it must be length 1 or the same as the number of rows. Period.
Though this has its problems and I discourage its use, perhaps you were expecting it to behave like this:
input3[which(2:nrow(input3) %% 1 == 0),] <- NA
The "good" of this is that the which(...) returns a vector of integer, so it does not need to be the same length as the number of rows in the frame/table.
From ?Extract (which includes [ and friends):
For '['-indexing only: 'i', 'j', '...' can be logical
vectors, indicating elements/slices to select. Such vectors
are recycled if necessary to match the corresponding extent.
'i', 'j', '...' can also be negative integers, indicating
elements/slices to leave out of the selection.
"Recycling" is why length 1 works: its logical value is used for all rows. If you use length 2 and there are an even number of rows (e.g., mtcars[c(T,F),]), then it will give every-other-row. On a similar vein, if you assume recycling and there are not an even multiple of rows (e.g., mtcars[c(T,F,F),]), then your assumptions start becoming less clear.
Add to that the behavior of data.table where it does not enforcing of this. Recycling can get you in trouble, so data.table doesn't encourage it.
library(data.table)
mt <- as.data.table(mtcars)
mt[c(T,F),] <- NA
# Error in `[.data.table`(x, i, which = TRUE) :
# i evaluates to a logical vector length 2 but there are 32 rows. Recycling of logical i is no longer allowed as it hides more bugs than is worth the rare convenience. Explicitly use rep(...,length=.N) if you really need to recycle.
mt[c(1,3),] <- NA
I have a dataframe ma
it has a factor called type
type is comprised of the following factors: I210, I210plus, I210plusc, KV2c, KV2cplus
I'd like to put some of these factors in a vector, say, selected_types
so, selected_types<-c("I210plusc","KV2c")
then, have this command subset the dataframe ma
ma1<-subset(ma, type==selected_types)
such that ma1 would be a subset of ma consisting of only the observations that had
type I210plusc and KV2c
however, when I do this, the number of observations in the resulting dataframe ma1 is less than the sum of the occurrences of the two types in selected_types from the original ma
Any ideas on what I'm doing incorrectly?
Thank you
I originally had this in a comment, but it's a bit lengthy, plus I wanted to add to it. Here some details on what's happening:
what you're doing with == is recycling your two length vector, so that every even row is compared to "KV2c", and every odd one to "I210plusc", so your final result will be the data frame of odd rows that are "KV2c" and even rows that are "I210plusc".
An alternate solution that might make the issue clear is as follows:
subset(ma, type == selected_types[[1]] | type == selected_types[[2]])
Or, more gracefully:
subset(ma, type %in% selected_types)
The %in% operator returns a logical vector of same length as type with TRUE for every position in type that "is in" selected_types (hence the name of the operator).
I'm struggling to remove a row in a matrix, where this matrix's name is "unknown". What I mean by "unknown" is that there are several matrices, and the last 3 characters of each matrix's name is different.
An example would make this a lot clearer I think.
Say I have 3 matrices, Trades_ABC, Trades_DEF, Trades_HIJ. Each of these matrices has x rows and 5 columns.
I currently have the following code:
for (k in 1:3)
assign(get(paste0("Trades_",sellLeg))[1,1],y)
next k
Where "sellLeg" is one of "ABC","DEF","HIJ"
In this code I am trying to change the value of the first element in each of the three matrices to some number, represented by "1", as an example. In reality, I'm not so much looking to CHANGE a value as I am looking to REMOVE a row, but my main problem is that I don't know how to assign a value to a matrix with an "unknown" name (once I can do this I should be able to remove a row)
Many thanks!