So here is what my problem is. I have a really big data.frame woth two columns, first one represents x coordinates (rows) and another one y coordinates (columns), for example:
x y
1 1
2 3
3 1
4 2
3 4
In another frame I have some data (numbers actually):
a b c d
8 7 8 1
1 2 3 4
5 4 7 8
7 8 9 7
1 5 2 3
I would like to add a third column in first data.frame with data from second data.frame based on coordinates from first data.frame. So the result should look like this:
x y z
1 1 8
2 3 3
3 1 5
4 2 8
3 4 8
Since my data.frames are really big the for loops are too slow. I think there is a way to do this with apply loop family, but I can't find how. Thanks in advance (and sorry for ugly message layout, this is my first post here and I don't know how to produce this nice layout with code and proper data.frames like in another questions).
This is a simple indexing question. No need in external packages or *apply loops, just do
df1$z <- df2[as.matrix(df1)]
df1
# x y z
# 1 1 1 8
# 2 2 3 3
# 3 3 1 5
# 4 4 2 8
# 5 3 4 8
A base R solution: (df1 and df2 are coordinates and numbers as data frames):
df1$z <- mapply(function(x,y) df2[x,y], df1$x, df1$y )
It works if the last y in the first data frame is corrected from 5 to 4.
I guess it was a typo since you don't have 5 columns in the second data drame.
Here's how I would do this.
First, use data.table for fast merging; then convert your data frames (I'll call them dt1 with coordinates and vals with values) to data.tables.
dt1<-data.table(dt)
vals<-data.table(vals)
Second, put vals into a new data.table with coordinates:
vals_dt<-data.table(x=rep(1:dim(vals)[1],dim(vals)[2]),
y=rep(1:dim(vals)[2],each=dim(vals)[1]),
z=matrix(vals,ncol=1)[,1],key=c("x","y"))
Now merge:
setkey(dt1,x,y)[vals_dt,z:=z]
You can also try the data.table package and update df1 by reference
library(data.table)
setDT(df1)[, z := df2[cbind(x, y)]][]
# x y z
# 1: 1 1 8
# 2: 2 3 3
# 3: 3 1 5
# 4: 4 2 8
# 5: 3 4 8
Related
I'd like to sum two dataframe with different size in R.
> x = data.frame(a=c(1,2,3),b=c(5,6,7))
> y = data.frame(x=c(1,1,1))
> x
a b
1 1 5
2 2 6
3 3 7
> y
x
1 1
2 1
3 1
The result I want is,
>
a b
1 2 6
2 3 7
3 4 8
How can I do this?
Maybe easiest to convert y to a vector with unlist and then perform the operation. Here, the vector in unlist(y) will be recycled over the columns of the data.frame x.
x + unlist(y)
a b
1 2 6
2 3 7
3 4 8
As a side note, data.frames are a special type of list object and sometimes performing operations on lists can be a bit more involved. On the otherhand, they tend to work fairly well with vectors as long as the dimensions line up (here, as long as the vector has the same length as the number of rows in the data.frame).
We can make the dimensions same and then get the sum
x + rep(y, ncol(x))
# a b
#1 2 6
#2 3 7
#3 4 8
Or another option is sweep
sweep(x, y$x, 1, `+`)
# a b
#1 2 6
#2 3 7
#3 4 8
Lets say I have a data frame with the following structure:
> DF <- data.frame(x=1:5, y=6:10)
> DF
x y
1 1 6
2 2 7
3 3 8
4 4 9
5 5 10
I need to build a new data frame with overlapping observations from the first data frame to be used as an input for building the A matrix for the Rglpk optimization library. I would use n-length observation windows, so that if n=2 the resulting data frame would join rows 1&2, 2&3, 3&4, and so on. The length of the resulting data frame would be
(numberOfObservations-windowSize+1)*windowSize
The result for this example with windowSize=2 would be a structure like
x y
1 1 6
2 2 7
3 2 7
4 3 8
5 3 8
6 4 9
7 4 9
8 5 10
I could do a loop like
DFResult <- NULL
numBlocks <- nrow(DF)-windowSize+1
for (i in 1:numBlocks) {
DFResult <- rbind(DFResult, DF[i:(i+horizon-1), ])
}
But this seems vey inefficient, especially for very large data frames.
I also tried
rollapply(data=DF, width=windowSize, FUN=function(x) x, by.column=FALSE, by=1)
x y
[1,] 1 6
[2,] 2 7
[3,] 2 7
[4,] 3 8
where I was trying to repeat a block of rows without applying any aggregate function. This does not work since I am missing some rows
I am a bit stumped by this and have looked around for similar problems but could not find any. Does anyone have any better ideas?
We could do a vectorized approach
i1 <- seq_len(nrow(DF))
res <- DF[c(rbind(i1[-length(i1)], i1[-1])),]
row.names(res) <- NULL
res
# x y
#1 1 6
#2 2 7
#3 2 7
#4 3 8
#5 3 8
#6 4 9
#7 4 9
#8 5 10
I have a list of dataframes. I need to subset a dataframe of this list according to a criteria in one column of the dataframe.
(all dataframes of the list have the same number and names of columns, and the same number of rows)
For example, I have:
l <- list(data.frame(x=c(2,3,4,5), y = c(4,4,4,4), z=c(2,3,4,5)),
data.frame(x=c(1,4,7,3), y = c(7,7,7,7), z=c(2,5,7,8)),
data.frame(x=c(2,3,1,8), y = c(1,1,1,1), z=c(6,4,1,3)))
names(l) <- c("MH1", "MH2","MH3")
output
$MH1
x y z
1 2 4 2
2 3 4 3
3 4 4 4
4 5 4 5
$MH2
x y z
1 1 7 2
2 4 7 5
3 7 7 7
4 3 7 8
$MH3
x y z
1 2 1 6
2 3 1 4
3 1 1 1
4 8 1 3
So I want to subset the dataframe for which column "y" is the closest to a given number. For example if I say a=3, the chosen dataframe should be "MH1" (where column y=4)
If "l" was a dataframe I will do something like:
closestDF <- subset(l, abs(l$y - a) == min(abs(l$y - a))
How can I do this with the list of dataframes?
Following the answers and comments of #David Arenburg, #akrun and #shadow, here there are three possible solutions to the problem I posted:
Option 1)
library(data.table)
rbindlist(l)[abs(y - a) == min(abs(y - a))]
Option 2) (needs an R version > 3.1.2)
library(dplyr)
bind_rows(l) %>% filter(abs(y-a)==which.min(abs(y-a)))
Option 3) (also works perfectly, but computationally less faster than the first 2 options if used within a big loop or an iterative process)
l[[which.min(sapply(l, function(df) sum(abs(df$y - a))))]]
I am looking for a loop in R which can compare a three column matrix. What I'd like the loop to do is take two matrices/dataframes of the format
A:
X Y Z
3 4 5
3 5 6
4 5 7
and
B:
X Y Z
3 4 5
3 4 4
3 4 7
4 5 7
and loop through each of these returning the row if all of the 3 columns are exact matches.
This would return
X Y Z
3 4 5
4 5 7
Ideally the code could be applied to longer matrices/data frames.
You could try intersect from dplyr if "A", "B" are "data.frames"
library(dplyr)
intersect(A,B)
Or
inner_join(A,B)
An option in base R is merge:
merge(A,B, by = c("X", "Y", "Z"))
# X Y Z
#1 3 4 5
#2 4 5 7
This approach works the same way with matrices and data.frames.
And in fact, you could even leave out the specification of the "by" argument in this case:
merge(A,B)
# X Y Z
#1 3 4 5
#2 4 5 7
If your data are big you can use data.table package:
library(data.table); setDT(A); setDT(B)
setkey(B)[A,nomatch=0]
# X Y Z
#1: 3 4 5
#2: 4 5 7
setkey(A)[B,nomatch=0] returns the same output since it is an intersection.
if i have the following data frame G:
z type x
1 a 4
2 a 5
3 a 6
4 b 1
5 b 0.9
6 c 4
I am trying to get:
z type x y
3 a 6 3
2 a 5 2
1 a 4 1
4 b 1 2
5 b 0.9 1
6 c 4 1
I.e. i want to sort the whole data frame within the levels of factor type based on vector x. Get the length of of each level a = 3 b=2 c=1 and then number in a decreasing fashion in a new vector y.
My starting place is currently with sort()
tapply(y, x, sort)
Would it be best to first try and use sapply to split everything first?
There are many ways to skin this cat. Here is one solution using base R and vectorized code in two steps (without any apply):
Sort the data using order and xtfrm
Use rle and sequence to genereate the sequence.
Replicate your data:
dat <- read.table(text="
z type x
1 a 4
2 a 5
3 a 6
4 b 1
5 b 0.9
6 c 4
", header=TRUE, stringsAsFactors=FALSE)
Two lines of code:
r <- dat[order(dat$type, -xtfrm(dat$x)), ]
r$y <- sequence(rle(r$type)$lengths)
Results in:
r
z type x y
3 3 a 6.0 1
2 2 a 5.0 2
1 1 a 4.0 3
4 4 b 1.0 1
5 5 b 0.9 2
6 6 c 4.0 1
The call to order is slightly complicated. Since you are sorting one column in ascending order and a second in descending order, use the helper function xtfrm. See ?xtfrm for details, but it is also described in ?order.
I like Andrie's better:
dat <- read.table(text="z type x
1 a 4
2 a 5
3 a 6
4 b 1
5 b 0.9
6 c 4", header=T)
Three lines of code:
dat <- dat[order(dat$type), ]
x <- by(dat, dat$type, nrow)
dat$y <- unlist(sapply(x, function(z) z:1))
I Edited my response to adapt for the comments Andrie mentioned. This works but if you went this route instead of Andrie's you're crazy.