Specific for loop too slow in R - r

Question

I have to use 2 data frames 2 million records and another 2 million records. I used a for loop to obtain the data from one another but it is too slow. I've created an example to demonstrate what I need to do.
ratings = data.frame(id = c(1,2,2,3,3),
rating = c(1,2,3,4,5),
timestamp = c("2006-11-07 15:33:57","2007-04-22 09:09:16","2010-07-16 19:47:45","2010-07-16 19:47:45","2006-10-29 04:49:05"))
stats = data.frame(primeid = c(1,1,1,2),
period = c(1,2,3,4),
user = c(1,1,2,3),
id = c(1,2,3,2),
timestamp = c("2011-07-01 00:00:00","2011-07-01 00:00:00","2011-07-01 00:00:00","2011-07-01 00:00:00"))
ratings$timestamp = strptime(ratings$timestamp, "%Y-%m-%d %H:%M:%S")
stats$timestamp = strptime(stats$timestamp, "%Y-%m-%d %H:%M:%S")
for (i in(1:nrow(stats)))
{
cat("Processing ",i," ...\r\n")
temp = ratings[ratings$id == stats$id[i],]
stats$idrating[i] = max(temp$rating[temp$timestamp < stats$timestamp[i]])
}
Can someone provide me with an alternative for this? I know apply may work but I have no idea how to translate the for function.
UPDATE: Thank you for the help. I am providing more information.
The table stats has unique combinations of primeid,period,user,id.
The table ratings has multiple id records with different ratings and timestamps.
What I want to do is the following. For each id found in stats, to find all the records in the ratings table (id column) and then get the max rating according to a specific timestamp obtained also from stats.

Answer 1

I love plyr, and most of the tools created by Hadley Wickham, but I find that it can be painfully slow, especially if I'm trying to split on an ID field. When this happens, I turn to sqldf. I usually get a speed up of 20x.
First I need to use lubridate because sqldf chokes on POSIXlt types:
library(lubridate)
ratings$timestamp = ymd_hms(ratings$timestamp)
stats$timestamp = ymd_hms(stats$timestamp)
Merge the dataframes, as Vincent did, and remove those violating the date constraint:
tmp <- merge(stats, ratings, by="id")
tmp <- subset(tmp, timestamp.y < timestamp.x )
Lastly, grab the max rating for each ID:
library(sqldf)
sqldf("SELECT *, MAX(rating) AS rating FROM tmp GROUP BY id")

Answer 2

Depending on the ratio of ids to data points this may work better:
r = split(ratings, ratings$id)
stats$idrating = sapply(seq.int(nrow(stats)), function(i) {
rd = r[[stats$id[i]]]
if (length(rd))
max(rd$rating[rd$timestamp < stats$timestamp[i]])
else NA
})
If your IDs are not contiguous integers (you can check that with all(names(r) == seq_along(r))) you'll have to add as.character() when referencing r[[ or use match once to create the mapping and it will cost you some speed.
Obviously, you can do the same without the split, but that's typically slower yet will use less memory:
stats$idrating = sapply(seq.int(nrow(stats)), function(i) {
rd = ratings[ratings$id == stats$id[i],]
if (nrow(rd))
max(rd$rating[rd$timestamp < stats$timestamp[i]])
else NA
})
You can also drop the if if you know there will be no mismatches.

Answer 3

I voted the answer provided although I used another approach to get to the same result
In the merge dataset I first removed dates that were older than the conditioned date and then run this:
aggregate (rating ~ id+primeid+period+user, data=new_stats, FUN = max)

Answer 4

From a data structure perspective it seems that you want to merge two tables and then perform a split-group-apply method.
Instead of for looping to check what row belongs to what row you can simply merge the two tables (much like a JOIN statement in SQL) and then perform an 'aaply' type of method. I recommend you download the 'plyr' library.
new_stats = merge(stats, ratings, by='id')
library(plyr)
ddply(new_stats,
c('primeid', 'period', 'user'),
function(new_stats)
c( max(new_stats[as.Date(new_stats$timestamp.x) > as.Date(new_stats$timestamp.y)]$rating )))
If the use of plyr confuses you, please visit this tutorial: http://www.creatapreneur.com/2013/01/split-group-apply/.