Creating summary table of user event data - r

Question

Edit 2: I realized I can use dcast() to do what I want to do. However I do not want to count all of the events in the Event Data, only those that happened before a date specified in another data set. I can't seem to figure out how to use the subset argument in dcast(). So far I've tried:
dcast(dt.events, Email ~ EventType, fun.aggregate = length, subset = as.Date(Date) <=
as.Date(dt.users$CreatedDate[dt.users$Email = dt.events$Email]))
However this doesn't work. I could add the CreatedDate column from the dt.users to the dt.events. And then subset using:
dcast(dt.events, Email ~ EventType, fun.aggregate = length, subset = as.Date(Date) <=
as.Date(CreatedDate)
I was wondering if it was possible to do this without having to add the extra column?
Edit: Just calculated that it'll probably take about 37 hours to complete in the way I'm currently doing it, so if anyone has any tips to make this faster. Please let me know :)
I'm new to R, I've figured out a way to do what I want to do. But it's extremely inefficient, and takes hours to complete.
I have the following:
Event data:
UserID Email EventType Date
User1 User1#*.com Type2 2016-01-02
User1 User1#*.com Type6 2016-01-02
User1 User1#*.com Type1 2016-01-02
User1 User1#*.com Type3 2016-01-02
User2 User2#*.com Type1 2016-01-02
User2 User2#*.com Type1 2016-01-02
User2 User2#*.com Type2 2016-01-02
User3 User3#*.com Type1 2016-01-02
User3 User3#*.com Type3 2016-01-02
User1 User1#*.com Type2 2016-01-04
User1 User1#*.com Type2 2016-01-04
User2 User2#*.com Type5 2016-01-04
User3 User3#*.com Type1 2016-01-04
User3 User3#*.com Type4 2016-01-04
Every time a user does something, an event is recorded with an event type, with a time stamp.
User list from different database:
UserID Email CreatedDate
DxUs1 User1#*.com 2016-01-01
DxUs2 User2#*.com 2016-01-03
DxUs3 User3#*.com 2016-01-03
I want to get the following:
A summarized list which counts the amount of each event type in the Event Data for each user in the User List. However, events should only be counted if the "CreatedDate" in the user list is before or equal to the "Date" in the Event Data.
So for the above data I would eventually want to get:
Email Type1 Type2 Type3 Type4 Type5 Type6
User1#*.com 1 3 1 0 0 1
User2#*.com 0 0 1 0 1 0
User3#*.com 1 0 0 1 0 0
How I've managed to do it so far
I've been able to do this by first creating a 'dt.master' data.table that includes all the columns for all events and the list of Emails. Which looks like this:
Email Type1 Type2 Type3 Type4 Type5 Type6
User1#*.com 0 0 0 0 0 0
User2#*.com 0 0 0 0 0 0
User3#*.com 0 0 0 0 0 0
And then filling out this table using the while loop below:
# The data sets
dt.events # event data
dt.users # user list
dt.master # blank master table
# Loop that fills master table
counter_limit = group_size(dt.master)
index = 1
while (index <= counter_limit) {
# Get events of user at current index
dt.events.temp = filter(dt.events, dt.events$Email %in% dt.users$Email[index],
as.Date(dt.events$Date) <= as.Date(dt.users$CreatedDate[index]))
# Count all the different events
dt.event.counter = as.data.table(t(as.data.table(table(dt.events.temp$EventType))))
# Clean the counter by 1: Rename columns to event names, 2: Remove event names row
names(dt.event.counter) = as.character(unlist(dt.event.counter[1,]))
dt.event.counter = dt.event.counter[-1]
# Fill the current index in on the blank master table
set(dt.master, index, names(dt.event.counter), dt.event.counter)
index = index + 1
}
The Problem
This does work... However I am dealing with 9+ million events, 250k+ users, 150+ Event Types. Therefore the above while loop takes HOURS before it has been processed. I tested it with a small batch of 500 users, which had the following processing time:
user system elapsed
179.33 62.92 242.60
I'm still waiting for the full batch to be processed haha. I've read somewhere that loops should be avoided, as they take a lot of time. However I am completely new to R and programming in general, and I've been learning through trial/error and Googling whatever I've needed. Clearly that leads to some messy code. I was wondering if anyone could point me in the direction of something that might be faster/more efficient?
Thanks!
Edit: Just calculated that it'll probably take about 37 hours to complete in the way I'm currently doing it, so if anyone has any tips to make this faster. Please let me know :)
TL,DR: My event aggregation/summarization code takes several hours to process my data (it's still not done). Is there any faster way to do it?

Answer 1

Assuming your data is already in a data.table, you could use the fun.aggregate parameter in dcast:
dcast(dat, Email ~ EventType, fun.aggregate = length)
gives:
Email Type1 Type2 Type3 Type4 Type5 Type6
1: User1#*.com 1 2 1 0 0 1
2: User2#*.com 4 1 0 0 1 0
3: User3#*.com 0 1 1 1 0 0
In response to the comments & updated question: you can get the desired result by using non-equi joins inside the dcast-function:
dcast(dt.events[dt.users, on = .(Email, Date >= CreatedDate)],
Email ~ EventType, fun.aggregate = length)
which gives:
Email Type1 Type2 Type3 Type4 Type5 Type6
1: User1#*.com 1 2 1 0 0 1
2: User2#*.com 1 0 0 0 1 0
3: User3#*.com 0 1 0 1 0 0

Answer 2

untested
library(dpylr)
library(tidyr)
your.dataset %>%
count(Email, EventType) %>%
spread(EventType, n)