What is the best way to do range based cross join in Presto? - window-functions

Question

I have a table event_log in Athena which has logs collected from an event processing system. There are various stages in the system and each stage processes these events in sequential order. start_time column indicates the time at which the event entered the system and end_time is the time at which it exited. The system processes millions of events per day. And, we have data for a year in the below table.
event_id
event_type
start_time
end_time
E1
TypeA
T1
T4
E2
TypeB
T2
T6
M1
TypeM
T2
T6
E3
TypeA
T3
T7
E4
TypeB
T4
T7
E5
TypeA
T5
T8
M2
TypeM
T5
T8
E6
TypeB
T6
T9
E7
TypeA
T7
T10
E8
TypeB
T8
T11
M3
TypeM
T8
T11
There are special type of events TypeM (marker events). I have to calculate the processing latency of these special events from these logs. From the table above, this can be achieved by filtering events of that type and compute the latency as end_time - start_time. In addition to that, I want to augment the latency with an additional info - number of events that were actively being processed in various stages of the system when this event was being processed.
-- sample event_log table
CREATE TABLE event_log AS
SELECT * FROM (
VALUES
('E1','TypeA', 1, 4),
('E2','TypeB', 2, 6),
('M1','TypeM', 2, 6),
('E3','TypeA', 3, 7),
('E4','TypeB', 4, 7),
('E5','TypeA', 5, 8),
('M2','TypeM', 5, 8),
('E6','TypeB', 6, 9),
('E7','TypeA', 7, 10),
('E8','TypeB', 8, 11),
('M3','TypeM', 8, 11)
) AS t (event_id, event_type, start_time, end_time)
-- filtered marker table
CREATE TABLE marker_table AS
SELECT * FROM event_log
WHERE event_type = 'TypeM'
-- Join with the filtered marker table on markers start and end time
SELECT mark.*,count(processed_events_in_band.event_id) AS events_processed_count
FROM event_log processed_events_in_band
JOIN marker_table mark
ON processed_events_in_band.end_time between mark.start_time AND mark.end_time
WHERE processed_events_in_band.event_type != 'TypeM'
GROUP BY mark.event_id
Expected result
event_id
event_type
start_time
end_time
events_processed_count
M1
TypeM
T2
T6
2 E1, E2
M2
TypeM
T5
T8
4 E2, E3, E4, E5
M3
TypeM
T8
T11
4 E5, E6, E7, E8
There are partitions on end_time (daily). Have been using them to reduce the data scan. Single day data can be up to 10m. Query should scale to that. Query took around 17 mins for marker table having 18K rows and event logs having 10m rows. There are around 2K parquet files to scan for this 10m rows. Don't think there is S3 read latency causing an issue here.
How do I optimize this query? What is the best way to get this data efficiently?

Answer 1

To improve performance:
be aware that CREATE TABLE will write the output of the query on disk (doc). Consider using a common table expression instead:
with marker_table as (SELECT * FROM event_log
WHERE event_type = 'TypeM')
select ...
try using a condition with an = sign in the join. Presto will do a hash join which is a lot more efficient. In your case I would attempt truncating down start and truncating up end times and write the ON condition using equality of truncated times (up or down)
always place the largest table to the left of the join (doc)
if you want to add the list of events being processed, and not just the count, you can use the array_agg function (doc) combined with array_distinct to produce a list of unique entries and array_join to join it into a string.