Top 3 of each group by time of day - SQL Lite - sqlite

Question

I'm currently learning how to use SQL Lite, and would like to sort the top 3 most popular pickup locations by hour. I have millions of rows of data with columns of interest being lpep_pickup_datetime (Pickup time) and POLocationID (Pickup location).
I'd like to the top 3 most popular pickup locations by hour.
Here is a sample of the data:
+----------------------+--------------+-----------------+
| lpep_pickup_datetime | PULocationID | passenger_count |
+----------------------+--------------+-----------------+
| 1/1/2017 0:01 | 42 | 1 |
| 1/1/2017 0:03 | 75 | 1 |
| 1/1/2017 0:04 | 82 | 5 |
| 1/1/2017 0:01 | 255 | 1 |
| 1/1/2017 0:00 | 166 | 1 |
| 1/1/2017 0:00 | 179 | 1 |
| 1/1/2017 0:02 | 74 | 1 |
| 1/1/2017 0:15 | 112 | 1 |
| 1/1/2017 0:06 | 36 | 1 |
| 1/1/2017 0:14 | 127 | 5 |
| 1/1/2017 0:01 | 41 | 1 |
| 1/1/2017 0:31 | 97 | 1 |
| 1/1/2017 0:01 | 255 | 5 |
| 1/1/2017 0:00 | 70 | 1 |
| 1/1/2017 0:03 | 255 | 1 |
| 1/1/2017 0:03 | 82 | 1 |
| 1/1/2017 0:00 | 36 | 1 |
| 1/1/2017 0:01 | 7 | 1 |
+----------------------+--------------+-----------------+
Trying this on SQLLiteStudio 3.2.1 - might I just need to use a full MySQL suite in order to be able to use the proper functions?
SELECT
PULocationID, count(PULocationID)
FROM GreenCabs2017
GROUP BY PULocationID
ORDER BY count(PULocationID) DESC
LIMIT 3
The query I've tried only returns top 3 pickup locations across the entire dataset and not by hour of day - how would I be able to group by hour of day? Other solutions on StackExchange reference date_time and date_format functions that won't execute when I try them on SQL Lite - what's a query that would work on SQL Lite?
Ideally would have something like the below:
+-------------+--------------+-----------------+
| Time of Day | PULocationID | PULocationCount |
+-------------+--------------+-----------------+
| 0:00 | 74 | 677 |
| 0:00 | 65 | 333 |
| 0:00 | 55 | 220 |
+-------------+--------------+-----------------+
This would be the output for top 3 pickup locations from midnight to 1:00 AM. This time range would have to apply across all the dates, i.e. 1/1 to 1/31 and not just 1/1 like the sample I provided.
UPDATE:
Changed the format of the timestamps to be YYYY-MM-DD HH:MM:SS format, so I can use the datetime functions now.
Was able to run a query which I think may bring me much closer to what I'm looking for:
SELECT lpep_pickup_datetime, PULocationID, count(PULocationID)
FROM GreenCabs2017
WHERE STRFTIME('%Y', lpep_pickup_datetime) = '2017' AND
STRFTIME('%H', lpep_pickup_datetime) <= '01' AND
STRFTIME('%H', lpep_pickup_datetime) >= '00'
GROUP BY PULocationID
ORDER BY count(PULocationID) DESC
LIMIT 3
That gave an output of
+----------------------+--------------+---------------------+
| lpep_pickup_datetime | PULocationID | count(PULocationID) |
+----------------------+--------------+---------------------+
| 1/31/2017 1:13 | 255 | 7845 |
| 1/31/2017 1:04 | 7 | 4596 |
| 1/31/2017 1:07 | 82 | 3892 |
+----------------------+--------------+---------------------+
But the lpep_pickup_datetime column still indicates that this would be in between 1:00 AM and 2:00 AM and not 12:00 AM and 1:00 AM? Removing the "=" sign in the query results in no results being returned. And I would prefer to not do this for every hour in the day - would there be a way to have an output by hour through one query?

Answer 1

The timestamp string format your data is using, m/d/YYYY H:MM, is not very good. It can't be used by sqlite date and time functions, can't be meaningfully ordered for sorting, and in general is very hard to work with in sqlite. Remember, sqlite does not have dedicated date or time types, just strings or numbers, so the format you're using has to obey the rules of those types. So your first step is to, by whatever means, fix those timestamps. The following assumes you changed them to YYYY-mm-dd HH:MM strings like 2017-01-01 00:01, or another compatible format. It also assumes you're using a fairly recent sqlite release, as it uses window functions which were added in 3.25.
(Edit: You appear to be using NYC taxi data from here, which has timestamps in a good format already, and is suitable for easy importing into sqlite. That makes it trivial to fix.)
Given all that, this query:
WITH ranked AS
(SELECT hour, PULocationID, pickups
, row_number() OVER (PARTITION BY hour ORDER BY pickups DESC) AS rn
FROM (SELECT strftime('%H:00', lpep_pickup_datetime) AS hour
, PULocationID
, count(*) AS pickups
FROM GreenCabs2017
GROUP BY strftime('%H:00', lpep_pickup_datetime), PULocationID))
SELECT * FROM ranked
WHERE rn <= 3
ORDER BY hour, rn
will give, for NYC Green Cab data for January 2017
hour PULocationID pickups rn
---------- ------------ ---------- ----------
00:00 255 4224 1
00:00 7 2518 2
00:00 82 2135 3
01:00 255 3621 1
01:00 7 2078 2
01:00 256 1870 3
02:00 255 3261 1
02:00 256 1798 2
02:00 7 1676 3
03:00 255 2854 1
03:00 256 1589 2
03:00 7 1475 3
and so on.
Basically, it counts the number of times each location appears in each hour, and for each hour, assigns each location a row number based on sorting by that number. Then only the first three rows of each hour are returned in the final outer select. You can also use rank() or dense_rank() instead of row_number(), which will potentially return more than 3 rows per hour in case of ties but also more accurately reflect the most popular locations in those cases.
(This query benefits a lot from having an index on the group by expression:
CREATE INDEX greencabs2017_idx_hour_loc ON GreenCabs2017(strftime('%H:00', lpep_pickup_datetime), PULocationID);
)
Test table created from the sqlite3 shell via:
sqlite> .mode csv
sqlite> .import '|curl -s https://s3.amazonaws.com/nyctlc/trip+data/green_tripdata_2017-01.csv | sed 2d' GreenCabs2017