My virtual radar program maintains a SQLite3 database. In the database are 2 tables, Aircraft and Flights. The first stores aircraft data and the other flights a particular aircraft made.
I'd like to get a list of aircraft that do not have a flight in the last 180 days.
Reading on the Internet I could create this query:
SELECT Registration,
STRFTIME('%d-%m-%Y',Flights.EndTime) AS 'Last Flight',
Flights.Callsign,
CAST ((JULIANDAY('now') - JULIANDAY(Flights.EndTime)) AS INTEGER) AS
'Days unseen'
FROM Aircraft
INNER JOIN Flights ON (Aircraft.AircraftID=Flights.AircraftID)
WHERE
CAST ((JULIANDAY('now') - JULIANDAY(Flights.EndTime))AS INTEGER) >= 180
GROUP BY Registration
ORDER BY Flights.EndTime
The problem is that the Flights.EndTime is written in ascending order and the query thus looks at the first Flights.EndTime entry instead of the last one.
Is it possible to query the database and retrieve the list of aircraft that were not seen for 180 days or more?
With this query:
SELECT *, ROW_NUMBER() OVER (PARTITION BY AircraftID ORDER BY EndTime DESC) rn
FROM Flights
you can rank all the flights of each aircraft by EndTime descending.
If you join the above query to Aircraft and filter only the last flight (row number = 1), then there is no need for GROUP BY:
SELECT a.Registration,
STRFTIME('%d-%m-%Y', f.EndTime) AS `Last Flight`,
f.Callsign,
CAST(JULIANDAY('now') - JULIANDAY(f.EndTime) AS INTEGER) AS `Days unseen`
FROM Aircraft a
INNER JOIN (
SELECT *, ROW_NUMBER() OVER (PARTITION BY AircraftID ORDER BY EndTime DESC) rn
FROM Flights
) f ON a.AircraftID = f.AircraftID
WHERE f.rn = 1 AND CAST(JULIANDAY('now') - JULIANDAY(f.EndTime) AS INTEGER) >= 180
ORDER BY f.EndTime
I would like to compute a moving average over data in a SQLite table. I found several method in MySQL, but couldn't find an efficient one in SQLite.
In SQL, I think something like this should do it (however, I was not able to try it...) :
SELECT date, value,
avg(value) OVER (ORDER BY date ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING) as MovingAverageWindow7
FROM t ORDER BY date;
However, I see two drawbacks :
This does not seems to work on sqlite
If data are not continuous for few dates on preceding/following rows, it computes a moving average on a window which is wider than what I actually want since it is only based on the number of surrounding rows. Thus, a date condition should be added
Indeed, I would like it to compute the average of 'value' at each date, over +/-3 days (weekly moving average) or +/-15 days (monthly moving average)
Here is an example data set :
CREATE TABLE t ( date DATE, value INTEGER );
INSERT INTO t (date, value) VALUES ('2018-02-01', 8);
INSERT INTO t (date, value) VALUES ('2018-02-02', 2);
INSERT INTO t (date, value) VALUES ('2018-02-05', 5);
INSERT INTO t (date, value) VALUES ('2018-02-06', 4);
INSERT INTO t (date, value) VALUES ('2018-02-07', 1);
INSERT INTO t (date, value) VALUES ('2018-02-10', 6);
INSERT INTO t (date, value) VALUES ('2018-02-11', 0);
INSERT INTO t (date, value) VALUES ('2018-02-12', 2);
INSERT INTO t (date, value) VALUES ('2018-02-13', 1);
INSERT INTO t (date, value) VALUES ('2018-02-14', 3);
INSERT INTO t (date, value) VALUES ('2018-02-15', 11);
INSERT INTO t (date, value) VALUES ('2018-02-18', 4);
INSERT INTO t (date, value) VALUES ('2018-02-20', 1);
INSERT INTO t (date, value) VALUES ('2018-02-21', 5);
INSERT INTO t (date, value) VALUES ('2018-02-28', 10);
INSERT INTO t (date, value) VALUES ('2018-03-02', 6);
INSERT INTO t (date, value) VALUES ('2018-03-03', 7);
INSERT INTO t (date, value) VALUES ('2018-03-04', 3);
INSERT INTO t (date, value) VALUES ('2018-03-08', 5);
INSERT INTO t (date, value) VALUES ('2018-03-09', 6);
INSERT INTO t (date, value) VALUES ('2018-03-15', 1);
INSERT INTO t (date, value) VALUES ('2018-03-16', 3);
INSERT INTO t (date, value) VALUES ('2018-03-25', 5);
INSERT INTO t (date, value) VALUES ('2018-03-31', 1);
Window functions were added in version 3.25.0 (2018-09-15). With the RANGE frame type added in version 3.28.0 (2019-04-16), you can now do:
SELECT date, value,
avg(value) OVER (
ORDER BY CAST (strftime('%s', date) AS INT)
RANGE BETWEEN 3 * 24 * 60 * 60 PRECEDING
AND 3 * 24 * 60 * 60 FOLLOWING
) AS MovingAverageWindow7
FROM t ORDER BY date;
I think I actually found a solution :
SELECT date, value,
(SELECT AVG(value) FROM t t2
WHERE datetime(t1.date, '-3 days') <= datetime(t2.date) AND datetime(t1.date, '+3 days') >= datetime(t2.date)
) AS MAVG
FROM t t1
GROUP BY strftime('%Y-%m-%d', date);
I don't know if it is the most efficient way, but it seems to work
Edit :
Applied to my real database containing 20 000 rows, a weekly moving average over two parameters takes approximately 1 minute to be calculated.
I see two options there :
There is a more efficient way to compute this with SQLite
I compute the moving average in Python after extracting data from SQLite
One approach is to create a intermediate table that maps each date to the groups it belong to.
CREATE TABLE groups (date DATE, daygroup DATE);
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '-1 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '-2 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '-3 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '+1 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '+2 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, strftime('%Y-%m-%d', datetime(date, '+3 days')) AS daygroup
FROM t;
INSERT INTO groups
SELECT date, date AS daygroup FROM t;
You get for example,
SELECT * FROM groups WHERE date = '2018-02-05'
date daygroup
2018-02-05 2018-02-04
2018-02-05 2018-02-03
2018-02-05 2018-02-02
2018-02-05 2018-02-06
2018-02-05 2018-02-07
2018-02-05 2018-02-08
2018-02-05 2018-02-05
indicating that '2018-02-05' belongs to groups '2018-02-02' to '2018-02-08'. If a date belongs to a group, then the value of the data joins the calculation of moving average for the group.
With this, calculating the moving average becomes straightforward:
SELECT
d.date, d.value, c.ma
FROM
t AS d
INNER JOIN
(SELECT
b.daygroup,
avg(a.value) AS ma
FROM
t AS a
INNER JOIN
groups AS b
ON a.date = b.date
GROUP BY b.daygroup) AS c
ON
d.date = c.daygroup
Note that the number of rows of intermediate table is 7 times as large as that of original table, it grows proportionately as taking wider the window. This should be acceptable unless you have much larger table.
I also experimented with 20 000 rows.
The insert query took 1.5s and select query took 0.5s on my laptop.
ADDED, perhaps better.
An alternative that does not require intermediate table.
The query below merges the table with itself, in a way 3 days lag is allowed, then takes average.
SELECT
t1.date, avg(t2.value) AS MVG
FROM
t AS t1
INNER JOIN
t AS t2
ON
datetime(t1.date, '-3 days') <= datetime(t2.date)
AND
datetime(t1.date, '+3 days') >= datetime(t2.date)
GROUP BY
t1.date
;
Here is my data context:
Photos (ID, Title)
Users (ID, FullName)
Ratings (PhotoID, UserID, Value, Date)
Business rules:
users can rate a photo from 1 to 5
a given user can rate a given photo only once
I want to select the top rated photos by day in the last let's say 3 days. So which photo got the best rating today, yesterday and the day before yesterday? I would like to make the number of days variable if it possible. I have to display the last N days only they rated excluding empty days.
I would like to get the photos in a single query/result because I want to bind it to a ListView to display them on a web form.
I've started this way:
DECLARE #days INT = 3
SELECT TOP (#days) ... FROM Ratings
INNER JOIN Photos ON Photos.ID = Ratings.PhotoID
GROUP BY DATEDIFF(day, [Date], CURRENT_TIMESTAMP)
ORDER BY DATEDIFF(day, [Date], CURRENT_TIMESTAMP) DESC
How can I group my groups by PhotoID, order them by SUM(Value) and select the first one from each group? Thank you very much for your help.
SELECT Date, TotalRating, Photos.*
FROM Photos
INNER JOIN
(SELECT ROW_NUMBER() OVER (ORDER BY Date DESC) AS RowNumber,
PhotoID, Date, TotalRating
FROM (SELECT ROW_NUMBER() OVER (PARTITION BY Date, ORDER BY TotalRating DESC) AS inRowNumber,
PhotoID, Date, TotalRating
FROM (SELECT PhotoID, Date, SUM(Value) AS TotalRating
FROM Photos
GROUP BY PhotoID, Date
HAVING SUM(Value) > 0 ) t)
WHERE inRowNumber = 1) t ON Photos.Id = t.PhotoID
WHERE RowNumber <= #days
I have a table with about 200 records, each record has a date, adzone, revenue, clicks, cpm.
What is the proper sqlite statement total count of how many records where grouped in the same statement?
select SUM(revenue), SUM(clicks), adzone
from mytable
where date='12345'
group by adzone
Add a COUNT(1) column:
select SUM(revenue), SUM(clicks), adzone, COUNT(1)
from mytable
where date='12345'
group by adzone
Hi my code doesn't work, Im trying to group my blogentries by year and month here's my sql
SELECT * FROM(
SELECT WP_BlogEntries.BlogEntryID, WP_BlogEntries.AddedDate,
WP_BlogEntries.AddedBy, WP_BlogEntries.BlogID,
WP_BlogEntries.Title, WP_BlogEntries.Description, WP_BlogEntries.Body,
WP_BlogEntries.ReleaseDate, WP_BlogEntries.ExpireDate,
WP_BlogEntries.Approved, WP_BlogEntries.Listed,
WP_BlogEntries.CommentsEnabled, WP_BlogEntries.OnlyForMembers,
WP_BlogEntries.ViewCount, WP_BlogEntries.Votes,
WP_BlogEntries.TotalRating
FROM WP_BlogEntries
WHERE WP_BlogEntries.ReleaseDate < GETDATE()
AND WP_BlogEntries.ExpireDate > GETDATE()
AND Approved = 1
AND Listed = 1
AND WP_BlogEntries.BlogID = #BlogID) MonthEntries
GROUP BY YEAR(ReleaseDate), MONTH(ReleaseDate)
It would be helpful to know the error message.
You can't do a SELECT * FROM if you specify GROUP BY.
The only valid columns are those in the GROUP BY or an aggregate function.
If you group by year and month, then each row will contain one year and month, it is impossible for SQL to know which other columns to display as there could be more than one. (e.g. two blog entries in one month)
Did you mean to ORDER BY instead?
Simething like This :
select convert(varchar(50),YEAR(date)) +'/'+convert (varchar(50), MONTH(date)) ,Name , COUNT( Name) ,[DATE] from table1
group by convert(varchar(50),YEAR(date)) +'/'+convert (varchar(50), MONTH(date)) ,Name,[date]
order by [Date] Desc