I have a table with about 1000 rows.
I want to query for the average value on column "Things" from row 0 to row 79.
I do the following:
SELECT AVG("Things") FROM "MyTable" LIMIT 80 OFFSET 0
At first it seemed like it worked, I get a number back.
then I tried to run this query just to test things:
SELECT AVG("Things") FROM "MyTable" LIMIT 80 OFFSET 10
and I get nothing back. Whatever number I put in the OFFSET I get nothing back unless I use zero (... OFFSET 0)
I am not sure why that is. Any idea?
The OFFSET/LIMIT clauses applay to the result rows of a query. With "LIMIT 80", you are saying that you want no more than 80 rows back.
AVG() returns only one row, so the LIMIT clause has no effect whatsoever.
To filter rows before AVG() is applied, you need to use WHERE, or a subquery.
Please note that there is no guaranteed order of rows unless you're using ORDER BY, so you always should use ORDER BY when using LIMIT:
SELECT AVG(Things)
FROM (SELECT Things
FROM MyTable
ORDER BY ...
LIMIT 80 OFFSET ...);
I verified the problem in sqlite 3.8.5 on my Mac. I don't understand why it doesn't work, but I'm not exactly a SQL expert.
As a workaround, you can do something like:
select avg("Things") from MyTable where (rowid>=10) and (rowid<=90);
This isn't exactly the same thing if you've deleted rows, though.
Here's a better way:
select avg("Things") from (select Things from MyTable limit 80 offset 10);
Related
I have a database with an PRIMARY KEY INTEGER AUTOINCREMENT column named id and a condition flag column, call it condition which is an INTEGER.
I would like to be able to SELECT a given number of rows N where conditon=1. That is easy enough to query (for example if N=10):
SELECT data FROM table_name WHERE condition=1 LIMIT 10;
However I would like to be guaranteed that the rows I receive are also those rows with the smallest values of id from the full set of rows where condition=1. For example if rows with id between 1 and 20 have condition=1 I would like my query to be guaranteed to return rows with id=1 - 10.
My understanding is that ORDER BY is completed after the query so I don't think including ORDER BY id would make this a guarantee. Is there a way to guarantee this?
Well you are wrong:
SELECT data FROM table_name WHERE condition=1 ORDER BY id LIMIT 10;
is what you need.
It will sort the rows you need and then the limit is applied.
From http://www.sqlitetutorial.net/sqlite-limit/
SQLite LIMIT and ORDER BY clause
We typically use the LIMIT clause with ORDER BY clause, because we are
interested in getting the numberof rows in a specified order, not in
unspecified order.
The ORDER BY clause appears before the LIMIT clause in the SELECT
statement.
SQLite sorts the result set before getting the number of
rows specified in the LIMIT clause.
First I'd like to apologize if the topic seems vague; I always have a hard time framing them succinctly. That done, I'll get into it.
Suppose I have a database table that looks like the following:
CREATE TABLE The_table(
item_id INTEGER PRIMARY KEY ASC AUTOINCREMENT,
item TEXT);
Now, I have a pretty basic query that will get items from said table and order them:
SELECT *
FROM The_table
ORDER BY x;
where x could be either item_id or item. I can guarantee that both fields are order-able. My question is this:
Is there a way to modify the query I gave to get a range of the ordered elements: say from 20th element in the table to the 40th element in the table (after the table has been ordered) or something similar.
Any help would be appreciated.
Thanks,
Yes - it's called "between"
SELECT *
FROM The_Table
WHERE item_id BETWEEN 20 AND 40
This does exactly what it says - it looks for a value between the two numbers supplied. Very useful for finding ranges; works in reverse too (i.e. NOT BETWEEN). For more see here.
If you want a specific row or group of rows (as your updated question suggests) after sorting you can use the LIMIT clause to select a range of entries
SELECT *
FROM The_Table
LIMIT 20, 20
Using LIMIT this way the first number is the starting point in the table and the second number is how many records to return from that point. This statement will return 20 rows starting at row 20 whatever that value is.
I need to show the SQLite results in a list view. Of course, I need to page the results.
The first option is to use the LIMIT clause. For example:
SELECT * FROM Table LIMIT 100, 5000
It returns records 5001 to 5100. The problem is that internally SQLite "reads" the first 5000 records and it is not too efficient.
What is the best approach for paging when there are a lot of records?
Please note that you always have to use an ORDER BY clause; otherwise, the order is arbitrary.
To do efficient paging, save the first/last displayed values of the ordered field(s), and continue just after them when displaying the next page:
SELECT *
FROM MyTable
WHERE SomeColumn > LastValue
ORDER BY SomeColumn
LIMIT 100;
(This is explained with more detail on the SQLite wiki.)
When you have multiple sort columns (and SQLite 3.15 or later), you can use a row value comparison for this:
SELECT *
FROM MyTable
WHERE (SomeColumn, OtherColumn) > (LastSome, LastOther)
ORDER BY SomeColumn, OtherColumn
LIMIT 100;
Following is the query that I use for getting a fixed number of records from a database with millions of records:-
select * from myTable LIMIT 100 OFFSET 0
What I observed is, if the offset is very high like say 90000, then it takes more time for the query to execute. Following is the time difference between 2 queries with different offsets:
select * from myTable LIMIT 100 OFFSET 0 //Execution Time is less than 1sec
select * from myTable LIMIT 100 OFFSET 95000 //Execution Time is almost 15secs
Can anyone suggest me how to optimize this query? I mean, the Query Execution Time should be same and fast for any number of records I wish to retrieve from any OFFSET.
Newly Added:-
The actual scenario is that I have got a database having > than 1 million records. But since it's an embedded device, I just can't do "select * from myTable" and then fetch all the records from the query. My device crashes. Instead what I do is I keep fetching records batch by batch (batch size = 100 or 1000 records) as per the query mentioned above. But as i mentioned, it becomes slow as the offset increases. So, my ultimate aim is that I want to read all the records from the database. But since I can't fetch all the records in a single execution, I need some other efficient way to achieve this.
As JvdBerg said, indexes are not used in LIMIT/OFFSET.
Simply adding 'ORDER BY indexed_field' will not help too.
To speed up pagination you should avoid LIMIT/OFFSET and use WHERE clause instead. For example, if your primary key field is named 'id' and has no gaps, than your code above can be rewritten like this:
SELECT * FROM myTable WHERE id>=0 AND id<100 //very fast!
SELECT * FROM myTable WHERE id>=95000 AND id<95100 //as fast as previous line!
By doing a query with a offset of 95000, all previous 95000 records are processed. You should make some index on the table, and use that for selecting records.
As #user318750 said, if you know you have a contiguous index, you can simply use
select * from Table where index >= %start and index < %(start+size)
However, those cases are rare. If you don't want to rely on that assumption, use a sub-query, for example using rowid, which is always indexed,
select * from Table where rowid in (
select rowid from Table limit %size offset %start)
This speeds things up especially if you have "fat" rows (e.g. that contain blobs).
If maintaining the record order is important (it usually isn't), you need to order the indices first:
select * from Table where rowid in (
select rowid from Table order by rowid limit %size offset %start)
select * from data where rowid = (select rowid from data limit 1 offset 999999);
With SQLite, you don't need to get all rows returned at once in a big fat array, you can get called back for every row. This way, you can process the results as they come in, which should address both your crashing and performance issues.
I guess you're not using C as you would already be using a callback, but this technique should be available in any other language.
Javascript example (from : https://www.npmjs.com/package/sqlite3 )
db.each("SELECT rowid AS id, info FROM lorem", function(err, row) {
console.log(row.id + ": " + row.info);
});
I want to get a number of rows in my table using max(id). When it returns NULL - if there are no rows in the table - I want to return 0. And when there are rows I want to return max(id) + 1.
My rows are being numbered from 0 and autoincreased.
Here is my statement:
SELECT CASE WHEN MAX(id) != NULL THEN (MAX(id) + 1) ELSE 0 END FROM words
But it is always returning me 0. What have I done wrong?
You can query the actual number of rows withSELECT Count(*) FROM tblName
see https://www.w3schools.com/sql/sql_count_avg_sum.asp
If you want to use the MAX(id) instead of the count, after reading the comments from Pax then the following SQL will give you what you want
SELECT COALESCE(MAX(id)+1, 0) FROM words
In SQL, NULL = NULL is false, you usually have to use IS NULL:
SELECT CASE WHEN MAX(id) IS NULL THEN 0 ELSE (MAX(id) + 1) END FROM words
But, if you want the number of rows, you should just use count(id) since your solution will give 10 if your rows are (0,1,3,5,9) where it should give 5.
If you can guarantee you will always ids from 0 to N, max(id)+1 may be faster depending on the index implementation (it may be faster to traverse the right side of a balanced tree rather than traversing the whole tree, counting.
But that's very implementation-specific and I would advise against relying on it, not least because it locks your performance to a specific DBMS.
Not sure if I understand your question, but max(id) won't give you the number of lines at all. For example if you have only one line with id = 13 (let's say you deleted the previous lines), you'll have max(id) = 13 but the number of rows is 1. The correct (and fastest) solution is to use count(). BTW if you wonder why there's a star, it's because you can count lines based on a criteria.
I got same problem if i understand your question correctly, I want to know the last inserted id after every insert performance in SQLite operation. i tried the following statement:
select * from table_name order by id desc limit 1
The id is the first column and primary key of the table_name, the mentioned statement show me the record with the largest id.
But the premise is u never deleted any row so the numbers of id equal to the numbers of rows.
Extension of VolkerK's answer, to make code a little more readable, you can use AS to reference the count, example below:
SELECT COUNT(*) AS c from profile
This makes for much easier reading in some frameworks, for example, i'm using Exponent's (React Native) Sqlite integration, and without the AS statement, the code is pretty ugly.