I create an sqlite3 database (using SQLite Expert Professional) with 1 table and more than 500,000 records;
If I command a simple query like:
select * from tableOne where entry like 'book one'
if it's my first command to be executed after connecting to database, it takes a considerably long time to be executed and retrieve the result(~15seconds) but just after first command, everything comes back to normal and now every command executes with a very acceptable speed;
even if I close my application(I use pure LUA with sqlite modules)(and within it's logic, reasonably close all connections) as long as Windows(8 x64) is running an not restarted, every command even the first one executes very well but after restarting windows, again, like always first command is slow to be executed;
what is the reason?
how can I prevent this?
Most likely after the first time you run this, you've loaded cache up with all your data, so subsequent queries are fast. Do you have an index on entry? An index will allow efficient querying using entry as a filter. You may want to create one:
CREATE INDEX i_tableone_entry ON tableOne( entry );
Related
I'm running an INSERT query into a Distributed table of ReplicatedMergeTree with 2 nodes (single shard).
After the INSERT, I want to check the number of INSERTED records, so I run a COUNT query on the Distributed table.
At first, the COUNT returns 0. After several seconds (it can take more than a minute) the count returns the correct number.
I've checked using SHOW PROCESSLIST that the INSERT query has finished running.
Is there a way to verify that everything is in order before executing the COUNT?
It seems you may need to use the FINAL keyword. It is mentioned that one should try to avoid it, so you might be better off checking the table design and storage engine, but it could be good interim solution.
https://clickhouse.com/docs/en/sql-reference/statements/select/from/
I have multiple txt files, each with 1 million records (Say 10 files) and these files are saved in LIST_OF_FILES.txt.
I have created a sequence and parallel job to extract data from those files to load into the tables (db2).
Just imagine, I am done with first 2 files. While loading the 3rd file (consider 10000 records are loaded into table so far), the parallel job got aborted due some environmental issue.
Now I want to load records from 10001, where the job got aborted.
JOB DESIGN
Execute command activity_1: wc -l LIST_OF_FILES.txt.
Starting loop: Start:1 , Step: 1 , To: output of Execute command activity_1.
Execute command activity_2: head -output_loop_counter LIST_OF_FILES.txt | tail -1.
parallel job: extract job to load records from file to table.
Execute command activity_3: Moving the extracted file to another folder.
End loop: above steps will continue till last file.
This is not an out-of-the-box capability. You need a job design that keeps track of the number of records processed, so that you can restart from a known good point. Keep in mind, too, that any pending transactions are rolled back if a job aborts - your design probably needs to check how many rows were actually loaded into the target table.
I would maintain the above sequence design and alter the extract job to perform a lookup against the target table on your UC/Primary key for the table, assuming that you have one.
Set failed lookup to reject, and connect your load stage to the reject link. You can dump the valid lookups into a copy stage to dead-end them and get rid of them. This slightly mimics the change capture stage, but with no sorting requirement and no warnings on values.
I'm running a query to create a table using the spatialite gui on my Windows 7 machine. It has been going for days and I would like to cancel it and try something different. Is there a way for me to view the results of the query so far? The .sqlite file has trippled in size and I'm curious about what is happening.
In SQLite, transactions are atomic and isolated. (And if you do not use explicit transactions, every command gets an automatic transaction.)
So there is no easy way to see partial results; the database goes to great efforts to ensure that the transaction either succeeds completely, or is rolled back.
If possible, try the command with a smaller data set, or write the query so that only a part of the data is processed.
I have a driver table with a flag that determines whether that record has been processed or not. I have a stored procedure that reads the table, picks a record up using a cursor, does some stuff (inserts into another table) and then updates the flag on the record to say it's been processed. I'd like to be able to execute the SP multiple times to increase processing.
Obvious answer seemed to be to use 'for update skip locked' in the select for the cursor but it seems this means I cannot commit within the loop (to update the processed flag and commit my inserts) without getting the fetch out of sequence error.
Googling tells me Oracle's AQ is the answer but for the time being this option is not available to me.
Other suggestions? This must be a pretty common request but I've been unable to find anything that useful.
TIA!
A
When I open up TOAD and do a select * from table,the results (first 500 rows) come back almost instantly. But the explain plan shows full table scan and the table is very huge.
How come the results are so quick?
In general, Oracle does not need to materialize the entire result set before it starts returning the data (there are, of course, cases where Oracle has to materialize the result set in order to sort it before it can start returning data). Assuming that your query doesn't require the entire result set to be materialized, Oracle will start returning the data to the client process whether that client process is TOAD or SQL*Plus or a JDBC application you wrote. When the client requests more data, Oracle will continue executing the query and return the next page of results. This allows TOAD to return the first 500 rows relatively quickly even if it would ultimately take many hours for Oracle to execute the entire query and to return the last row to the client.
Toad only returns the first 500 rows for performance, but if you were to run that query through an Oracle interface, JDBC for example, it would return the entire result. My best guess is that the explain plan shows you the results in the case it doesn't get a subset of the records; that's how i use it. I don't have a source for this other than my own experience with it.