I read that wrapping a lot of SELECT into BEGIN TRANSACTION/COMMIT was an interesting optimization.
But are these commands really necessary if I use "PRAGMA journal_mode = OFF" before? (Which, if I remember, disables the log and obviously the transaction system too.)
Note that I don't agree with BigMacAttack.
For SQLITE, wrapping SELECTs in a Transaction does do something:
It reduces the number of SHARED locks that are obtained and then dropped.
Reference:
http://www.mail-archive.com/sqlite-users%40sqlite.org/msg79839.html
So I think the transaction would also be beneficial even if you had journal_mode turned off, because there is still the locking overhead to consider.
Maybe read_uncommitted would be something you could consider - I would guess that it would disable the SHARED locking.
"Use transactions – even if you’re just reading the data. This may yield a few milliseconds."
I'm not sure where the Katashrophos.net blog is getting this information, but wrapping SELECT statements in transactions does nothing. Transactions are always and only used when making changes to the database, and transactions cannot be disabled. They are a requirement. What many don't understand is that unless you manually BEGIN and COMMIT a transaction, each statement will be automatically put in their own unique transaction. Be sure to read the de facto SO question on improving sqlite performance. What the author of the blog might have been trying to say, is that if you plan to do an INSERT, then a SELECT, then another INSERT, then it would increase performance to manually wrap these statements in a single transaction. Otherwise sqlite will automatically put the two insert statements in separate unique transactions.
According to the "SQL as Understood by SQLite" documentation concerning transactions:
"No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect."
Lastly, disabling journaling via PRAGMA journal_mode = OFF does not disable transactions, only logging. But disabling the log is a good way to increase performance as well. Normally after each transaction, sqlite will document the transaction in the journal. When it doesn't have to do this, you get a performance boost.
UPDATE:
So it has been brought to my attention by "elegant dice" that the SQLite documentation statement I quote above is misleading. SELECT statements do in fact use the transaction system. This is used to acquire and release a SHARED lock on the database. As a result, it is indeed more efficient to wrap multiple SELECT statements in a single transaction. By doing so, the lock is only acquired and released once, rather than for each individual SELECT statement. This ends up being slightly more efficient while also assuring that all SELECT statements will access the same version of the database in case something has been added/deleted by some other program.
Related
There are some groups of queries that include creating tables, fields, etc. How to implement a mechanism by which a group of requests should be executed all, or if somewhere the error was canceled all? That is, the principle of transactionality with ALTER TABLE queries for example (which are committed implicitly)
You are talking about Commit and Rollback.
However if you are mixing normal SQL with DML (CREATE/ALTER/DROP etc) the DML commands have an implied COMMIT as part of there execution, you cannot avoid it. So this will likely cause you problems if mixed with your normal insert/update/delete type queries
Transactional DML is not possible until MySQL 8.0. In that version, there is a "Data Dictionary" that is stored in InnoDB tables. (I don't want to think about the bootstrap required!) The DD allows, at least in theory, the ROLLBACK of DDL actions such as ALTER and DROP TABLE. Study the 8.0 docs to see if it does enough for your needs.
In the past (pre-8.0), Many DML operations were at least reasonable "crash safe". For example, an ALTER used to copy the table over, then quickly swap in the new table. This provided a reasonably good way to recover from, a crash during an ALTER.
There have been a lot of major improvements to ALTER since 5.6. Before then, the only really "instant" alter was adding an option to an ENUM, and that had caveats. There are still things that mandate a complete, time-consuming, rebuild of the table -- such as any change to the PRIMARY KEY.
DML operations should be minimized; they are not designed for frequent use.
A transaction in an Oracle db makes changes in the db, and the changes are committed. Is it possible that other transactions see the performed changes only after several seconds, not immediately?
Background:
We have an application that performs db changes, commits them, and THEN (immediately) it reads the changed data back from the database. However, sometimes it happens that it finds no changes. When the same read is repeated later (by executing the same select manually from SQL Developer), the changed data are returned correctly. The db is standalone, not clustered.
The application does not communicate with the database directly, that would be easy, several layers (including MQ messaging) are involved. We've already eliminated other potential causes of the behaviour (like incorrect parameters, caching etc.). Now I'd like to eliminate an unexpected behaviour of the Oracle db as the cause.
Edit:
At first, I'd like to emphasize that I'm NOT asking whether uncommited changes can be visible to other sessions.
At second, the Oracle COMMIT statement has several modifiers, like WRITE BATCH or NOWAIT. I don't know whether these modifiers can have any influence on the answer to my questions, but we are not using them anyway.
Assuming that your sessions are all using a read committed isolation level, changes would be visible to any query that starts after the data was committed. It was possible in early versions of RAC to have a small delay between when a change was committed on one node and when it was visible on another node but that has been eliminated for a while and you're not using RAC so that's presumably not it.
If your transactions are using the serializable isolation level and the insert happens in a different session than the select, the change would only be visible to other sessions whose transactions began after the change was committed. If sessions A & B both start serializable transactions at time 0, A inserts a row at time 1, and B queries the data at time 2, B would see the state of the data at time 0 and wouldn't see the data that was inserted at time 1 until after it committed its transaction. Note that this would only apply if the two statements are in different sessions-- session A would see the row because it was inserted in A's transaction.
Barring an isolation level issue, I would expect that the SELECT wasn't actually running after the INSERT committed.
due to my impatience i corrupted my database with a replace query now all rows in that column shows 0.
is there any support for Sqlite manager in Firefox using which i will be able to undo to my previous state of DB.
Thanks
There's no such thing as UNDO -- however if you began a transaction prior to starting your query (or queries), you could simple not commit the transaction. Short of that, you need to restore from a backup (and if you don't have a backup, you'll probably think to make one next time?).
I came across the .import command to do this (bulk insert), but is there a query version of this which I can execute using sqlite3_exec().
I would just like to copy a small text file contents into a table.
A query version of this one below,
".import demotab.txt mytable"
Sqlite's performance doesn't benefit from bulk insert. Simply performing the inserts separately (but within a single transaction!) provides very good performance.
You might benefit from increasing sqlite's page cache size; that depends on the number of indexes and/or the order in which the data is inserted. If you don't have any indexes, for a pure insert, the cache size is likely not to matter much.
Be sure to use a prepared query, as opposed to regenerating a query plan in the innermost loop. It's extremely important to wrap the statements in a transaction since this avoids the need for the filesystem to sync the database to disk - afterall, partially a written transaction is atomically aborted anyhow, meaning that all fsync()'s are delayed until the transaction completes.
Finally, indexes will limit your insert performance since their creation is somewhat expensive. If you're really dealing with a lot of data and start off with an empty table, it may be beneficial to add the indexes after the data - though this isn't a huge factor.
Oh, and you might want to get one of those intel X25-E SSD's and ensure you have an AHCI controller ;-).
I'm maintaining an app with sqlite db's with about 500000000 rows (spread over several tables) - much of which was bulk inserted using plain old begin-insert-commit: it works fine.
I have a routine which is updating my business entity. The update involves about 6 different tables. All the commands are being executed within a transaction.
Recently, I needed to add some code into the routine which accesses a lookup table from the database. The lookup code already existed in another business object so I used that business object. For example:
Using tr As DbTransaction = myConnection.BeginTransaction()
ExecuteCommand1(tr)
ExecuteCommand2(tr)
If myLookupTable.GetLookupTable().FindById(id).HasFlagSet Then
ExecuteCommand3(tr)
End If
End Using
However, the lookup table business object hangs/deadlocks. I think this is because it doesn't have a reference to the transaction being used by the original routine.
After doing some research, I attempted to put the lookup table logic in its own transaction, setting the IsolationLevel to ReadUncommitted. This gave me the results I desired. However, after further research, I'm now second-guessing if I've implemented this correctly.
Assuming a reference to the active transaction is unavailable to my lookup table object, is what I've described considered best practice? I feel like I might be missing something.
If you're doing a read in the middle of your transaction then you should do it under the transaction context, not using a different transaction and dirty reads. Luckily there is an easy solution: instead of using the ADO.Net transaction objects use the .Net TransactionScope object. The ADO.Net code is sensible to it and will enlist all your operations in this transaction, including your other business component reads. Just make sure your business object does not open a different connection, this will result in attempting to escalate the existing transaction into a distributed transaction and enlist the new connection into it.
The alternative is to pass down your SqlConnection/SqlTransaction pair on each call, but that propagates horribly ugly everywhere in your code.
If it were me I would rewrite the logic so I do not have to do an uncommitted read.
The golden rule to avoid deadlocks is to always take table locks in the same order in every transaction. So look at the code in the other transactions to see what order they take table locks; then make sure you use the same order in your transaction.
Apparently your look up is attempting to access a row(s) that is exclusively locked by transaction tr. If you use a readuncommitted transaction or alternatively use WITH(NOLOCK) in your lookup query, you will see all uncommitted changes by transactions that might be occurring and effecting your lookup logic. So I am not so sure how desirable this would be.
I think it is best to find a way to ensure that your lookup query participates in the current transaction if you need to do the lookup during that transaction. If all of these operations are to be executed in the same thread, one thing that you can do is to store the transaction object in thread local storage when you create one and have GetLookupTable method check the thread local storage for a transaction object and if there is a transaction set, you can get the connection from that transaction object. Otherwise, you create a new connection. This way your lookup will become part of that transaction and it should run its logic without getting blocked by the current transaction and in turn blocking the current transaction and thus leading to a deadlock.