Reading through the Intershop documentation I have come accross mentions of pesimistic and optimistic locking mechanics (where optimistic is implemented through the OCA attribute in database tables).
I'm wondering how to control which type of locking is used throughout the database and where to read more about these locking types.
As far as I know, there is only optimistic locking on object level and this cant be disabled. Every table in intershop has an OCA column for this reason. You do have other ways of locking however
For example:
ORMObject.tryLock. This lock the actual row in the database by using a query : select for update nowait. I wont recommend using this, you might end up with some rather hard to debug deadlock bugs. See the javadoc for more.
Then there is Locking Framework. It doesn't actually stop any process from updating data, it is a ways for intershop to orchestrate different process by having them lock resources so that they dont get in each others way. However u can still ignore these locks.
I generally try to avoid locking. This KB might be interesting for you especially the part on Transaction Scopes.
Optimistic caching is set through class modifier "oca" when creating the corresponding EDL (Enfinity Definition Language) models.
See Intershop documentation at https://support.intershop.com/kb/index.php/Display/247P28
Related
From sqlite FAQ I've known that:
Multiple processes can have the same database open at the same time.
Multiple processes can be doing a SELECT at the same time. But only
one process can be making changes to the database at any moment in
time, however.
So, as far as I understand I can:
1) Read db from multiple threads (SELECT)
2) Read db from multiple threads (SELECT) and write from single thread (CREATE, INSERT, DELETE)
But, I read about Write-Ahead Logging that provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently.
Finally, I've got completely muddled when I found it, when specified:
Here are other reasons for getting an SQLITE_LOCKED error:
Trying to CREATE or DROP a table or index while a SELECT statement is
still pending.
Trying to write to a table while a SELECT is active on that same table.
Trying to do two SELECT on the same table at the same time in a
multithread application, if sqlite is not set to do so.
fcntl(3,F_SETLK call on DB file fails. This could be caused by an NFS locking
issue, for example. One solution for this issue, is to mv the DB away,
and copy it back so that it has a new Inode value
So, I would like to clarify for myself, when I should to avoid the locks? Can I read and write at the same time from two different threads? Thanks.
For those who are working with Android API:
Locking in SQLite is done on the file level which guarantees locking
of changes from different threads and connections. Thus multiple
threads can read the database however one can only write to it.
More on locking in SQLite can be read at SQLite documentation but we are most interested in the API provided by OS Android.
Writing with two concurrent threads can be made both from a single and from multiple database connections. Since only one thread can write to the database then there are two variants:
If you write from two threads of one connection then one thread will
await on the other to finish writing.
If you write from two threads of different connections then an error
will be – all of your data will not be written to the database and
the application will be interrupted with
SQLiteDatabaseLockedException. It becomes evident that the
application should always have only one copy of
SQLiteOpenHelper(just an open connection) otherwise
SQLiteDatabaseLockedException can occur at any moment.
Different Connections At a Single SQLiteOpenHelper
Everyone is aware that SQLiteOpenHelper has 2 methods providing access to the database getReadableDatabase() and getWritableDatabase(), to read and write data respectively. However in most cases there is one real connection. Moreover it is one and the same object:
SQLiteOpenHelper.getReadableDatabase()==SQLiteOpenHelper.getWritableDatabase()
It means that there is no difference in use of the methods the data is read from. However there is another undocumented issue which is more important – inside of the class SQLiteDatabase there are own locks – the variable mLock. Locks for writing at the level of the object SQLiteDatabase and since there is only one copy of SQLiteDatabase for read and write then data read is also blocked. It is more prominently visible when writing a large volume of data in a transaction.
Let’s consider an example of such an application that should download a large volume of data (approx. 7000 lines containing BLOB) in the background on first launch and save it to the database. If the data is saved inside the transaction then saving takes approx. 45 seconds but the user can not use the application since any of the reading queries are blocked. If the data is saved in small portions then the update process is dragging out for a rather lengthy period of time (10-15 minutes) but the user can use the application without any restrictions and inconvenience. “The double edge sword” – either fast or convenient.
Google has already fixed a part of issues related to SQLiteDatabase functionality as the following methods have been added:
beginTransactionNonExclusive() – creates a transaction in the “IMMEDIATE mode”.
yieldIfContendedSafely() – temporary seizes the transaction in order to allow completion of tasks by other threads.
isDatabaseIntegrityOk() – checks for database integrity
Please read in more details in the documentation.
However for the older versions of Android this functionality is required as well.
The Solution
First locking should be turned off and allow reading the data in any situation.
SQLiteDatabase.setLockingEnabled(false);
cancels using internal query locking – on the logic level of the java class (not related to locking in terms of SQLite)
SQLiteDatabase.execSQL(“PRAGMA read_uncommitted = true;”);
Allows reading data from cache. In fact, changes the level of isolation. This parameter should be set for each connection anew. If there are a number of connections then it influences only the connection that calls for this command.
SQLiteDatabase.execSQL(“PRAGMA synchronous=OFF”);
Change the writing method to the database – without “synchronization”. When activating this option the database can be damaged if the system unexpectedly fails or power supply is off. However according to the SQLite documentation some operations are executed 50 times faster if the option is not activated.
Unfortunately not all of PRAGMA is supported in Android e.g. “PRAGMA locking_mode = NORMAL” and “PRAGMA journal_mode = OFF” and some others are not supported. At the attempt to call PRAGMA data the application fails.
In the documentation for the method setLockingEnabled it is said that this method is recommended for using only in the case if you are sure that all the work with the database is done from a single thread. We should guarantee than at a time only one transaction is held. Also instead of the default transactions (exclusive transaction) the immediate transaction should be used. In the older versions of Android (below API 11) there is no option to create the immediate transaction thru the java wrapper however SQLite supports this functionality. To initialize a transaction in the immediate mode the following SQLite query should be executed directly to the database, – for example thru the method execSQL:
SQLiteDatabase.execSQL(“begin immediate transaction”);
Since the transaction is initialized by the direct query then it should be finished the same way:
SQLiteDatabase.execSQL(“commit transaction”);
Then TransactionManager is the only thing left to be implemented which will initiate and finish transactions of the required type. The purpose of TransactionManager – is to guarantee that all of the queries for changes (insert, update, delete, DDL queries) originate from the same thread.
Hope this helps the future visitors!!!
Not specific to SQLite:
1) Write your code to gracefully handle the situation where you get a locking conflict at the application level; even if you wrote your code so that this is 'impossible'. Use transactional re-tries (ie: SQLITE_LOCKED could be one of many codes that you interpret as "try again" or "wait and try again"), and coordinate this with application-level code. If you think about it, getting a SQLITE_LOCKED is better than simply having the attempt hang because it's locked - because you can go do something else.
2) Acquire locks. But you have to be careful if you need to acquire more than one. For each transaction at the application level, acquire all of the resources (locks) you will need in a consistent (ie: alphabetical?) order to prevent deadlocks when locks get acquired in the database. Sometimes you can ignore this if the database will reliably and quickly detect the deadlocks and throw exceptions; in other systems it may just hang without detecting the deadlock - making it absolutely necessary to take the effort to acquire the locks correctly.
Besides the facts of life with locking, you should try to design the data and in-memory structures with concurrent merging and rolling back planned in from the beginning. If you can design data such that the outcome of a data race gives a good result for all orders, then you don't have to deal with locks in that case. A good example is to increment a counter without knowing its current value, rather than reading the value and submitting a new value to update. It's similar for appending to a set (ie: adding a row, such that it doesn't matter which order the row inserts happened).
A good system is supposed to transactionally move from one valid state to the next, and you can think of exceptions (even in in-memory code) as aborting an attempt to move to the next state; with the option to ignore or retry.
You're fine with multithreading. The page you link lists what you cannot do while you're looping on the results of your SELECT (i.e. your select is active/pending) in the same thread.
What are the benefits of using implicit transactions (TransactionScope) over explicit transactions (SQLTransaction)?
I have spent some time investigating this and there are lots of websites that explain the differences between the two, which I do understand e.g. this one: http://sqlserverpedia.com/wiki/Transaction_Overview, and this one: named explicit & implicit transactions. I understand the properties of a Transaction i.e. ACID. I have also looked at the documentation on MSDN.
I believe that implicit transactions are more flexible and easier to use (because rollback is done for you and the transaction is not tied to a specific connection). I believe that explicit transactions give you more control e.g. when to rollback. I wanted to confirm: a) whether or not what I have said is true, b) Is there any criteria used to decide whether to use an implicit or explicit transaction.
On MSDN it says: "It is highly recommended that you use the easier implicit model for development"
Explicit transactions are a thing of the past. Don't use them unless you have a really good reason, and I can't think of any. TransactionScope is the way to go, practically always.
Don't worry about DTC, TransactionScope is smart enough to use the DTC only when two resources need to be coordinated. If you're accessing just one database, TransactionScope will be every bit as fast as explicit transactions.
I suggest avoiding using MSDTC if at all possible, unless you have the chops troubleshoot, write exhaustive troubleshooting guides on it, and write thorough explanation to users of your software about how to configure it. For example, users will need to know how to set up its authentication, how to ensure the DTC service is running make sure remote machines can communicate, how to configure SQL with FQDN for cross domain transactions, etc. It's been nothing but a problem for me, for years. Slowly have been winning the war to get it removed from the system. In other words, yes. Go ahead and use if it you want to guarantee issues with it for the next ten years. IMHO, it is not an enterprise class tool. There is nothing automagical about it at all.
All,
In Nhibernate I saw a lot of code written as:
using(ISession sess = factory.OpenSession()) {
using(ITransaction trans = sess.BeginTransaction()) {
// query, or save
trans.Commit();
}}
Starting transactions for queries or even single entity update always puzzled me why? Then after reading I learned that if you follow this pattern you will get 2 benefits:
Automatic connection release
Automatic flush
Fair enough.
My question is in regards to Castle ActiveRecord and connection pooling.
I am using Active Record in ASP.NET app, and the common pattern is to create a session for entire request.
My questions are:
1.
Should I use SessionScope or TransactionScope (and use start/end transaction to get/release my connection) so that I achieve efficient connection pooling,- ie. I want to hold on to the database connection (ADO.net connection object) during my persistence logic only, not for the entire life-time of the request (use connection semantics that are implemented by nhibernate transactions mentioned above)?
2.
Does SessionScope flushes when it goes out of scope (ie. in its Dispose method)?
LK (Answered): Yes, unless the scope is read-only.
3.
Does TransactionScope rollback when it goes out of scope (ie. in its Dispose method)?
LK (Answered): Depends on onDispose action, but by default it commits.
4.
Where's the official Castle Active Record documentation, - I see bits and pieces on various sites and various sites with broken links. Is there an official PDF documentation like for nhibernate or even a book?
You can use SessionScope. It will handle the common transaction semantics for you. You can use a TransactionScope when you need more fine grained control on the transaction. You can even open a TransactionScope within a SessionScope to handle multiple transactions within a single session.
The official documentation is here:
http://docs.castleproject.org/Active%20Record.MainPage.ashx
It is not great.
The old documentation is here:
http://old.castleproject.org/activerecord/documentation/trunk/index.html
The old stuff is better for some topics.
The topic of sql server deadlock has been discussed many times, however, I was unsure that even two simultaneous inserts on a table can end up in a deadlock situation.
Scenario:
While testing our application (SQL Server 2005 as backend, ASP.net 3.5) we inserted records into a table simultaneously (simplified overview) and that resulted into a deadlock for more than 70% of users.
I could not get a hang of this as how an insert is being deadlock as this is not a case of multiple resources. After a detailed analysis (of reproducing the bug by two users) I found that both the processes were holding a RangeS-S lock on the primary key index of the table and were trying to convert this into RangeI-N lock, that resulted into a deadlock and one transaction being killed.
Question:
Can we avoid or reduce these kind of deadlocks as this is not a case of change in order of access of resources? Cant' we force the transaction to get exclusive lock initially so that it blocks the other process and avoid deadlock? What (adverse) effects that may have?
Also could some one explain more about RangeI-N lock.
Isolation Level for this was "Serializable".
Any suggestion will be helpful.
Thanks,
Gaurav
Change your ADO isolation level. Unless you have clear requirements for Serializable, you shouldn't use it. If you do use it, then you must clearly understand the consequences, and frequent deadlocks due to range locks are one of these consequences.
The isolation level of System.Transactions is controlled by the IsolationLevel property.
Use sp_getapplock to acquire a custom exclusive lock
We are developing an ASP.NET HR Application that will make thousands of calls per user session to relatively static database tables (e.g. tax rates). The user cannot change this information, and changes made at the corporate office will happen ~once per day at most (and do not need to be immediately refreshed in the application).
About 2/3 of all database calls are to these static tables, so I am considering just moving them into a set of static objects that are loaded during application initialization and then refreshed every 24 hours (if the app has not restarted during that time). Total in-memory size would be about 5MB.
Am I making a mistake? What are the pitfalls to this approach?
From the info you present, it looks like you definitely should cache this data -- rarely changing and so often accessed. "Static" objects may be inappropriate, though: why not just access the DB whenever the cached data is, say, more than N hours old?
You can vary N at will, even if you don't need special freshness -- even hitting the DB 4 times or so per day will be much better than "thousands [of times] per user session"!
Best may be to keep with the DB info a timestamp or datetime remembering when it was last updated. This way, the check for "is my cache still fresh" is typically very light weight, just get that "latest update" info and check it with the latest update on which you rebuilt the local cache. Kind of like an HTTP "if modified since" caching strategy, except you'd be implementing most of it DB-client-side;-).
If you decide to cache the data (vs. make a database call each time), use the ASP.NET Cache instead of statics. The ASP.NET Cache provides functionality for expiry, handles multiple concurrent requests, it can even invalidate the cache automatically using the query notification features of SQL 2005+.
If you use statics, you'll probably end up implementing those things anyway.
There are no drawbacks to using the ASP.NET Cache for this. In fact, it's designed for caching data too (see the SqlCacheDependency class http://msdn.microsoft.com/en-us/library/system.web.caching.sqlcachedependency.aspx).
With caching, a dbms is plenty efficient with static data anyway, especially only 5M of it.
True, but the point here is to avoid the database roundtrip at all.
ASP.NET Cache is the right tool for this job.
You didnt state how you will be able to find the matching data for a user. If it is as simple as finding a foreign key in the cached set then you dont have to worry.
If you implement some kind of filtering/sorting/paging or worst searching then you might at some point miss the quereing capabilities of SQL.
ORM often have their own quereing and linq makes things easy to, but it is still not SQL.
(try to group by 2 columns)
Sometimes it is a good way to have the db return the keys of a resultset only and use the Cache to fill the complete set.
Think: Premature Optimization. You'll still need to deal with the data as tables eventually anyway, and you'd be leaving an "unusual design pattern".
With event default caching, a dbms is plenty efficient with static data anyway, especially only 5M of it. And the dbms partitioning you're describing is often described as an antipattern. One example: multiple identical databases for multiple clients. There are other questions here on SO about this pattern. I understand there are security issues, but doing it this way creates other security issues. I've recently seen this same concept in a medical billing database (even more highly sensitive) that ultimately had to be refactored into a single database.
If you do this, then I suggest you at least wait until you know it's solving a real problem, and then test to measure how much difference it makes. There are lots of opportunities here for Unintended Consequences.