After a change in code, my database reports a lot of deadlock incidents, which are resolved after a while ( 1 < x < 5 minutes). I use SHOW ENGINE INNODB STATUS to view what happened but the information is not correct: statements and tables reported here are different from what I see in INNODB_LOCKS table (when it's not resolved yet).
The question is: How can I get a history, a log, of what deadlocks happened? not just the most recent one.
innodb_print_all_deadlocks = ON
SHOW ENGINE INNODB STATUS is transient; the above should persist it by writing to a log file.
I recommend a few things to decrease the number of deadlocks:
Do thing is the same order in different transactions. This includes which rows are touched.
Speed up the code. (Better indexes, often 'composite', reformulate queries, etc)
See if anything can reasonably be pulled out of the BEGIN...COMMIT.
For further discussion, please show us the SQL in a transaction, plus SHOW CREATE TABLE for the relevant tables.
In any case, test for errors throughout each transaction and be ready to replay when you hit a deadlock.
Note: lock_wait_timeout is a related item. It defaults to an unreasonably high 50 seconds. If you raise that you are asking for more trouble. Decreasing it (to, say, 5) is not a real solution, but it may change the problems in interesting ways. Again, test for errors and react to them. Hitting this "timeout" is as bad as a "deadlock". Not hitting it, but waiting, is a silent way that InnoDB resolves conflicts successfully.
Related
I'm facing a weird phenomenal when putting items to DynamoDB.
It seems like if putting items too fast, DynamoDB can't put the whole data to the table(kinda like a broken data, it has partial attributes but with some weird values)?
I'm using the AWS JavaScript SDK to putting items, no errors shown up, everything seemed to work fine, but once I checked the data from web console, some of the inserted data was broken. Is this is related to write capacity units? (but no errors tell me it's caused by the write capacity units..) I could confirm the spike of my write capacity units was about 60/min, the setting is "on-demand".
I tried to slow down the putting speed with one second interval and with the exactly same data, the data could be inserted correctly...
Anyone knows why and how to fix this..?
The answer is no: If DynamoDB decides to throttle your requests because you exceeded your provisioned capacity or exceed their own hardware's capacity or whatever - it will refuse to do whole requests, or in the case of BatchWriteItems do some of the writes and not do others (and it will tell you which it did and which it didn't). DynamoDB will never write part of a request or corrupt parts of one attribute.
If you are seeing that, the most likely culprit is a bug in your own code that does the write. Maybe your own code is not thread-safe, so if your code is trying to prepare two items for writing concurrently, the code doing this preperation has a data race and results in a corrupt item to be written. Obviously, it is also possible that DynamoDB has a bug causing this, but it can't be as simple a bug as "writing more than 60 items a minute causes corruption" - if this were the case, everyone would have encountered this bug...
One of the main reason for using firestore batche writes is that they are atomic and ensure data consistency. However they have a limit of 500 operations. Considering a large application, one may have denormalized user data in more than 500 documents. So when a user updates any of his/her profile details, I have to update it in all those more than 500 documents while maintaining data consistency (atomic updates) at the same time.
An intuitive solution would be maintaining an array of batches, and keeping track of those which fail, and then retry the failed batches manually.
However I want to ask that:
1) If there are any best practices or some other more easy and reliable methods of achieving this, because considering the limit 500 operations per batch, most of the commercial apps have to face the same issue.
2) Also is there a more smart approach present out there than just denormalizing data, so that through "that smart approach", this whole issue of data consistency (as stated above) can be avoided in the first place.
An intuitive solution would be maintaining an array of batches, and keeping track of those which fail, and then retry the failed batches manually.
That's a viable solution that you can go ahead with.
1) If there are any best practices or some other more easy and reliable methods of achieving this, because considering the limit 500 operations per batch, most of the commercial apps have to face the same issue.
I can tell you what I do. I usually create a counter variable and increment its value every time I add an update operation to the batch. Then create an if statement and every time you increment the counter, check to see if it reached 500. At that time, commit the current batch, reset the counter and start a new batch, picking up where you left off. Do this till you finish all batch writes.
2) Also is there a more smart approach present out there than just denormalizing data, so that through "that smart approach", this whole issue of data consistency (as stated above) can be avoided in the first place.
The problem of the batch writes cannot be solved with the help of denormalization. Duplicating data, isn't a solution.
I have a AX2012R2 CU6 (build&client 6.2.1000.1437, kernel 6.2.1000.5268) with the following problem:
On AP>Journals>Invoices>Invoice Journal>lines (form LedgerJournalTransVendInvoice), when I select Vendor as Account type and then activate the lookup on the Account field, AX freezes for a couple minutes and when it recovers, the lookup is closed/never opened. This happens every time when account type vendor, other account types work just fine.
I debugged this to LedgerJournalEngine.accountNumLookup() --> VendTable.lookupVendor line
formSegmentedEntryControl.performFormLookup(formRun);
The above process takes up the time.
Any ideas before I hire an exorcist?
There is a known KB for this for R3, look for it on Lifecycle services
KB 3086961 Performance issue of VendorLookup on the volume data,
during the GFM Bugbash 6/11 took over 30 minutes
Even though the fix is for R3 it should be easy to backport as the changes are described as
The root cause seemed to be the DirPartyLookupGridView, which had
around 14 joins on views and tables. This view is used in many places
and hence seemed to have grown quite a lot over time.
The changes in the hotfix remove the view and add only the required
datasources - dirpartytable and logisticsaddress to the
VendTableLookup form.
The custtableLookup is not using the view and using custom datasource
joins instead, so no changes there.
Try implementing that change and see what happens.
I'm not sure this will fix your issue as in your execution plan the only operation that seems really expensive is the sort operator which needs to spill to tempdb (you might need more memory to solve that) but the changes in the datasource could have the effect of removing the sort operator from the execution plan as the data may be sorted by an index.
Probably the SQL Server chose the wrong query plan.
First check that you have not disabled any indexes on the involved tables, then do a synchronize on them.
If still a problem, then to run a STATISTICS UPDATE on the involved tables (including the tables in the view).
I want to implement a views counter like most forums, Youtube and several others have. So every time a user reads an article, that is stored and remembered. I also want to know who looked at the article.
My queston is: How do you implement this efficiently? What is the best practice?
One way would be to call a stored procedure for every view, but that would result in a lot of unneeded calls to the database.
Another way would be to store this to some global application object, and then store in DB every 5 minutes or so (and can you even do that in a good way?)
What's the best way to do this?
Database operations are surprisingly cheap and really are not worth worrying about. In the event that a DB operation was even marginally expensive then you can always delegate the blocking operation to a new thread thus freeing-up your page-generation thread (you can trivially do this for UPDATE and INSERT operations that return nothing from the database - they are inconsequential).
Sprocs aren't really in-fashion right now - the performance advantage they might have had from pre-computed execution plans is almost eliminated because modern servers cache plans from all previous queries, and for trivial SELECT, INSERT, and UPDATEs you begin to suffer from increased code complexity. There's nothing wrong with inline SQL commands now.
Anyway, back on-topic and in summary: your assumptions are wrong. There is nothing wrong with running UPDATE Pages SET ViewCount = ViewCount + 1 WHERE PageId = #pageId on every page-view. There is also nothing wrong with doing this either: INSERT INTO UserPageviews (UserId, PageId, DateTime) VALUES ( #userId, #pageId, NOW() ). Both operations are very cheap and will execute in under 2-3 miliseconds on even an old and aged database server.
Another way would be to store this to some global application object,
and then store in DB every 5 minutes or so (and can you even do that
in a good way?)
This method is very prone to data loss unless you use a durable queueing mechanism (like MSMQ). Unless you anticipate massive traffic, I wouldn't even think about this approach.
Writes of this nature are inexpensive and hundreds of operations per second are not a big deal. I recently built a comment/rating framework that acheives throughput of 3000+ complete transactions per second just on my local all-in-one workstation. This included processing the request, validation, and creating multiple records within a transaction.
As a note, you should take steps to ensure that your statistics data isn't vulnerable to artificial inflation/manipulation. This part of the process will probably be more complex than the view tracking itself. For example, a user should not be able to sit and hold down the F5 key and inflate the number of views on their video. Nor should these values be manipulable by HTTP (e.g. creating a small script to send an AJAX request over and over).
This suggests that each INSERT would be preceded by a SELECT to ensure that the same user ID or IP hadn't already been recorded in some period of time. Of course, this isn't foolproof (unless you invest a great deal of effort), but it errs on the side of conservatism which is usually a good approach.
One way would be to call a stored procedure for every view, but that
would result in a lot of unneeded calls to the database.
I regularly have to remind myself (and other developers) to not fear the database. People (me included) sometimes go to great lengths to avoid a few simple database calls. Keep your tables narrow and well-indexed, and operations like this are faster than you might think.
We've developed a system with a search screen that looks a little something like this:
(source: nsourceservices.com)
As you can see, there is some fairly serious search functionality. You can use any combination of statuses, channels, languages, campaign types, and then narrow it down by name and so on as well.
Then, once you've searched and the leads pop up at the bottom, you can sort the headers.
The query uses ROWNUM to do a paging scheme, so we only return something like 70 rows at a time.
The Problem
Even though we're only returning 70 rows, an awful lot of IO and sorting is going on. This makes sense of course.
This has always caused some minor spikes to the Disk Queue. It started slowing down more when we hit 3 million leads, and now that we're getting closer to 5, the Disk Queue pegs for up to a second or two straight sometimes.
That would actually still be workable, but this system has another area with a time-sensitive process, lets say for simplicity that it's a web service, that needs to serve up responses very quickly or it will cause a timeout on the other end. The Disk Queue spikes are causing that part to bog down, which is causing timeouts downstream. The end result is actually dropped phone calls in our automated VoiceXML-based IVR, and that's very bad for us.
What We've Tried
We've tried:
Maintenance tasks that reduce the number of leads in the system to the bare minimum.
Added the obvious indexes to help.
Ran the index tuning wizard in profiler and applied most of its suggestions. One of them was going to more or less reproduce the entire table inside an index so I tweaked it by hand to do a bit less than that.
Added more RAM to the server. It was a little low but now it always has something like 8 gigs idle, and the SQL server is configured to use no more than 8 gigs, however it never uses more than 2 or 3. I found that odd. Why isn't it just putting the whole table in RAM? It's only 5 million leads and there's plenty of room.
Poured over query execution plans. I can see that at this point the indexes seem to be mostly doing their job -- about 90% of the work is happening during the sorting stage.
Considered partitioning the Leads table out to a different physical drive, but we don't have the resources for that, and it seems like it shouldn't be necessary.
In Closing...
Part of me feels like the server should be able to handle this. Five million records is not so many given the power of that server, which is a decent quad core with 16 gigs of ram. However, I can see how the sorting part is causing millions of rows to be touched just to return a handful.
So what have you done in situations like this? My instinct is that we should maybe slash some functionality, but if there's a way to keep this intact that will save me a war with the business unit.
Thanks in advance!
Database bottlenecks can frequently be improved by improving your SQL queries. Without knowing what those look like, consider creating an operational data store or a data warehouse that you populate on a scheduled basis.
Sometimes flattening out your complex relational databases is the way to go. It can make queries run significantly faster, and make it a lot easier to optimize your queries, since the model is very flat. That may also make it easier to determine if you need to scale your database server up or out. A capacity and growth analysis may help to make that call.
Transactional/highly normalized databases are not usually as scalable as an ODS or data warehouse.
Edit: Your ORM may have optimizations as well that it may support, that may be worth looking into, rather than just looking into how to optimize the queries that it's sending to your database. Perhaps bypassing your ORM altogether for the reports could be one way to have full control over your queries in order to gain better performance.
Consider how your ORM is creating the queries.
If you're having poor search performance perhaps you could try using stored procedures to return your results and, if necessary, multiple stored procedures specifically tailored to which search criteria are in use.
determine which ad-hoc queries will most likely be run or limit the search criteria with stored procedures.. can you summarize data?.. treat this
app like a data warehouse.
create indexes on each column involved in the search to avoid table scans.
create fragments on expressions.
periodically reorg the data and update statistics as more leads are loaded.
put the temporary files created by queries (result sets) in ramdisk.
consider migrating to a high-performance RDBMS engine like Informix OnLine.
Initiate another thread to start displaying N rows from the result set while the query
continues to execute.