We have a MariaDB database running WordPress 4.8 and found a lot of transient named records in the wp_options table. The table was cleaned up with a Plugin and reduced from ~800K records down to ~20K records. Still getting slow query entries regarding the table:
# User#Host: wmnfdb[wmnfdb] # localhost []
# Thread_id: 950 Schema: wmnf_www QC_hit: No
# Query_time: 34.284704 Lock_time: 0.000068 Rows_sent: 1010 Rows_examined: 13711
SET timestamp=1510330639;
SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes';
Found another post to create an index and did:
ALTER TABLE wp_options ADD INDEX (`autoload`);
That was taking too long and taking website offline. I found a lot of 'Waiting for table metadata lock' in the processlist. After canceling the ALTER TABLE, got all running again still with high loads and entries of course in the slow query log. I also tried creating the index with the web server offline and a clean processlist. Should it take so long if I try to create again tonight?
If you are deleting most of a table, it is better to create a new table, copy the desired rows over, then rename. The unfortunate aspect is that any added/modified rows during the steps would not get reflected in the copied table. (A plus: You could have had the new index already in place.)
In this, I give multiple ways to do big deletes.
What is probably hanging your system:
A big DELETE stashes away all the old values in case of a rollback -- which killing the DELETE invoked! It might have been faster to let it finish.
ALTER TABLE .. ADD INDEX -- If you are using MySQL 5.5 or older, that must copy the entire table over. Even if you are using a newer version (that can do ALGORITHM=INPLACE) there is still a metadata lock. How often is wp_options touched? (Sounds like too many times.)
Bottom line: If you recover from your attempts, but the delete is still to be done, pick the best approach in my link. After that, adding an index to only 20K rows should take some time, but not a deadly long time. And consider upgrading to 5.6 or newer.
If you need further discussion, please provide SHOW CREATE TABLE wp_options.
But wait! If autoload is a simple yes/no 'flag', the index might not be used. That is, it may be a waste to add the index! (For low cardinality, it is faster to do a table scan than to bounce back and forth between the index BTree and the data BTree.) Please provide a link to that post; I want to spit at them.
Related
Something went wrong during a structure synchronization between two databases.
One of our production databases now is missing a key table 'customers' (which just about every other table has foreign keys to)
I'm trying to recreate the table from last night's backup (I don't want to restore the entire db - just recreate this table as the data in it does not change that much and I don't want to lose the transactional data from today)
The hassle seems to be that all the foreign key data for this table still exists in INFORMATION_SCHEMA.KEY_COLUMN_USAGE and I am getting 121 and 150 errors when I try run the CREATE TABLE query.
I've manually deleted all FK to the missing table and I am still getting errno 150 when trying to recreate the table. Any ideas where else there might be lost references to this table that is stopping me creating it again?
This was eventually resolved by multiple consultations of the SHOW ENGINE INNODB STATUS query.
The missing table had various indexes - example on the customer name there was an index "customer_name_idx". The CREATE TABLE query asked for this index to be created. The show engine innodb status return was "could not create table because index customer_name_idx already exists."
There was no reference to this index, to any primary key or to the table itself in any of the meta-data tables - I checked
INFORMATION_SCHEMA.INNODB_SYS_INDEXES
INFORMATION_SCHEMA.TABLE_SCHEMA
INFORMATION_SCHEMA.STATISTICS -INFORMATION_SCHEMA.TABLE
so I could not explain why this error was being thrown.
My guess, after the fact, is that MySQL is holding a cached copy of the information_schema meta data in memory and was consulting that, and maybe that only gets refreshed if you restart MySQL?
The solution was to give the indexes new names as a short term fix, and to rename them during our next scheduled downtime.
Once these were made, the table was created and the backup data could be reinstated.
How can you remove old records from the BAMPrimaryImport TDDS_FailedTrackingData table?
... not the TDDS_FailedTrackingData in the BizTalkDTADb database
Our production system has 2+ million records in the BAMPrimaryImport.dbo.TDDS_FailedTrackingData, and the various BizTalk SQL Agent jobs are running fine, but these records are still there.
UPDATE: We sorted the issue that was generating the fails (fingers crossed), so there are no new records.
This might be helpful for you as well:
http://www.codit.eu/blog/2014/07/03/maintaining-biztalk-bam-databases/
I'm not claiming this is an actual answer to your question, but it is about maintaining the BAM databases using NSVacuum.
Looks like it's a case of manually deleting the records (TRUNCATE TABLE or DELETE FROM) ...
I've used Red Gate's SQL Search and looked for TDDS_FailedTrackingData throughout the database ...
all objects and all databases
Found 8 references in the entire system ... see below
Records are removed from the [BizTalkDTADb].[dbo].[TDDS_FailedTrackingData] in two stored procedures ...
[dtasp_CleanHMData] does a TRUNCATE TABLE
[dtasp_PurgeTrackingDatabase_Internal] does a DELETE FROM for 100 records at a time
However the [BAMPrimaryImport] database only has one stored procedure that has any mention of the [BAMPrimaryImport].[dbo].[TDDS_FailedTrackingData] table ...
[BAMPrimaryImport].[dbo].[TDDS_InsertFailedTrackingData]
and it just inserts records, with the addition of current date & time from GETUTCDATE()
Found lots of posts about clearing down the [BizTalkDTADb] table, but very few on clearing down the [BAMPrimaryImport]
This on TechNet from a BizTalk MVP
And this on MSDN from another BizTalk expert.
You can manually perform a simple DELETE TSQL script :
DELETE FROM [BAMPrimaryImport].[dbo].[TDDS_FailedTrackingData]
I would like to monitor 10 tables with 1000 records per table. I need to know when a record, and which record changed.
I have looked into SQL Dependencies, however it appears that SQL Dependencies would only be able to tell me that the table changed, and not which record changed. I would then have to compare all the records in the table to find the modified record. I suspect this would be a problem for me as the records constantly change.
I have also looked into SQL Trigger's, however I am not sure if triggers would work for monitoring which record changed.
Another thought I had, is to create a "Monitoring" table which would have records added to it via the application code whenever a record is modified.
Do you know of any other methods?
EDIT:
I am using SQL Server 2008
I have looked into Change Data Capture which is available in SQL 2008 and suggested by Martin Smith. Change Data Capture appears to be a robust, easy to implement and very attractive solution. I am going to roll CDC on my database.
You can add triggers and have them add rows to an audit table. They can audit the primary key of the rows that changed, and even additional information about the changes. For instance, in the case of an UPDATE, they can record the columns that changed.
Before you write/implement your own take a look at AutoAudit :
AutoAudit is a SQL Server (2005, 2008) Code-Gen utility that creates
Audit Trail Triggers with:
Created, CreatedBy, Modified, ModifiedBy, and RowVersion (incrementing INT) columns to table
Insert event logged to Audit table
Updates old and new values logged to Audit table
Delete logs all final values to the Audit table
view to reconstruct deleted rows
UDF to reconstruct Row History
Schema Audit Trigger to track schema changes
Re-code-gens triggers when Alter Table changes the table
What version and edition of SQL Server? Is Change Data Capture available? – Martin Smith
I am using SQL 2008 which supports Change Data Capture. Change Data Capture is a very robust method for tracking data changes as I would like to. Thanks for the answer.
Here's an idea.You can have a flag on each table that every time a record is created or updated is filled with current datetime. Then when you notice that a record has changed set its flag to null again.Thus unchanged records have null in their flag field and you can query not null values to see which record has changed/created and when (and set their flags to null again) .
We have a table(say T1) that is referenced by about 16 other tables with foreign keys in our SQL Server database. The data is accessed through an ASP.NET application with LINQToSQL. When the user tried to delete a record from T1 the statement would time out. So we decided to first delete the records from the tables that reference T1 and only then delete the record in T1. The problem is that deletion from T1 does not work as fast as expected.
My question is: is it normal that deletion from a table referenced by many other tables to be so time-consuming even if the record itself does not have any 'children' records?
EDIT: Apparently the cause for the timeout was not the delete itself but another query that retrieved data from the same DataContext. Thank you for your suggestions, I have marked as answer the suggestion to add indexes for all foreign keys because it improved our script's execution plan.
I suspect that you may need to look into the indexing on your child tables.
It sounds as if you FKs are set to Cascade Deletes, so I would suspect that some of your tables do not have an index that includes the key to the parent as the first in the index.
In this way your delete will be full scanning the child tables - even if you've already deleted the child records it will still check as you've still got the Cascade set.
When you define a relationship in DB, you can set the Delete rule as Cascade in SQL server. In this way, when you delete the record from the parent table, it will be automatically deleted from the child tables.
Please see the image below:
If it taking long time, you may have set other constraint that will slow
down the process of deletion.
Linq does not do bulk deletes if you're having it operate directly on the record set -- instead, it is probably deleting one record at a time.
To improve performance, use a stored procedure instead for any bulk insert, update or delete operations.
I need to modify a column in a SQLite database but I have to do it programatically due to the database already being in production. From my research I have found that in order to do this I must do the following.
Create a new table with new schema
Copy data from old table to new table
Drop old table
Rename new table to old tables name
That seems like a ridiculous amount of work for something that should be relatively easy. Is there not an easier way? All I need to do is change a constraint on a existing column and give it a default value.
That's one of the better-known drawbacks of SQLite (no MODIFY COLUMN support on ALTER TABLE), but it's on the list of SQL features that SQLite does not implement.
edit: Removed bit that mentioned it may being supported in a future release as the page was updated to indicate that is no longer the case
If the modification is not too big (e.g. change the length of a varchar), you can dump the db, manually edit the database definition and import it back again:
echo '.dump' | sqlite3 test.db > test.dump
then open the file with a text editor, search for the definition you want to modify and then:
cat test.dump | sqlite3 new-test.db
As said here, these kind of features are not implemented by SQLite.
As a side note, you could make your two first steps with a create table with select:
CREATE TABLE tmp_table AS SELECT id, name FROM src_table
When I ran "CREATE TABLE tmp_table AS SELECT id, name FROM src_table", I lost all the column type formatting (e.g., time field turned into a integer field
As initially stated seems like it should be easier, but here is what I did to fix. I had this problem b/c I wanted to change the Not Null field in a column and Sqlite doesnt really help there.
Using the 'SQLite Manager' Firefox addon browser (use what you like). I created the new table by copying the old create statement, made my modification, and executed it. Then to get the data copied over, I just highlighted the rows, R-click 'Copy Row(s) as SQL', replaced "someTable" with my table name, and executed the SQL.
Various good answers already given to this question, but I also suggest taking a look at the sqlite.org page on ALTER TABLE which covers this issue in some detail: What (few) changes are possible to columns (RENAME|ADD|DROP) but also detailed workarounds for other operations in the section Making Other Kinds Of Table Schema Changes and background info in Why ALTER TABLE is such a problem for SQLite. In particular the workarounds point out some pitfalls when working with more complex tables and explain how to make changes safely.