I have a user table, for which UPDATE queries are extremely slow to complete. SELECT queries seem fine.
The DB is MariaDB 10.1.48.
CREATE TABLE `user` (
id int(10) UNSIGNED NOT NULL,
active tinyint(1) UNSIGNED NOT NULL DEFAULT '0',
email varchar(100) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
mobile varchar(50) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
isMobile_verified enum('0','1') NOT NULL DEFAULT '0',
device_type enum('0','1','none') NOT NULL DEFAULT 'none' COMMENT '1-Ios,0-Android',
device_token text,
password varchar(255) DEFAULT NULL,
updated datetime NOT NULL,
updated_by int(10) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
ALTER TABLE user
ADD PRIMARY KEY (id),
ADD UNIQUE KEY email (email),
ADD UNIQUE KEY mobile (mobile),
ADD KEY active (active);
ALTER TABLE user
MODIFY id int(10) UNSIGNED NOT NULL AUTO_INCREMENT;
Even via PHPMyAdmin or console, a very basic UPDATE query on an indexed column will take 4 to 5 seconds to run. As this is a core table for my application the cumulative effect causes some transactions to run for up to 40 seconds.
There are only c.190K records in the table. Other tables in the DB, including those of similar or larger size, do not seem to be affected.
Is there some issue with the DB version I should upgrade away from, or perhaps a cache problem? Proplem surfaced in the past few months and seems to be getting progressively worse with time. Any help greatly appreciated.
EDIT: Sample UPDATE query, with EXPLAIN output:
EXPLAIN UPDATE user SET isMobile_verified = 0 WHERE id = 1;
+---------+--------+--------+----------+----------+----------+-------+-------+-------------+
| select | table | type | keys | key | key_len | ref | rows | Extra |
+---------+--------+--------+----------+----------+----------+-------+-------+-------------+
| SIMPLE | user | range | PRIMARY | PRIMARY | 4 | NULL | 1 | Using where |
+---------+--------+--------+----------+----------+----------+-------+-------+-------------+
Foreign key constraints, specifically from other DBs, targeting the ID column had built up over time. Refactoring to remove those constraints (and allow for their removal!) resolves issue.
Related
As posted on another forum, when upgrading XWiki from v7.0.1 to v13.10.9, a non-critical database table xwikistatsvisit, for the user visiting statistics, was preventing the after-upgrade migrations. It contained over seven million records and sized 3GB in total. As a workaround, we had to delete all records in the table, but the SQL command delete from table xwikistatsvisit took over two hours.
I have verified from ER diagram that the table is stand-alone without any foreign key referring to or from other tables. And the database is MariaDB v10.9.2 installed on the same host.
The host under test is a medium virtual machine with SSD, 4 CPUs of Intel i9 and 8GB of RAM, running MariaDB v10.9.2. Also, the hypervisor needs to enable “PAE/NX” and “Nested VT-x/ADM-V” for higher performance; otherwise, the computing task will get stuck forever.
My Questions:
Why the SQL command took so long? Is there any way to proceed faster? E.g. disabling the keys and restrictions, etc.; but I am unfamiliar with this area.
I will highly appreciate any hints or suggestions.
The definition of the table xwikistatsvisit:
--
-- Table structure for table `xwikistatsvisit`
--
DROP TABLE IF EXISTS `xwikistatsvisit`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `xwikistatsvisit` (
`XWV_ID` bigint(20) NOT NULL,
`XWV_NUMBER` int(11) DEFAULT NULL,
`XWV_NAME` varchar(255) NOT NULL,
`XWV_CLASSNAME` varchar(255) DEFAULT NULL,
`XWV_IP` varchar(255) NOT NULL,
`XWV_USER_AGENT` longtext NOT NULL,
`XWV_COOKIE` longtext NOT NULL,
`XWV_UNIQUE_ID` varchar(255) NOT NULL,
`XWV_PAGE_VIEWS` int(11) DEFAULT NULL,
`XWV_PAGE_SAVES` int(11) DEFAULT NULL,
`XWV_DOWNLOADS` int(11) DEFAULT NULL,
`XWV_START_DATE` datetime DEFAULT NULL,
`XWV_END_DATE` datetime DEFAULT NULL,
PRIMARY KEY (`XWV_ID`),
KEY `XWVS_END_DATE` (`XWV_END_DATE`),
KEY `XWVS_UNIQUE_ID` (`XWV_UNIQUE_ID`),
KEY `XWVS_PAGE_VIEWS` (`XWV_PAGE_VIEWS`),
KEY `XWVS_START_DATE` (`XWV_START_DATE`),
KEY `XWVS_NAME` (`XWV_NAME`),
KEY `XWVS_PAGE_SAVES` (`XWV_PAGE_SAVES`),
KEY `XWVS_DOWNLOADS` (`XWV_DOWNLOADS`),
KEY `XWVS_IP` (`XWV_IP`),
KEY `xwv_user_agent` (`XWV_USER_AGENT`(255)),
KEY `xwv_classname` (`XWV_CLASSNAME`),
KEY `xwv_number` (`XWV_NUMBER`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = #saved_cs_client */;
The DELETE command deletes all the matching rows one at a time. If you want to delete all the records from such a big table, TRUNCATE should be a lot faster, since it empties the full table at once.
TRUNCATE TABLE xwikistatsvisit;
Difference between DELETE and TRUNCATE.
I'm trying to export/import a BD from one system to another but the import fails with the following error:
ERROR 1062 (23000) at line 8232: Duplicate entry '0-3-30168717-com_liferay_product_navigation_product_menu_web_...' for key 'IX_C7057FF7'
That table is defined as such:
CREATE TABLE `PortletPreferences` (
`portletPreferencesId` bigint(20) NOT NULL,
`ownerId` bigint(20) DEFAULT NULL,
`ownerType` int(11) DEFAULT NULL,
`plid` bigint(20) DEFAULT NULL,
`portletId` varchar(200) DEFAULT NULL,
`preferences` longtext DEFAULT NULL,
`mvccVersion` bigint(20) NOT NULL DEFAULT 0,
`companyId` bigint(20) DEFAULT NULL,
PRIMARY KEY (`portletPreferencesId`),
UNIQUE KEY `IX_C7057FF7` (`ownerId`,`ownerType`,`plid`,`portletId`),
In the mysql dump file, I see these two entries:
(31453178,0,3,30168717,'com_liferay_product_navigation_product_menu_web_portlet_ProductMenuPortlet','<portlet-preferences />',0,10132)
(31524539,0,3,30168717,'com_liferay_product_navigation_product_menu_web_portlet_ProductMenuPortlet','<portlet-preferences />',0,10132)
So, yep, there are two entries with the same unique key. How is that possible?!?
Knowing this, I ran the following select statement against the source DB:
select portletPreferencesId, ownerId, ownerType, plid, portletId from PortletPreferences where ownerId = 0 AND ownerType = 3 AND plid = 30168717 AND portletId like 'com_liferay_product_navigation_product_menu_web%';
And it outputs just ONE LINE!
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
| portletPreferencesId | ownerId | ownerType | plid | portletId |
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
| 31524539 | 0 | 3 | 30168717 | com_liferay_product_navigation_product_menu_web_portlet_ProductMenuPortlet |
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
By the portletPreferencesId field, it outputs the second entry in the dump file. So I did one more select for the other row as such:
select portletPreferencesId, ownerId, ownerType, plid, portletId from PortletPreferences where portletPreferencesId = 31453178;
And I get:
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
| portletPreferencesId | ownerId | ownerType | plid | portletId |
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
| 31453178 | 0 | 3 | 30168717 | com_liferay_product_navigation_product_menu_web_portlet_ProductMenuPortlet |
+----------------------+---------+-----------+----------+----------------------------------------------------------------------------+
My question is, what's going on?!? Why is that entry not output by the first select statement and why is it there in the first place if those fields were supposed to be unique???
I have a bad feeling about the state of the source database :-( Oh, and that's just one. I have multiple duplicate keys like that in that table :-(
Thanks
There are occasional bugs like MDEV-15250 that can cause duplicate entries to occur.
Sometimes you may not see these as parts of the optimizer expect only a single row because of the unique constraint so won't search beyond it. Maybe a query across a range including the entries would be more likely to show it (which is what mysqldump would have done).
If you don't think its related to ALTER TABLE occurring at the same time as insertions (like MDEV-15250), and have a good hunch as the set of operations on the table that may have escaped the unique key enforcement, can you please create a bug report.
Mariadb 10.3
MariaDB [demo]> show create table people;
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| people | CREATE TABLE `people` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`address_id` int(11) DEFAULT NULL,
`civility_id` int(11) DEFAULT NULL,
`marital_status_id` int(11) DEFAULT NULL,
`language_id` int(11) DEFAULT NULL,
`country_id` int(11) DEFAULT NULL,
`import_timestamp` int(11) DEFAULT NULL WITHOUT SYSTEM VERSIONING,
PRIMARY KEY (`id`),
FULLTEXT KEY `indexPeopleMultiWords` (`lastname`,`lastname_of_birth`,`firstname`)
) ENGINE=InnoDB AUTO_INCREMENT=5818 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci WITH SYSTEM VERSIONING
PARTITION BY SYSTEM_TIME
(PARTITION `p_hist` HISTORY ENGINE = InnoDB,
PARTITION `p_cur` CURRENT ENGINE = InnoDB) |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)
But when I try to drop SYSTEM VERSIONING, I got this message error :
MariaDB [nemo]> ALTER TABLE people DROP SYSTEM VERSIONING;
ERROR 4131 (HY000): Can not DROP SYSTEM VERSIONING for table `people` partitioned BY SYSTEM_TIME
Question:
How to drop system versioning in my case ? Thank you
ref: dbfiddle
PARITITION BY SYSTEM TIME is a system versioned construct.
Per the error message you received, you need to remove the partitioning in order to removing the system versioning:
ALTER TABLE people REMOVE PARTITIONING
ALTER TABLE people DROP SYSTEM VERSIONING
I have table my_transitions and I am trying to delete field called myStage.
I am using this command:
alter table `my_transitions` drop `myStage`
I am seeing this error:
Key column 'myStage' doesn't exist in table
But when I list all fields using this command:
describe my_transitions
I can see
id bigint(20) unsigned NO PRI NULL auto_increment
myStage varchar(255) NO MUL NULL
updated_at timestamp YES NULL
Anyone can see if I am doing something wrong?
EDIT:
If I run show create table my_transitions;, I am getting:
CREATE TABLE `my_transitions` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`myStage` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`myStage1` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_stage_combination` (`myStage`,`myStage1`),
KEY `my_transitions_myStage` (`myStage`),
KEY `my_transitions_myStage1` (`myStage1`)
) ENGINE=InnoDB AUTO_INCREMENT=26 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
I have solved it by first deleting the unique key
ALTER TABLE my_transitions DROP INDEX unique_stage_combination;
It seems like it is not possible to delete a column if it is a part of index key in Maria DB 10.5.8.
This is a pecular bug in MariaDB. It affects MariaDB 10.5.
Demo: https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=867204670347fa29e40bd5eb510c6956
The workaround is to drop the UNIQUE KEY that column mystage is part of first, then drop the column.
alter table my_transitions drop key unique_stage_combination, drop column mystage;
P.S.: I tested this on MySQL 8.0 and it does not require the workaround. It does drop the column, but it leaves the table with a UNIQUE KEY column on just one column mystage1, which might not be what you want.
I have the following SQLite table:
CREATE TABLE podcast_search (
_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
search TEXT NOT NULL UNIQUE
)
Whenever a user inserts/updates a row in the table, I want to sort that row at the end of the table. Thus, if I insert the following values:
_id | search | sort
===================
1 | foo | 1
2 | bar | 2
3 | quiz | 3
And then later update the 1 row from foo to foo2, the values should look like:
_id | search | sort
===================
2 | bar | 2
3 | quiz | 3
1 | foo2 | 4
I've implemented this thusly:
CREATE TABLE podcast_search (
_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
search TEXT NOT NULL UNIQUE,
update_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
)
CREATE TRIGGER update_date_update_trigger
AFTER UPDATE ON podcast_search FOR EACH ROW
BEGIN
UPDATE podcast_search
SET update_date = CURRENT_TIMESTAMP
WHERE _id = OLD._id;
END
However, my unit tests require a 1000ms sleep between insert/update operations in order to reliably sort, and this amount of delay is very annoying for unit testing.
I thought I could implement a vector clock instead, but it seems that AUTOINCREMENT values only exist for primary key columns. Does SQLite offer any other AUTOINCREMENT or AUTOINCREMENT-like option?
I'm running this on Android P, but this should be a generic SQLite problem.
UPDATE
I'm now using an sort INTEGER NOT NULL UNIQUE column, and SELECT-ing the largest row in that column and manually incrementing it before an INSERT/UPDATE:
CREATE TABLE podcast_search (
_id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
search TEXT NOT NULL UNIQUE,
sort INTEGER NOT NULL UNIQUE
)
SELECT sort from podcast_search ORDER BY sort DESC
either increment sort in application code, or set it to 0
Could I do this in a TRIGGER instead?
I thought I could implement a vector clock instead, but it seems that
AUTOINCREMENT values only exist for primary key columns. Does SQLite
offer any other AUTOINCREMENT or AUTOINCREMENT-like option?
They are not in fact AUTOINCREMENT values rather a column with AUTOINCREMENT will be an alias of the rowid column; not because AUTOINCREMENT has been coded but because INTEGER PRIMARY KEY has been coded.
All coding AUTOINCREMENT does is add a constraint that an auto-generated value MUST be greater than any other existing or used value. This only in fact becomes apparent if when a rowid with the value of 9223372036854775807 exists. In which case an attempt to insert a new row with an auto-generated rowid (i.e. no value is specified for the rowid column or an alias thereof) will result in an SQLITE_FULL error.
Without AUTOINCREMENT and when the highest rowid is 9223372036854775807 (the highest possible value for a rowid) an attempt is made to use a free value, which would obviously be lower than 9223372036854775807.
SQLite Autoincrement
You may wish to note the very first line of the linked page which says :-
The AUTOINCREMENT keyword imposes extra CPU, memory, disk space, and
disk I/O overhead and should be avoided if not strictly needed. It is
usually not needed.
I can't see any need from your description.
So what you want is a means of assigning a value for the column that is to be sorted that is 1 greater than the highest current value for that column, so it becomes the latest for sorting purposes, a subquery that retrieves max(the_column) + 1 would do what you wish. This could be in an UPDATE, TRIGGER or in an INSERT.
rowid = max(rowid) + 1 is basically how SQLite assigns a value to rowid unless AUTOINCREMENT is used when 1 is added to the greater of max(rowid) and the value, for the respective table, obtained from the table sqlite_sequence (will only exist if AUTOINCREMENT is used). It is referencing and maintaining sqlite_sequence that incurs the penalties.
For example you could use the following (which eliminates the need for an additional column and the additional index) :-
-- SETUP THE DATA FOR TESTING
DROP TABLE IF EXISTS podcast_searchv1;
CREATE TABLE IF NOT EXISTS podcast_searchv1 (
_id INTEGER NOT NULL PRIMARY KEY,
search TEXT NOT NULL UNIQUE
);
INSERT INTO podcast_searchv1 (search)
VALUES('foo'),('bar'),('guide')
;
-- Show original data
SELECT * FROM podcast_searchv1;
-- DO THE UPDATE
UPDATE podcast_searchv1 SET search = 'new value', _id = (SELECT max(_id) + 1 FROM podcast_searchv1) WHERE search = 'foo';
-- Show the changed data
SELECT * FROM podcast_searchv1;
The results being :-
and then :-