One of our MariaDB users decided to copy a database via a shell script that pipes the output of a mysqldump to another table.
If you restore the mysqldump to a database that hasn't been created before, it works just fine. However, if you then drop that database, re-create it, and then try to run the script again, it throws the following error:
ERROR 1005 (HY000) at line 25: Can't create table
core_dev.addresses (errno: 121 "Duplicate key on write or update")
mysqldump: Got errno 32 on write
SHOW ENGINE INNODB STATUS yields the following insight:
2017-05-03 09:34:19 7f9929fb4b00 Error in foreign key constraint
creation for table core_dev.addresses. A foreign key constraint of
name core_dev.addresses_ibfk_2 already exists. (Note that
internally InnoDB adds 'databasename' in front of the user-defined
constraint name.) Note that InnoDB's FOREIGN KEY system tables store
constraint names as case-insensitive, with the MySQL standard
latin1_swedish_ci collation. If you create tables or databases whose
names differ only in the character case, then collisions in constraint
names can occur. Workaround: name your constraints explicitly with
unique names.
However, the core_dev database is entirely empty:
MariaDB [(none)]> SHOW TABLES IN core_dev;
Empty set (0.01 sec)
What could the problem be? I'm not able to replicate this problem on any of our other MariaDB 10.1 servers. The only difference is that this particular one is running on CentOS 6, while the others are on CentOS 7.
Related
What is the right way to backup and restore a MariaDB database that has sequence generation enabled (i.e. NOT autoincrement)? (This includes migrating to a new server.)
Is it possible to instruct the sequence generator to pick up indexing table data at a specific ID value? How?
Steps I take to create my issue
I wish to transfer an application to a new server:
Backup data on source server:
mysqldump --skip-opt --no-create-db --no-create-info --hex-blob [database-name] [...list of tables...] > data-backup.sql
On target server, create new empty database (same name)
Build/run JHipster Spring application on target server: java -jar myapp.jar (Running this application recreates/configures a new instance of the database on the target server.)
Restore data:
mysql [database-name] < data-backup.sql
All the above steps produce no errors (so far).
Problem
When I follow these steps, the database is restored (apparently perfectly). I can log in to the application and access all information. BUT when I attempt to create new entities (i.e. save something to the database), I get an ID 'Duplicate entry' error in the server logs:
2022-03-24 12:54:43.775 ERROR 11277 --- [ XNIO-1 task-1] o.h.e.jdbc.batch.internal.BatchingBatch : HHH000315: Exception executing batch [java.sql.BatchUpdateException: (conn=33) Duplicate entry '1001' for key 'PRIMARY'], SQL: insert into product (name, id) values (?, ?)
2022-03-24 12:54:43.776 WARN 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1062, SQLState: 23000
2022-03-24 12:54:43.776 ERROR 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=33) Duplicate entry '1001' for key 'PRIMARY'
2022-03-24 12:54:43.779 ERROR 11277 --- [ XNIO-1 task-1] o.z.problem.spring.common.AdviceTraits : Internal Server Error
org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [insert into product (name, id) values (?, ?)]; constraint [PRIMARY]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:276)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:566)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at com.mycompany.app.web.rest.ProductResource$$EnhancerBySpringCGLIB$$84c14d6d.createProduct(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
Clearly my backup/restore process is not accounting properly for the sequence generator, which generates ID values that conflict with the existing data.
What I am doing wrong? What is the right process of backing up/restoring such a database?
Environment: JHipster 7.7.0 (Angular, monolithic), MariaDB 10.4, OpenJDK 16.0.2_7, OS Windows 10 Pro and openSUSE 15.2, Firefox 98.0.2 and Chrome 99.0.4844.84.
PS: I previously reported this issue here, aimed at the JHipster community, but got limited response. I think I need a MySQL/MariaDB expert opinion on this.
(Apologies in advance: I'm not a database expert. The technique I outline above has served me well for years, but previously I was dealing with AUTO_INCREMENT. This sequence generator has me baffled.)
Ok! I have solutions.
[For the sake if these notes, let's call the database: mydata. Also, in JHipster, the MariaDB sequence generator is called: sequence_generator]
Let's consider two situations:
(1) Simple migration
If you are merely migrating the application to a new server, the process is straight forward:
Step 1: On the original server backup and secure your database: mysqldump -u root -p mydata > mydata.sql
Step 2: Transfer the SQL file to the new server, along with the JHipster JAR file
Step 3: On the new server, create an empty database with the same name, and restore the data: mysql -u root -p mydata < mydata.sql
Step 4: Now launch your JHipster application, and everything should work
(2) Model modification
The assumption is that you have modified your model in some way (e.g. added properties to one or more entities). This solution is fiddly, but it works (for me).
Step 1: Backup your database, and secure it (in case something goes wrong): mysqldump -u root -p mydata > mydata.sql
Step 2: Backup and secure the original JHipster JAR that works with the original database
Step 3: Duplicate your database (schema and data) in a new table: mydata_bk
Step 4: Drop your original database, and create a new empty database
Step 5: Launch your new JHipster JAR, and give it time to create the new database schema, then stop the application
Step 6: Use a tool (DataGrip, sqlYog, etc) to compare the old (mydata_bk) and new schema (mydata), and modify the old schema to match the new schema
Step 7: Restore/copy all data from mydata_bk to mydata, EXCEPT for the tables DATABASECHANGELOG, DATABASECHANGELOGLOCK and the special sequence_generator table
Step 8: Open the mydata.sql SQL file, and at the top, after initial comments, one of the first instructions will read:
--
-- Sequence structure for `sequence_generator`
--
DROP SEQUENCE IF EXISTS `sequence_generator`;
CREATE SEQUENCE `sequence_generator` start with 2000 minvalue 1 maxvalue 9223372036854775806 increment by 50 cache 1000 nocycle ENGINE=InnoDB;
SELECT SETVAL(`sequence_generator`, 201050, 0);
The specific numbers may vary, but the broad details will be similar. In a MariaDB SQL console type/execute each of those SQL statements: DROP SEQUENCE ...;, CREATE SEQUENCE ...;, and SELECT SETVAL(...);
Step 9: Launch your JHipster application.
Hope this helps others that run into similar issues. Let me know if you have a better approach!
I use PHPUnit to test by PDO wrapper. One of my tests is an integration test that proves that the abstraction layer produces the right syntax, or binds PDOStatement in the right way, etc. For this, I use an in-memory SQLite database, and declare ext-pdo_sqlite as a dev-dependency of my project.
When Github Actions runs a build, dependencies all get installed, hinting that pdo_sqlite is present on the system. This is further confirmed by the fact that the connection succeeds. However, when I run an insert query to set up my database, I get the following error:
PDOException: SQLSTATE[HY000]: General error: 1 no such column: TRUE
The query looks like this:
INSERT INTO `persons` (`id`, `name`, `dob`, `weight`, `is_dev`, `teeth`) VALUES (NULL, 'Anton', '1987-11-03', 71.3, TRUE, 31)
Table creation is done with PDO like this:
$pdo->exec("CREATE TABLE `$tableName` (`id` int(6) PRIMARY KEY, `name` varchar(255), `dob` datetime, `weight` float(5), `is_dev` bool, `teeth` int(2))");
This only manifests on CI, and local tests pass fine - even on the same PHP version as CI.
Any ideas? Am I just missing something in the syntax? Could it be that GH Actions has only a dummy implementation of SQLite or something?
Note: This is an open source project, and you can see everything, including the failing build, in dhii/pdo-query#1.
I just discovered the silly new issue of MariaDB's latest version having mysql.user as a view. All my imported Wordpress databases suddenly cannot connect from the blogs. When I try to even list mysql.user it shows me this:
> select * from mysql.user;
ERROR 1356 (HY000): View 'mysql.user' references invalid table(s)
or column(s) or function(s) or definer/invoker of view lack
rights to use them
What can we do to solve this?
Edit: Found this question, but it does not have a solution, only a suggestion. The ALTER USER command -- where to use and with what settings? Do I have to somehow alter the rights for every Blog database?
Update:
Further investigation revealed that the issue described in this Question, and my initial response to it (below) may be related to an Incorrect definition of table mysql.event problem. In my case, I had 1) loaded a full dump (including the mysql database) from MySQL 5.7.33 to a fresh installation of MariaDB10.5.9; 2) discovered that this was not a good idea; 3) edited my dump file to exclude the mysql database, and 4) repeated the load without deleting any databases or configurations.
This caused the database to function properly, but (in addition to the issue described in this Question) a) /usr/sbin/mariadbd --verbose --help would try to run the database server rather than print help, b) on startup the following error always occurred:
Apr 05 08:52:46 xxx mariadbd[22668]: 2021-04-05 8:52:46 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPTY_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_B
Apr 05 08:52:46 xxx mariadbd[22668]: 2021-04-05 8:52:46 0 [ERROR] mariadbd: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler
Today, I was able to correct these problems (under Amazon Linux 2) by:
Uninstalling MariaDB-server and MariaDB-client
Removing /etc/my.*
Removing /var/lib/mysql
Reinstalling MariaDB-server and MariaDB-client
Reloading the database dump, again omitting the dump of the mysql database
At this point, I not only have clean database startup and proper operation of /usr/sbin/mariadbd --verbose --help, I also find that select * from mysql.user works properly!
So the problem of not being able to select from mysql.user appears not to have resulted from the change of mysql.user from table to view as I had originally thought, but from some other issue related to my "improper" database migration.
My initial answer:
(included as a reference only)
After considerable research I have found at least part of the answer to this question:
tl;dr: select * from mysql.global_priv
then for each User,
show grants for 'XXX'#'localhost';
Longer version, from Authentication in MariaDB 10.4 — Understanding the Changes:
The password storage has changed. All user accounts, passwords, and global privileges are now stored in a mysql.global_priv table. What happened to the mysql.user table? It still exists and has exactly the same set of columns as before, but it’s now a view over mysql.global_priv...."
The aforementioned article provides not only what what but also the why. I do not agree with all of it. In particular the claim is made that Old mysql.user table still exists, you can select from it as before, but you cannot (hence this question). Nonetheless I am relieved to discover a relatively coherent explanation from MariaDB.
Finally, here is an example:
MariaDB [(none)]> select * from mysql.global_priv\G
*************************** 1. row ***************************
Host: localhost
User: mariadb.sys
Priv: {"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}
*************************** 2. row ***************************
Host: localhost
User: root
Priv: {"access": 1844674407370915, "plugin": "mysql_native_password", "authentication_string": "*9A87226E872127C756290C504DB5D9076E", "auth_or": [{}, {"plugin": "unix_socket"}], "password_last_changed": 1617303275}
*************************** 3. row ***************************
Host: localhost
User: mysql
Priv: {"access":1844674407371615,"plugin":"mysql_native_password","authentication_string":"invalid","auth_or":[{},{"plugin":"unix_socket"}]}
*************************** 4. row ***************************
MariaDB [(none)]> show grants for 'root'#'localhost'\G
*************************** 1. row ***************************
Grants for root#localhost: GRANT ALL PRIVILEGES ON *.* TO `root`#`localhost` IDENTIFIED VIA mysql_native_password USING '*9A87226E872127C756290C5BF177504DB5D9076E' OR unix_socket WITH GRANT OPTION
*************************** 2. row ***************************
Grants for root#localhost: GRANT PROXY ON ''#'%' TO 'root'#'localhost' WITH GRANT OPTION
Im using xampp control panel and from there i start the process for apache and mysql. Then i go to mysql workbench and server status seems to be ok, here is some info
Host: Windows-PC
Socket: C:/xampp/mysql/mysql.sock
Port: 3306
Version 10.1.31-MariaDB mariadb.org binary distribution
Compiled For: Win32(32)
Configuratin File: unknown
Then everytime when i try to add the foreign key for my dummy schema like:
ALTER TABLE `puppies`.`animals`
ADD INDEX `Breed_idx` (`BreedID` ASC) VISIBLE;
;
ALTER TABLE `puppies`.`animals`
ADD CONSTRAINT `Breed`
FOREIGN KEY (`BreedID`)
REFERENCES `puppies`.`breeds` (`Breed`)
ON DELETE NO ACTION
ON UPDATE NO ACTION;
I get the following error
ERROR 1064: You have an error in your SQL syntax; check the manual that
corresponds to your MariaDB server version for the right syntax to use near
'' at line 2
SQL Statement:
ALTER TABLE `puppies`.`animals`
ADD INDEX `Breed_idx` (`BreedID` ASC) VISIBLE
So what can i do so that xampp will start using mysql syntax over mariaDb?
Or if im wrong in my understanding of the problem, then what should i do so that i dont have to face this kind of issues again when using xampp?
Problem is the word VISIBLE, remove it and it will work.
Index are visible by default.
Your question: "If i remove VISIBLE it works just fine, so why did mysql workbench decided to add visible?"
My answer: The option to mark index invisible is not yet implemented in MariaDB (afaik!).
Update:
The syntax for MariaDB is different, please see this reference: https://jira.mariadb.org/browse/MDEV-7317
Just to add to those who are using Maria DB with MySQL Workbench, you don't need to install mysql. You can just change 'Default Target MySQL Version' from Preferences to 5.7 or 5.6, and the VISIBLE keyword will be removed by workbench.
Here is a link from mysql bugs
https://bugs.mysql.com/bug.php?id=92269
I am using MySQL Workbench and have same problem. Change in the Preferences but it did not work.
Solution: If you export forward-engineer the model you need to change the configuration on another place.
Go to Model > Model Options
Inside the Model Options, go to MySQL
Then change the "Target MySQL Version" to 5.6
This is making me kind of crazy: I did a mysqldump of a partitioned table on one server, moved the resulting SQL dump to another server, and attempted to run the insert. It fails, but I'm having difficulty figuring out why. Google and the MySQL forums and docs have not been much help.
The failing query looks like this (truncated for brevity and clarity, names changed to protect the innocent):
CREATE TABLE `my_precious_table` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`somedata` varchar(20) NOT NULL,
`aTimeStamp` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`,`aTimeStamp`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 DATA DIRECTORY='/opt/data/data2/data_foo/' INDEX DIRECTORY='/opt/data/data2/idx_foo/'
/*!50100 PARTITION BY RANGE (year(aTimeStamp)) SUBPARTITION BY HASH ( TO_DAYS(aTimeStamp))
(PARTITION p0 VALUES LESS THAN (2007) (SUBPARTITION foo0 DATA DIRECTORY = '/opt/data/data2/data_foo' INDEX DIRECTORY = '/opt/data/data2/idx_foo' ENGINE = MyISAM),
PARTITION p1 VALUES LESS THAN (2008) (SUBPARTITION foo1 DATA DIRECTORY = '/opt/data/data2/data_foo' INDEX DIRECTORY = '/opt/data/data2/idx_foo' ENGINE = MyISAM),
PARTITION p2 VALUES LESS THAN (2009) (SUBPARTITION foo2 DATA DIRECTORY = '/opt/data/data2/data_foo' INDEX DIRECTORY = '/opt/data/data2/idx_foo' ENGINE = MyISAM),
PARTITION p3 VALUES LESS THAN MAXVALUE (SUBPARTITION foo3 DATA DIRECTORY = '/opt/data/data2/data_foo' INDEX DIRECTORY = '/opt/data/data2/idx_foo' ENGINE = MyISAM)) */;
The error is:
ERROR 1 (HY000): Can't create/write to file '/opt/data/data2/idx_foo/my_precious_table#P#p0#SP#foo0.MYI' (Errcode: 13)
"Can't create/write to file" looked like a permissions issue to me, but permissions on the targeted folders look thus:
drwxrwxrwx 2 mysql mysql 4096 Dec 1 16:24 data_foo
drwxrwxrwx 2 mysql mysql 4096 Dec 1 16:25 idx_foo
For kicks, I've tried chowning to root:root and myself. This did not fix the issue.
Source MySQL server is version 5.1.22-rc-log. Destination server is 5.1.29-rc-community. Both are running on recent CentOS installations.
Edit: A little more research shows that Errcode 13 is, in fact, a permissions error. But how can I get that on rwxrwxrwx?
Edit: Bill Karwin's excellent suggestion didn't pan out. I'm working as the root user, and have all privilege flags set.
Edit: Creating the table WITHOUT specifying data directories for the individual partitions works - but I need to put these partitions on a larger disk than the one on which this MySQL instance puts tables by default. And I can't just specify the DATA/INDEX DIRECTORY at the table level - that's not legit in the version of MySQL I'm using (5.1.29-rc-community).
Edit: Finally came across the answer, thanks to the MySQL mailing list and internal IT staff. See below.
On Ubuntu look into the apparmor settings for mysql
vi /etc/apparmor.d/usr.sbin.mysql
This should solve the permission issues. For a quick test you can even try
/etc/init.d/apparmor stop
But don't forget to restart the service.
This took me some time to figure out. And after reading "SELinux" it was clear that I have forgotten this new kind of protection on Ubuntu.
http://bugs.mysql.com/bug.php?id=19557
You will also receive an error message
of the MySQL user ID running the query
does not have "DATA FILE" privileges
that allows the user ID to write to
the file system.
In other words, it can be a permission problem with respect to SQL privileges, not operating system file permissions.
It turned out to be an SElinux issue - all my filesystem permissions were fine, but there was a higher-level policy set against MySQL accessing that disk partition.
Lesson: When you have a permissions issue but ownership and filesystem permissions are obviously correct, look to SElinux.