We created a new database by coping from an older flyway managed database. (flyway version table also copied over)
Running flyway against this new database produces the following flyway error
Apr 24 19:00:19 ip-xx flyway: Unable to insert row for version '184' in Schema History table "public"."flyway_schema_history"
Apr 24 19:00:19 ip--xx flyway: -----------------------------------------------------------------------------------------------
Apr 24 19:00:19 ip--xx flyway: SQL State : 23502
Apr 24 19:00:19 ip--xx flyway: Error Code : 0
Apr 24 19:00:19 ip--xx flyway: Message : ERROR: null value in column "installed_on" violates not-null constraint
In my case, the error occurred while upgrading from flyway version 5.x to 6.4.3.
The definition of installed_on column of the flyway_schema_history table (at least in PostgreSQL) has changed from:
installed_on timestamp NOT NULL;
to:
installed_on timestamp NOT NULL DEFAULT now ()
Adding DEFAULT now() on the existing table has fixed the problem.
Which version of Flyway are you using, and has it changed when you moved database? Which database are you using?
If you could create an issue at github.com/flyway/flyway/issues with reproduction steps then the team will investigate for you.
Related
What is the right way to backup and restore a MariaDB database that has sequence generation enabled (i.e. NOT autoincrement)? (This includes migrating to a new server.)
Is it possible to instruct the sequence generator to pick up indexing table data at a specific ID value? How?
Steps I take to create my issue
I wish to transfer an application to a new server:
Backup data on source server:
mysqldump --skip-opt --no-create-db --no-create-info --hex-blob [database-name] [...list of tables...] > data-backup.sql
On target server, create new empty database (same name)
Build/run JHipster Spring application on target server: java -jar myapp.jar (Running this application recreates/configures a new instance of the database on the target server.)
Restore data:
mysql [database-name] < data-backup.sql
All the above steps produce no errors (so far).
Problem
When I follow these steps, the database is restored (apparently perfectly). I can log in to the application and access all information. BUT when I attempt to create new entities (i.e. save something to the database), I get an ID 'Duplicate entry' error in the server logs:
2022-03-24 12:54:43.775 ERROR 11277 --- [ XNIO-1 task-1] o.h.e.jdbc.batch.internal.BatchingBatch : HHH000315: Exception executing batch [java.sql.BatchUpdateException: (conn=33) Duplicate entry '1001' for key 'PRIMARY'], SQL: insert into product (name, id) values (?, ?)
2022-03-24 12:54:43.776 WARN 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 1062, SQLState: 23000
2022-03-24 12:54:43.776 ERROR 11277 --- [ XNIO-1 task-1] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=33) Duplicate entry '1001' for key 'PRIMARY'
2022-03-24 12:54:43.779 ERROR 11277 --- [ XNIO-1 task-1] o.z.problem.spring.common.AdviceTraits : Internal Server Error
org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [insert into product (name, id) values (?, ?)]; constraint [PRIMARY]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:276)
at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:566)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at com.mycompany.app.web.rest.ProductResource$$EnhancerBySpringCGLIB$$84c14d6d.createProduct(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
Clearly my backup/restore process is not accounting properly for the sequence generator, which generates ID values that conflict with the existing data.
What I am doing wrong? What is the right process of backing up/restoring such a database?
Environment: JHipster 7.7.0 (Angular, monolithic), MariaDB 10.4, OpenJDK 16.0.2_7, OS Windows 10 Pro and openSUSE 15.2, Firefox 98.0.2 and Chrome 99.0.4844.84.
PS: I previously reported this issue here, aimed at the JHipster community, but got limited response. I think I need a MySQL/MariaDB expert opinion on this.
(Apologies in advance: I'm not a database expert. The technique I outline above has served me well for years, but previously I was dealing with AUTO_INCREMENT. This sequence generator has me baffled.)
Ok! I have solutions.
[For the sake if these notes, let's call the database: mydata. Also, in JHipster, the MariaDB sequence generator is called: sequence_generator]
Let's consider two situations:
(1) Simple migration
If you are merely migrating the application to a new server, the process is straight forward:
Step 1: On the original server backup and secure your database: mysqldump -u root -p mydata > mydata.sql
Step 2: Transfer the SQL file to the new server, along with the JHipster JAR file
Step 3: On the new server, create an empty database with the same name, and restore the data: mysql -u root -p mydata < mydata.sql
Step 4: Now launch your JHipster application, and everything should work
(2) Model modification
The assumption is that you have modified your model in some way (e.g. added properties to one or more entities). This solution is fiddly, but it works (for me).
Step 1: Backup your database, and secure it (in case something goes wrong): mysqldump -u root -p mydata > mydata.sql
Step 2: Backup and secure the original JHipster JAR that works with the original database
Step 3: Duplicate your database (schema and data) in a new table: mydata_bk
Step 4: Drop your original database, and create a new empty database
Step 5: Launch your new JHipster JAR, and give it time to create the new database schema, then stop the application
Step 6: Use a tool (DataGrip, sqlYog, etc) to compare the old (mydata_bk) and new schema (mydata), and modify the old schema to match the new schema
Step 7: Restore/copy all data from mydata_bk to mydata, EXCEPT for the tables DATABASECHANGELOG, DATABASECHANGELOGLOCK and the special sequence_generator table
Step 8: Open the mydata.sql SQL file, and at the top, after initial comments, one of the first instructions will read:
--
-- Sequence structure for `sequence_generator`
--
DROP SEQUENCE IF EXISTS `sequence_generator`;
CREATE SEQUENCE `sequence_generator` start with 2000 minvalue 1 maxvalue 9223372036854775806 increment by 50 cache 1000 nocycle ENGINE=InnoDB;
SELECT SETVAL(`sequence_generator`, 201050, 0);
The specific numbers may vary, but the broad details will be similar. In a MariaDB SQL console type/execute each of those SQL statements: DROP SEQUENCE ...;, CREATE SEQUENCE ...;, and SELECT SETVAL(...);
Step 9: Launch your JHipster application.
Hope this helps others that run into similar issues. Let me know if you have a better approach!
I just discovered the silly new issue of MariaDB's latest version having mysql.user as a view. All my imported Wordpress databases suddenly cannot connect from the blogs. When I try to even list mysql.user it shows me this:
> select * from mysql.user;
ERROR 1356 (HY000): View 'mysql.user' references invalid table(s)
or column(s) or function(s) or definer/invoker of view lack
rights to use them
What can we do to solve this?
Edit: Found this question, but it does not have a solution, only a suggestion. The ALTER USER command -- where to use and with what settings? Do I have to somehow alter the rights for every Blog database?
Update:
Further investigation revealed that the issue described in this Question, and my initial response to it (below) may be related to an Incorrect definition of table mysql.event problem. In my case, I had 1) loaded a full dump (including the mysql database) from MySQL 5.7.33 to a fresh installation of MariaDB10.5.9; 2) discovered that this was not a good idea; 3) edited my dump file to exclude the mysql database, and 4) repeated the load without deleting any databases or configurations.
This caused the database to function properly, but (in addition to the issue described in this Question) a) /usr/sbin/mariadbd --verbose --help would try to run the database server rather than print help, b) on startup the following error always occurred:
Apr 05 08:52:46 xxx mariadbd[22668]: 2021-04-05 8:52:46 0 [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPTY_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','NOT_USED','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_B
Apr 05 08:52:46 xxx mariadbd[22668]: 2021-04-05 8:52:46 0 [ERROR] mariadbd: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler
Today, I was able to correct these problems (under Amazon Linux 2) by:
Uninstalling MariaDB-server and MariaDB-client
Removing /etc/my.*
Removing /var/lib/mysql
Reinstalling MariaDB-server and MariaDB-client
Reloading the database dump, again omitting the dump of the mysql database
At this point, I not only have clean database startup and proper operation of /usr/sbin/mariadbd --verbose --help, I also find that select * from mysql.user works properly!
So the problem of not being able to select from mysql.user appears not to have resulted from the change of mysql.user from table to view as I had originally thought, but from some other issue related to my "improper" database migration.
My initial answer:
(included as a reference only)
After considerable research I have found at least part of the answer to this question:
tl;dr: select * from mysql.global_priv
then for each User,
show grants for 'XXX'#'localhost';
Longer version, from Authentication in MariaDB 10.4 — Understanding the Changes:
The password storage has changed. All user accounts, passwords, and global privileges are now stored in a mysql.global_priv table. What happened to the mysql.user table? It still exists and has exactly the same set of columns as before, but it’s now a view over mysql.global_priv...."
The aforementioned article provides not only what what but also the why. I do not agree with all of it. In particular the claim is made that Old mysql.user table still exists, you can select from it as before, but you cannot (hence this question). Nonetheless I am relieved to discover a relatively coherent explanation from MariaDB.
Finally, here is an example:
MariaDB [(none)]> select * from mysql.global_priv\G
*************************** 1. row ***************************
Host: localhost
User: mariadb.sys
Priv: {"access":0,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}
*************************** 2. row ***************************
Host: localhost
User: root
Priv: {"access": 1844674407370915, "plugin": "mysql_native_password", "authentication_string": "*9A87226E872127C756290C504DB5D9076E", "auth_or": [{}, {"plugin": "unix_socket"}], "password_last_changed": 1617303275}
*************************** 3. row ***************************
Host: localhost
User: mysql
Priv: {"access":1844674407371615,"plugin":"mysql_native_password","authentication_string":"invalid","auth_or":[{},{"plugin":"unix_socket"}]}
*************************** 4. row ***************************
MariaDB [(none)]> show grants for 'root'#'localhost'\G
*************************** 1. row ***************************
Grants for root#localhost: GRANT ALL PRIVILEGES ON *.* TO `root`#`localhost` IDENTIFIED VIA mysql_native_password USING '*9A87226E872127C756290C5BF177504DB5D9076E' OR unix_socket WITH GRANT OPTION
*************************** 2. row ***************************
Grants for root#localhost: GRANT PROXY ON ''#'%' TO 'root'#'localhost' WITH GRANT OPTION
when trying to migrate the following files,
drop table test2;
CREATE TABLE test2
(id INTEGER PRIMARY KEY AUTOINCREMENT,
name VARCHAR(30),
message VARCHAR(200),
timestamp TIMESTAMP(7));
I got the following result:
2019-12-07 10:54:33,129 [main] INFO migratus.core - Starting migrations
2019-12-07 10:54:33,253 [main] DEBUG migratus.migrations - Looking for migrations in #object[java.io.File 0x4f880f4a /home/jonas/Dropbox/prog/web/clojure/test2/resources/migrations]
2019-12-07 10:54:33,258 [main] INFO migratus.core - Ending migrations
It seems that it doesn't find the connection and therefore no database is created. Could that be the case?
found the solution, set the DATABASE_URL to the corresponding database:
export DATABASE_URL="jdbc:sqlite:./test2.db"
where test2 is the name of the app and sqlite is the name of the database
I have to move Ubuntu mariadb(10.1.41) data(78GB) to CentOS mariadb(5.5.64).
I used to mysqldump and export on heidiSQL, But every try is failed.
I think problem is that column 's default is current_timestamp.
So, I did test.
here is test query:
CREATAE TABLE test (
ts INT(11),
ts2 DATETIME DEFAULT CURRENT_TIMESTAMP
);
Result is
"SQL ERROR(1067) : Invalid default value for 'test'"
So, I tried : edit configure /etc/my.cnf
sql-mode="ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
and
sql-mode=""
and
sql-mode="NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
But, Everything is Failed. what should I do??
In MariaDB 5.5 CURRENT_TIMESTAMP is not supported as default for column type DATETIME.
Workarounds:
Update MariaDB Server to 10.x
change column type from DATETIME to
TIMESTAMP
After I upgraded my MySQL version from 5.6 to 5.7, the AzerothCore installer fails with errors like:
ERROR 1067 (42000) at line 234: Invalid default value for 'last_login'
ERROR 1292 (22007) at line 266: Incorrect datetime value: '0000-00-00 00:00:00' for column 'last_login' at row 1
and:
Can't find any matching row in the user table
In fact, you better do it like this (to avoid removing default modes):
-- Select the sql modes
SELECT ##sql_mode;
-- Remove the 2 modes NO_ZERO_IN_DATE and NO_ZERO_DATE and run this query
SET sql_mode = 'mode_1,mode_2,mode_3,mode_4,mode_5';
Also this can be put directly in mysql configuration.
After [mysqld]:
[mysqld]
sql_mode = mode_1,mode_2,mode_3,mode_4,mode_5
Note: ONLY_FULL_GROUP_BY can be kept
EDIT: As of march 2019, it's probably not needed at all, we have fixed this but it might occur again in the future
EDIT: this answer is a bit outdated, some of the issues with MySQL 5.7 have been resolved in the latest version
As stated in the requirements page of the AC wiki:
AzerothCore does not officially support MySQL version >= 5.7, but there is a way to get it up and running.
You have to remove ONLY_FULL_GROUP_BY,NO_ZERO_IN_DATE and NO_ZERO_DATE flags from MySQL's sql_mode variable in the MySQL config file so that all queries updates and core statements can be applied correctly.
I've found it useful to set my SQL mode to the empty string (which in my case solved all the issues with MySQL 5.7):
SET GLOBAL sql_mode = ''
you can run that sql statement via terminal using:
mysql -e "SET GLOBAL sql_mode = '';"
or alternatively you can manually set the SQL mode before running any SQL statements:
SET sql_mode = '';