I'm using PostgreSQL 9.5 and flyway 5.0.7
Everything worked fine for the previous 6 migrations but now it blocks for the latest
I have the following error :
22:27:45.230 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Flyway Community Edition 5.0.7 by Boxfuse
22:27:45.408 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Database: jdbc:postgresql://localhost:32767/my_db (PostgreSQL 9.5)
22:27:45.566 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Successfully validated 7 migrations (execution time 00:00.061s)
22:27:45.658 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Current version of schema "public": 6
22:27:45.733 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Migrating schema "public" to version 7 - update
Exception in thread "main" org.flywaydb.core.internal.exception.FlywaySqlException:
Unable to insert row for version '7' in Schema History table "public"."flyway_schema_history"
---------------------------------------------------------------------------------------------
SQL State : 23502
Error Code : 0
Message : ERROR: null value in column "version_rank" violates not-null constraint
Détail : Failing row contains (null, 7, 7, update, SQL, V7__update.sql, -1303600795, postgres, 2018-02-25 22:28:00.536556, 158, t).
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.doAddAppliedMigration(JdbcTableSchemaHistory.java:171)
at org.flywaydb.core.internal.schemahistory.SchemaHistory.addAppliedMigration(SchemaHistory.java:146)
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:378)
at org.flywaydb.core.internal.command.DbMigrate.access$400(DbMigrate.java:52)
at org.flywaydb.core.internal.command.DbMigrate$5.call(DbMigrate.java:297)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:75)
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:294)
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:259)
at org.flywaydb.core.internal.command.DbMigrate.access$300(DbMigrate.java:52)
at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:179)
at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:176)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:71)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:110)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:148)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:176)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:145)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1206)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1168)
at org.flywaydb.core.Flyway.execute(Flyway.java:1655)
at org.flywaydb.core.Flyway.migrate(Flyway.java:1168)
at com.test.MyApplication.main(MainApplication.java:47)
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "version_rank" violates not-null constraint
Détail : Failing row contains (null, 7, 7, update, SQL, V7__update.sql, -1303600795, postgres, 2018-02-25 22:28:00.536556, 158, t).
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2167)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132)
at org.flywaydb.core.internal.util.jdbc.JdbcTemplate.update(JdbcTemplate.java:334)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.doAddAppliedMigration(JdbcTableSchemaHistory.java:165)
... 20 more
Any idea why this column "version_rank" is not generated or not initialized ?
Thanks in advance for your help
You upgraded from Flyway 3.x to 5.x, skipping 4.x. This is not possible as written in the release notes: https://flywaydb.org/documentation/releaseNotes#5.0.0
Upgrade to 4.2.0 first before upgrading to 5.x and everything will work as expected.
Also please take a minute to check the release notes next time you upgrade a major version.
Here's the commit with the metadata table changes if you need to apply the changes manually. I've linked to the version for postgres - search for the version of upgradeMetaDataTable.sql in the folder matching the required dialect.
Fortunately you can apply the changes as a standard flyway migration, as the metadata tables are not used until the end of each script.
E.G. create a migration V999.00__FlywayFix.sql to correct a flyway version table called flyway_table as follows:
DROP INDEX "flyway_table_vr_idx";
DROP INDEX "flyway_table_ir_idx";
ALTER TABLE "flyway_table" DROP COLUMN "version_rank";
ALTER TABLE "flyway_table" DROP CONSTRAINT "flyway_table_pk";
ALTER TABLE "flyway_table" ALTER COLUMN "version" DROP NOT NULL;
ALTER TABLE "flyway_table" ADD CONSTRAINT "flyway_table_pk" PRIMARY KEY ("installed_rank");
UPDATE "flyway_table" SET "type"='BASELINE' WHERE "type"='INIT';
it's worked for me for Postgres
CREATE TABLE flyway_schema_history (
installed_rank INTEGER NOT NULL,
version varchar(50) DEFAULT NULL,
description varchar(200) NOT NULL,
type varchar(20) NOT NULL,
script varchar(1000) NOT NULL,
checksum INTEGER DEFAULT NULL,
installed_by varchar(100) NOT NULL,
installed_on timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
execution_time INTEGER NOT NULL,
success BOOLEAN NOT NULL,
PRIMARY KEY (installed_rank)
);
INSERT INTO flyway_schema_history (installed_rank, version, description, type, script, checksum, installed_by, installed_on, execution_time, success)
SELECT installed_rank, version, description, type, script, checksum, installed_by, installed_on, execution_time, success
FROM schema_version;
ALTER TABLE schema_version RENAME TO bak_schema_version;
Related
I'm using / setting up the Symfony DoctrineMigrationsBundle v2.2 configured as followed:
doctrine_migrations:
name: 'My Migrations'
migrations_paths:
'DoctrineMigrations': '%kernel.project_dir%/src/Migrations'
storage:
table_storage:
table_name: 'migrations'
version_column_name: 'version'
version_column_length: 1024
executed_at_column_name: 'executed_at'
# Seems not to be supported:
# Unrecognized option "execution_time_column_name" under "doctrine_migrations.storage.table_storage"
# execution_time_column_name: 'execution_time'
organize_migrations: false
# custom_template: ~
all_or_nothing: false
The RDBMS is MySQL v8, running (locally) on Ubuntu Desktop v20.04:
$ mysql --version
mysql Ver 8.0.22-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))
The DEFAULT_CHARACTER_SET_NAME is utf8mb4, the DEFAULT_COLLATION_NAME is utf8mb4_unicode_ci:
SELECT
`DEFAULT_CHARACTER_SET_NAME`, `DEFAULT_COLLATION_NAME`
FROM
`INFORMATION_SCHEMA`.`SCHEMATA`
WHERE
`SCHEMA_NAME` = "payment"
;
+----------------------------+------------------------+
| DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME |
+----------------------------+------------------------+
| utf8mb4 | utf8mb4_unicode_ci |
+----------------------------+------------------------+
While I was playing with the configs, I created two migrations tables (I changed the doctrine_migrations.storage.table_storage.table_name multiple times), somehow... Now, after the configuration has bee completed I want to go the setup through cleanly again from scratch. So I removed both migrations tables and started again. But now I'm getting following error:
$ ./bin/console doctrine:migrations:status
...
In AbstractMySQLDriver.php line 106:
An exception occurred while executing 'CREATE TABLE migrations (version VARCHAR(1024) NOT NULL, executed_at DATETIME NOT NULL COMMENT '(DC2Type:datetime_immutable)', PRIMARY KEY(version)) DEFAULT CHARACTER
SET utf8mb4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB':
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
In PDOConnection.php line 43:
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
In PDOConnection.php line 41:
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
I tried to reduce the doctrine_migrations.storage.table_storage.version_column_length, but then I'm running in another error:
$ ./bin/console doctrine:migrations:status
...
In BaseNode.php line 348:
Invalid configuration for path "doctrine_migrations.storage.table_storage.version_column_length": The minimum length for the version column is 1024.
In ExprBuilder.php line 189:
The minimum length for the version column is 1024.
How to set this up correctly and get the migrations tables generated?
It's not a (clean) solution, but at least a workaround:
Removing the configuration doctrine_migrations.storage.table_storage.version_column_length makes it work again. The resulting configuration looks then like this:
$ bin/console debug:config doctrine_migrations
...
Current configuration for extension with alias "doctrine_migrations"
====================================================================
doctrine_migrations:
name: 'My Migrations'
migrations_paths:
DoctrineMigrations: /var/www/html/src/Migrations
storage:
table_storage:
table_name: migrations
version_column_name: version
executed_at_column_name: executed_at
version_column_length: null
organize_migrations: false
all_or_nothing: false
dir_name: /var/www/html/src/bundles/Wings/DoctrineMigrations
namespace: Application\Migrations
table_name: migration_versions
column_name: version
column_length: 14
executed_at_column_name: executed_at
custom_template: null
I tried to set up a database with a table with timestamp column . I am trying to implement timestamp mode to capture incremental changes in the DB.
But kafka-connect-jdbc is not reading any data from the table. Here is what I have done.
Created a table.
sqlite> CREATE TABLE test_timestamp(id integer primary key not null,
...> payment_type text not null,
...> Timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
...> user_id int not null);
sqlite> INSERT INTO test_timestamp (ID, PAYMENT_TYPE, USER_ID) VALUES (3,'FOO',1);
sqlite> select * from test_timestamp;
3|FOO|2019-06-18 05:31:22|1
My jdbc-source connector configuration is as follows:
$ curl -s "http://localhost:8083/connectors/jdbc-source/config"|jq '.'
{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"mode": "timestamp",
"timestamp.column.name": "timestamp",
"topic.prefix": "testdb-",
"validate.non.null": "false",
"tasks.max": "1",
"name": "jdbc-source",
"connection.url": "jdbc:sqlite:/tmp/test.db"
}
jdbc-source-connector successfully loads and topic is created
$ kafka-topics --list --bootstrap-server localhost:9092
..
testdb-test_timestamp
But no data appears in the topic.
Any help ?
Thanks in advance.
Stumbled across similar issue. In my case, even the topic didn't create. Found out that timezone for connect worker must be same as that of DB. Setting db.timezone property in the connect worker's property file correctly based on this list (under SHORT_IDS section) makes it work:
db.timezone=Asia/Kolkata
You are hitting a known issue, detailed here: https://github.com/confluentinc/kafka-connect-jdbc/issues/219
Steps to reproduce:
Create database:
$ echo 'DROP TABLE test_timestamp; CREATE TABLE test_timestamp(id integer primary key not null,
payment_type text not null,
Timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
user_id int not null);
INSERT INTO test_timestamp (ID, PAYMENT_TYPE, USER_ID) VALUES (3,\'FOO\',1);
select * from test_timestamp;' | sqlite3 /tmp/test.db
3|FOO|2019-07-03 08:28:43|1
Create connector
curl -X PUT "http://localhost:8083/connectors/jdbc-source/config" -H "Content-Type:application/json" -d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"mode": "timestamp",
"timestamp.column.name": "timestamp",
"topic.prefix": "testdb-",
"validate.non.null": "false",
"tasks.max": "1",
"name": "jdbc-source",
"connection.url": "jdbc:sqlite:/tmp/test.db"
}'
Check status of connector
$ curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
xargs -I{connector_name} curl -s "http://localhost:8083/connectors/"{connector_name}"/status"| \
jq -c -M '[.name,.connector.state,.tasks[].state]|join(":|:")'| \
column -s : -t| sed 's/\"//g'| sort
jdbc-source | RUNNING | RUNNING
Check Kafka Connect worker log, observe error parsing date, per issue 219
[2019-07-03 10:40:58,260] ERROR Failed to run query for table TimestampIncrementingTableQuerier{table="test_timestamp", query='null', topicPrefix='testdb-', incrementingColumn='', timestampColumns=[timestamp]}: {} (io.confluent.connect.jdbc.source.JdbcSourceTask:332)
java.sql.SQLException: Error parsing time stamp
at org.sqlite.jdbc3.JDBC3ResultSet.getTimestamp(JDBC3ResultSet.java:576)
at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.currentTimeOnDB(GenericDatabaseDialect.java:484)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.endTimetampValue(TimestampIncrementingTableQuerier.java:203)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.setQueryParametersTimestamp(TimestampIncrementingCriteria.java:164)
at io.confluent.connect.jdbc.source.TimestampIncrementingCriteria.setQueryParameters(TimestampIncrementingCriteria.java:126)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.executeQuery(TimestampIncrementingTableQuerier.java:176)
at io.confluent.connect.jdbc.source.TableQuerier.maybeStartQuery(TableQuerier.java:92)
at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.maybeStartQuery(TimestampIncrementingTableQuerier.java:60)
at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:310)
at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:245)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:221)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.text.ParseException: Unparseable date: "2019-07-03 08:40:58" does not match (\p{Nd}++)\Q-\E(\p{Nd}++)\Q-\E(\p{Nd}++)\Q \E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q:\E(\p{Nd}++)\Q.\E(\p{Nd}++)
at org.sqlite.date.FastDateParser.parse(FastDateParser.java:299)
at org.sqlite.date.FastDateFormat.parse(FastDateFormat.java:490)
at org.sqlite.jdbc3.JDBC3ResultSet.getTimestamp(JDBC3ResultSet.java:573)
... 17 more
I'm working with Sylius framework. I'm following the guide to customize models.
I am trying to add a field notice to a model Taxon which is already overridden in my project. For that, I added the field description to Taxon.orm.yml of the model:
MyProject\Bundle\ShopBundle\Entity\Taxon:
type: entity
table: sylius_taxon
# {Relationships code...}
fields:
# {Some existing fields...}
notice:
type: text
nullable: true
I also added a field, a getter and a setter to the overriding Taxon class.
Then I'm trying to run bin/console doctrine:migrations:diff, but when I run bin/console even without any arguments, it crashes with the following exception:
[Doctrine\DBAL\Exception\InvalidFieldNameException]
An exception occurred while executing 'SELECT s0_.code AS code_0, s0_.tree_left AS tree_left_1, s0_.tree_right AS tree_right_2, s0_.tree_level AS tree_level_3, s0_.position AS position_4, s0_.id AS id_5, s0_
.created_at AS created_at_6, s0_.updated_at AS updated_at_7, s0_.enabled AS enabled_8, s0_.default_markup AS default_markup_9, s0_.notice AS notice_10, s0_.tree_root AS tree_root_11, s0_.parent_id AS parent_
id_12 FROM sylius_taxon s0_ WHERE s0_.parent_id IS NULL ORDER BY s0_.tree_left ASC':
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'`
[Doctrine\DBAL\Driver\PDOException]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'`
[PDOException]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'
If I remove the changes to Taxon.orm.yml then bin/console works again. What is missing in my changes?
One of my bundles' configruation contained that model's repository, that's it. I temporarily deleted the bundle's configuration from config.yml, and bin/console worked.
When you add new field you should doctrine:schema:update
I installed the DB with updates and I got this error:
Opening DatabasePool 'wotlk_world'. Asynchronous connections: 1, synchronous connections: 1. MySQL client library: 5.6.42 MySQL server ver: 5.6.42 MySQL client library: 5.6.42 MySQL server ver: 5.6.42 [ERROR]: In mysql_stmt_prepare() id: 4, sql: "SELECT entryorguid, source_type, id, link, event_type, event_phase_mask, event_chance, event_flags, event_param1, event_param2, event_param3, event_param4, event_param5, action_type, action_param1, action_param2, action_param3, action_param4, action_param5, action_param6, target_type, target_param1, target_param2, target_param3, target_param4, target_x, target_y, target_z, target_o FROM smart_scripts ORDER BY entryorguid, source_type, id, link" [ERROR]: Unknown column 'event_param5' in 'field list' [ERROR]: In mysql_stmt_prepare() id: 54, sql: "SELECT difficulty_entry_1, difficulty_entry_2, difficulty_entry_3, KillCredit1, KillCredit2, modelid1, modelid2, modelid3, modelid4, name, subname, IconName, gossip_menu_id, minlevel, maxlevel, exp, faction, npcflag, speed_walk, speed_run, scale, rank, mindmg, maxdmg, dmgschool, attackpower, DamageModifier, BaseAttackTime, RangeAttackTime, unit_class, unit_flags, unit_flags2, dynamicflags, family, trainer_type, trainer_spell, trainer_class, trainer_race, minrangedmg, maxrangedmg, rangedattackpower, type, type_flags, lootid, pickpocketloot, skinloot, resistance1, resistance2, resistance3, resistance4, resistance5, resistance6, spell1, spell2, spell3, spell4, spell5, spell6, spell7, spell8, PetSpellDataId, VehicleId, mingold, maxgold, AIName, MovementType, InhabitType, HoverHeight, HealthModifier, ManaModifier, ArmorModifier, RacialLeader, movementId, RegenHealth, mechanic_immune_mask, flags_extra, ScriptName FROM creature_template WHERE entry = ?" [ERROR]: Unknown column 'DamageModifier' in 'field list' DatabasePool wotlk_world NOT opened. There were errors opening the MySQL connections. Check your SQLDriverLogFile for specific errors. Cannot connect to world database
127.0.0.1;3306;root;ascent;wotlk_world
The problem is:
[ERROR]: Unknown column 'DamageModifier' in 'field list'
It looks like your world DB is not up to date. So you need to update it properly. To do that you can either use the DB assembler script (bin/acore-db-asm) or manually importing the missing sql files from data/sql/updates/db_world.
To make sure your DB is up to date, check the name of the last column of the table version_db_world of your world database. It should match with the most recent sql file name of the direcotry data/sql/updates/db_world.
I recommend reading:
How to make sure that the DB is up to date
I' following a symfony2 tutorial and I've problems with one step.
Tutorial link: http://intelligentbee.com/blog/2013/08/12/symfony2-jobeet-day-6-more-with-the-model/
I'm in 'Refactoring' step. I've one this 3 steps:
1- I've correctly modified /src/Ibw/JobeetBundle/Resources/config/doctrine/Job.orm.yml file specifiying the repository
2- I've run the command: php app/console doctrine:generate:entities IbwJobeetBundle
3- And I've added the specified tutorial function por JobRepository.php
$qb = $this->createQueryBuilder('j')
->where('j.expires_at > :date')
->setParameter('date', date('Y-m-d H:i:s', time()))
->orderBy('j.expires_at', 'DESC');
BUT when I refresh my code I get this error:
An exception occurred while executing 'SELECT j0_.id AS id0, j0_.type
AS type1, j0_.company AS company2, j0_.logo AS logo3, j0_.url AS url4,
j0_.position AS position5, j0_.location AS location6, j0_.description
AS description7, j0_.how_to_apply AS how_to_apply8, j0_.token AS
token9, j0_.is_public AS is_public10, j0_.is_activated AS
is_activated11, j0_.email AS email12, j0_.expires_at AS expires_at13,
j0_.created_at AS created_at14, j0_.updated_at AS updated_at15,
j0_.category_id AS category_id16 FROM Job j0_ WHERE j0_.expires_at > ?
ORDER BY j0_.expires_at DESC' with params ["2016-03-17 15:47:19"]:
SQLSTATE[42S02]: Base table or view not found: 1146 Table 'jobeet.Job'
doesn't exist
In the symfony profiler I can see the full query:
SELECT j0_.id AS id0, j0_.type AS type1, j0_.company AS company2,
j0_.logo AS logo3, j0_.url AS url4, j0_.position AS position5,
j0_.location AS location6, j0_.description AS description7,
j0_.how_to_apply AS how_to_apply8, j0_.token AS token9, j0_.is_public
AS is_public10, j0_.is_activated AS is_activated11, j0_.email AS
email12, j0_.expires_at AS expires_at13, j0_.created_at AS
created_at14, j0_.updated_at AS updated_at15, j0_.category_id AS
category_id16 FROM Job j0_ WHERE j0_.expires_at > '2016-03-17
15:47:19' ORDER BY j0_.expires_at DESC
the table name it's in uppercase! 'Job', not 'job'
Any can help me, please?
You can try by modifying the entity definition in Job.orm.yml by adding the table attribute:
IbwJobeetBundleEntityJob:
type: entity
table: job
repositoryClass: IbwJobeetBundleRepositoryJobRepository