How to fix the "key too long" error and generate the Doctrine migrations table? - symfony

I'm using / setting up the Symfony DoctrineMigrationsBundle v2.2 configured as followed:
doctrine_migrations:
name: 'My Migrations'
migrations_paths:
'DoctrineMigrations': '%kernel.project_dir%/src/Migrations'
storage:
table_storage:
table_name: 'migrations'
version_column_name: 'version'
version_column_length: 1024
executed_at_column_name: 'executed_at'
# Seems not to be supported:
# Unrecognized option "execution_time_column_name" under "doctrine_migrations.storage.table_storage"
# execution_time_column_name: 'execution_time'
organize_migrations: false
# custom_template: ~
all_or_nothing: false
The RDBMS is MySQL v8, running (locally) on Ubuntu Desktop v20.04:
$ mysql --version
mysql Ver 8.0.22-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))
The DEFAULT_CHARACTER_SET_NAME is utf8mb4, the DEFAULT_COLLATION_NAME is utf8mb4_unicode_ci:
SELECT
`DEFAULT_CHARACTER_SET_NAME`, `DEFAULT_COLLATION_NAME`
FROM
`INFORMATION_SCHEMA`.`SCHEMATA`
WHERE
`SCHEMA_NAME` = "payment"
;
+----------------------------+------------------------+
| DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME |
+----------------------------+------------------------+
| utf8mb4 | utf8mb4_unicode_ci |
+----------------------------+------------------------+
While I was playing with the configs, I created two migrations tables (I changed the doctrine_migrations.storage.table_storage.table_name multiple times), somehow... Now, after the configuration has bee completed I want to go the setup through cleanly again from scratch. So I removed both migrations tables and started again. But now I'm getting following error:
$ ./bin/console doctrine:migrations:status
...
In AbstractMySQLDriver.php line 106:
An exception occurred while executing 'CREATE TABLE migrations (version VARCHAR(1024) NOT NULL, executed_at DATETIME NOT NULL COMMENT '(DC2Type:datetime_immutable)', PRIMARY KEY(version)) DEFAULT CHARACTER
SET utf8mb4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB':
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
In PDOConnection.php line 43:
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
In PDOConnection.php line 41:
SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 3072 bytes
I tried to reduce the doctrine_migrations.storage.table_storage.version_column_length, but then I'm running in another error:
$ ./bin/console doctrine:migrations:status
...
In BaseNode.php line 348:
Invalid configuration for path "doctrine_migrations.storage.table_storage.version_column_length": The minimum length for the version column is 1024.
In ExprBuilder.php line 189:
The minimum length for the version column is 1024.
How to set this up correctly and get the migrations tables generated?

It's not a (clean) solution, but at least a workaround:
Removing the configuration doctrine_migrations.storage.table_storage.version_column_length makes it work again. The resulting configuration looks then like this:
$ bin/console debug:config doctrine_migrations
...
Current configuration for extension with alias "doctrine_migrations"
====================================================================
doctrine_migrations:
name: 'My Migrations'
migrations_paths:
DoctrineMigrations: /var/www/html/src/Migrations
storage:
table_storage:
table_name: migrations
version_column_name: version
executed_at_column_name: executed_at
version_column_length: null
organize_migrations: false
all_or_nothing: false
dir_name: /var/www/html/src/bundles/Wings/DoctrineMigrations
namespace: Application\Migrations
table_name: migration_versions
column_name: version
column_length: 14
executed_at_column_name: executed_at
custom_template: null

Related

Symfony 4.2 utf8 data cut off when saving to MySQL

I'm saving a heading from a CSV file to the database.
Viewed with less on Ubuntu the file starts like this:
Date,Supermarket,Speciality,Takeaway,Caf<E9>/restaurant
1/06/2019,0.039175903,-0.01496395,0.03603785,0.029072835
1/07/2019,0.039399919,-0.008250166,0.022385733,0.015478668
The heading data is ($csvHeader)
Array
(
[0] => Date
[1] => Supermarket
[2] => Speciality
[3] => Takeaway
[4] => Caf�/restaurant
)
ord(substr($csvHeader,3,1)) === 233
This is read with the following function
protected function getCsvHeaders()
{
$fh = fopen( $this->getCsvPath(), 'r+' );
$firstrow = fgetcsv( $fh );
fclose( $fh );
return $firstrow;
}
This is saved to a table DataConfiguration:
$dataConf
->setColumns(serialize($csvHeader));
which is set to utf8mb4:
show create table data_configuration;
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data_configuration | CREATE TABLE `data_configuration` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`data_set_id` int(11) NOT NULL,
`file_type_id` int(11) NOT NULL,
`columns` varchar(7500) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_idx` (`data_set_id`,`file_type_id`),
KEY `IDX_54A0B1FD70053C01` (`data_set_id`),
KEY `IDX_54A0B1FD9E2A35A8` (`file_type_id`),
CONSTRAINT `FK_54A0B1FD70053C01` FOREIGN KEY (`data_set_id`) REFERENCES `data_set` (`id`),
CONSTRAINT `FK_54A0B1FD9E2A35A8` FOREIGN KEY (`file_type_id`) REFERENCES `file_type` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=13176 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci |
Doctrine seems to be configured for utf8mb4 as well:
doctrine:
dbal:
# configure these for your database server
driver: 'pdo_mysql'
# server_version: '5.7'
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
url: '%env(resolve:DATABASE_URL)%'
options:
1001: true
However the data gets cut off at the utf8 character and subsequent unserialize fails. I can reproduce this on my Ubuntu 18/ AWS RDS environment as well as my local MacOS/Brew environment.
What other avenues can I explore to solve this problem?
You are parsing a text file with fgetcsv(). Its documentation states you can encounter issues when using single-byte encoded files:
The locale settings are taken into account by this function. If LC_CTYPE is e.g. en_US.UTF-8, files in one-byte encodings may be read wrongly by this function.
https://www.php.net/manual/en/function.fgetcsv.php
If your file contains French characters that exist beyond the basic ASCII table, you may set this variable to another value:
List installed locales : sh locale -a
en_US.utf8
fr_FR.iso885915
...
The output may vary. I cannot tell you a locale that's guaranteed to exist on your machine. You have to pick something like ISO-8859-1, Windows-1252, not UTF-8.
Before calling fgetcsv(), set the locale to something that matches the file encoding:
setlocale(LC_CTYPE, 'fr_FR.iso885915');
Call fgetcsv()
Alternatively, you can manually convert the encoding:
$row_utf8 = mb_convert_encoding($row_raw, "Windows-1252", "UTF-8");
What does file your.csv gives?

Can't add a field to a model in Symfony, bin/console crashes

I'm working with Sylius framework. I'm following the guide to customize models.
I am trying to add a field notice to a model Taxon which is already overridden in my project. For that, I added the field description to Taxon.orm.yml of the model:
MyProject\Bundle\ShopBundle\Entity\Taxon:
type: entity
table: sylius_taxon
# {Relationships code...}
fields:
# {Some existing fields...}
notice:
type: text
nullable: true
I also added a field, a getter and a setter to the overriding Taxon class.
Then I'm trying to run bin/console doctrine:migrations:diff, but when I run bin/console even without any arguments, it crashes with the following exception:
[Doctrine\DBAL\Exception\InvalidFieldNameException]
An exception occurred while executing 'SELECT s0_.code AS code_0, s0_.tree_left AS tree_left_1, s0_.tree_right AS tree_right_2, s0_.tree_level AS tree_level_3, s0_.position AS position_4, s0_.id AS id_5, s0_
.created_at AS created_at_6, s0_.updated_at AS updated_at_7, s0_.enabled AS enabled_8, s0_.default_markup AS default_markup_9, s0_.notice AS notice_10, s0_.tree_root AS tree_root_11, s0_.parent_id AS parent_
id_12 FROM sylius_taxon s0_ WHERE s0_.parent_id IS NULL ORDER BY s0_.tree_left ASC':
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'`
[Doctrine\DBAL\Driver\PDOException]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'`
[PDOException]
SQLSTATE[42S22]: Column not found: 1054 Unknown column 's0_.notice' in 'field list'
If I remove the changes to Taxon.orm.yml then bin/console works again. What is missing in my changes?
One of my bundles' configruation contained that model's repository, that's it. I temporarily deleted the bundle's configuration from config.yml, and bin/console worked.
When you add new field you should doctrine:schema:update

[ERROR]: Unknown column 'DamageModifier' in 'field list'

I installed the DB with updates and I got this error:
Opening DatabasePool 'wotlk_world'. Asynchronous connections: 1, synchronous connections: 1. MySQL client library: 5.6.42 MySQL server ver: 5.6.42 MySQL client library: 5.6.42 MySQL server ver: 5.6.42 [ERROR]: In mysql_stmt_prepare() id: 4, sql: "SELECT entryorguid, source_type, id, link, event_type, event_phase_mask, event_chance, event_flags, event_param1, event_param2, event_param3, event_param4, event_param5, action_type, action_param1, action_param2, action_param3, action_param4, action_param5, action_param6, target_type, target_param1, target_param2, target_param3, target_param4, target_x, target_y, target_z, target_o FROM smart_scripts ORDER BY entryorguid, source_type, id, link" [ERROR]: Unknown column 'event_param5' in 'field list' [ERROR]: In mysql_stmt_prepare() id: 54, sql: "SELECT difficulty_entry_1, difficulty_entry_2, difficulty_entry_3, KillCredit1, KillCredit2, modelid1, modelid2, modelid3, modelid4, name, subname, IconName, gossip_menu_id, minlevel, maxlevel, exp, faction, npcflag, speed_walk, speed_run, scale, rank, mindmg, maxdmg, dmgschool, attackpower, DamageModifier, BaseAttackTime, RangeAttackTime, unit_class, unit_flags, unit_flags2, dynamicflags, family, trainer_type, trainer_spell, trainer_class, trainer_race, minrangedmg, maxrangedmg, rangedattackpower, type, type_flags, lootid, pickpocketloot, skinloot, resistance1, resistance2, resistance3, resistance4, resistance5, resistance6, spell1, spell2, spell3, spell4, spell5, spell6, spell7, spell8, PetSpellDataId, VehicleId, mingold, maxgold, AIName, MovementType, InhabitType, HoverHeight, HealthModifier, ManaModifier, ArmorModifier, RacialLeader, movementId, RegenHealth, mechanic_immune_mask, flags_extra, ScriptName FROM creature_template WHERE entry = ?" [ERROR]: Unknown column 'DamageModifier' in 'field list' DatabasePool wotlk_world NOT opened. There were errors opening the MySQL connections. Check your SQLDriverLogFile for specific errors. Cannot connect to world database
127.0.0.1;3306;root;ascent;wotlk_world
The problem is:
[ERROR]: Unknown column 'DamageModifier' in 'field list'
It looks like your world DB is not up to date. So you need to update it properly. To do that you can either use the DB assembler script (bin/acore-db-asm) or manually importing the missing sql files from data/sql/updates/db_world.
To make sure your DB is up to date, check the name of the last column of the table version_db_world of your world database. It should match with the most recent sql file name of the direcotry data/sql/updates/db_world.
I recommend reading:
How to make sure that the DB is up to date

Flyway migration blocked by null version_rank

I'm using PostgreSQL 9.5 and flyway 5.0.7
Everything worked fine for the previous 6 migrations but now it blocks for the latest
I have the following error :
22:27:45.230 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Flyway Community Edition 5.0.7 by Boxfuse
22:27:45.408 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Database: jdbc:postgresql://localhost:32767/my_db (PostgreSQL 9.5)
22:27:45.566 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Successfully validated 7 migrations (execution time 00:00.061s)
22:27:45.658 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Current version of schema "public": 6
22:27:45.733 [INFO ] o.f.c.i.u.l.slf4j.Slf4jLog - Migrating schema "public" to version 7 - update
Exception in thread "main" org.flywaydb.core.internal.exception.FlywaySqlException:
Unable to insert row for version '7' in Schema History table "public"."flyway_schema_history"
---------------------------------------------------------------------------------------------
SQL State : 23502
Error Code : 0
Message : ERROR: null value in column "version_rank" violates not-null constraint
Détail : Failing row contains (null, 7, 7, update, SQL, V7__update.sql, -1303600795, postgres, 2018-02-25 22:28:00.536556, 158, t).
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.doAddAppliedMigration(JdbcTableSchemaHistory.java:171)
at org.flywaydb.core.internal.schemahistory.SchemaHistory.addAppliedMigration(SchemaHistory.java:146)
at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:378)
at org.flywaydb.core.internal.command.DbMigrate.access$400(DbMigrate.java:52)
at org.flywaydb.core.internal.command.DbMigrate$5.call(DbMigrate.java:297)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:75)
at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:294)
at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:259)
at org.flywaydb.core.internal.command.DbMigrate.access$300(DbMigrate.java:52)
at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:179)
at org.flywaydb.core.internal.command.DbMigrate$4.call(DbMigrate.java:176)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:71)
at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:110)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:148)
at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:176)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:145)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1206)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:1168)
at org.flywaydb.core.Flyway.execute(Flyway.java:1655)
at org.flywaydb.core.Flyway.migrate(Flyway.java:1168)
at com.test.MyApplication.main(MainApplication.java:47)
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "version_rank" violates not-null constraint
Détail : Failing row contains (null, 7, 7, update, SQL, V7__update.sql, -1303600795, postgres, 2018-02-25 22:28:00.536556, 158, t).
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2167)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:132)
at org.flywaydb.core.internal.util.jdbc.JdbcTemplate.update(JdbcTemplate.java:334)
at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.doAddAppliedMigration(JdbcTableSchemaHistory.java:165)
... 20 more
Any idea why this column "version_rank" is not generated or not initialized ?
Thanks in advance for your help
You upgraded from Flyway 3.x to 5.x, skipping 4.x. This is not possible as written in the release notes: https://flywaydb.org/documentation/releaseNotes#5.0.0
Upgrade to 4.2.0 first before upgrading to 5.x and everything will work as expected.
Also please take a minute to check the release notes next time you upgrade a major version.
Here's the commit with the metadata table changes if you need to apply the changes manually. I've linked to the version for postgres - search for the version of upgradeMetaDataTable.sql in the folder matching the required dialect.
Fortunately you can apply the changes as a standard flyway migration, as the metadata tables are not used until the end of each script.
E.G. create a migration V999.00__FlywayFix.sql to correct a flyway version table called flyway_table as follows:
DROP INDEX "flyway_table_vr_idx";
DROP INDEX "flyway_table_ir_idx";
ALTER TABLE "flyway_table" DROP COLUMN "version_rank";
ALTER TABLE "flyway_table" DROP CONSTRAINT "flyway_table_pk";
ALTER TABLE "flyway_table" ALTER COLUMN "version" DROP NOT NULL;
ALTER TABLE "flyway_table" ADD CONSTRAINT "flyway_table_pk" PRIMARY KEY ("installed_rank");
UPDATE "flyway_table" SET "type"='BASELINE' WHERE "type"='INIT';
it's worked for me for Postgres
CREATE TABLE flyway_schema_history (
installed_rank INTEGER NOT NULL,
version varchar(50) DEFAULT NULL,
description varchar(200) NOT NULL,
type varchar(20) NOT NULL,
script varchar(1000) NOT NULL,
checksum INTEGER DEFAULT NULL,
installed_by varchar(100) NOT NULL,
installed_on timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
execution_time INTEGER NOT NULL,
success BOOLEAN NOT NULL,
PRIMARY KEY (installed_rank)
);
INSERT INTO flyway_schema_history (installed_rank, version, description, type, script, checksum, installed_by, installed_on, execution_time, success)
SELECT installed_rank, version, description, type, script, checksum, installed_by, installed_on, execution_time, success
FROM schema_version;
ALTER TABLE schema_version RENAME TO bak_schema_version;

SQLSTATE[HY000]: General error: 20003 Adaptive Server connection timed out [20003]

I'm trying to execute a store procedure on a SQL Server 2008.
After about 30 seconds it ends in
SQLSTATE[HY000]: General error: 20003 Adaptive Server connection timed
out [20003]
I've checked if on the SQL Server remote connections are allowed and they are with a timeout of 600 seconds (default).
This is my config.yml
doctrine:
dbal:
default_connection: default
connections:
default: ...
mssql:
driver_class: \Lsw\DoctrinePdoDblib\Doctrine\DBAL\Driver\PDODblib\Driver
host: mssql_freetds
port: "%stage_database_port%"
dbname: "%stage_database_name%"
user: "%stage_database_user%"
password: "%stage_database_password%"
charset: UTF8
options:
timeout: 600 // I don't know if it's correct but it doesn't work even without this
And this is my code
$connection = $this->getDoctrine()->getManager('mssql')
->getConnection();
$stmt = $connection->prepare("Exec SP_MyStoreProcedure ?, ?");
$stmt->bindValue(1, $sd->format("Y-m-d") /* This is a date */);
$stmt->bindValue(2, $ed->format("Y-m-d") /* This is a date */);
$stmt->execute();
[EDIT]
So, i've found that i should configure time out on freetds configuration because of LswDoctrinePdoDblib. I'v edited the /etc/freetds/freetds.conf file but it still ends up connection after 30 seconds
/etc/freetds/freetds.conf
# $Id: freetds.conf,v 1.12 2007/12/25 06:02:36 jklowden Exp $
#
# This file is installed by FreeTDS if no file by the same
# name is found in the installation directory.
#
# For information about the layout of this file and its settings,
# see the freetds.conf manpage "man freetds.conf".
# Global settings are overridden by those in a database
# server specific section
[global]
# TDS protocol version
; tds version = 4.2
# Whether to write a TDSDUMP file for diagnostic purposes
# (setting this to /tmp is insecure on a multi-user system)
; dump file = /tmp/freetds.log
; debug flags = 0xffff
# Command and connection timeouts
timeout = 600
connect timeout = 600
[mssql_freetds]
host = 192.168.0.xx
port = 1433
timeout = 600
tds version = 8.0
client charset = UTF-8
text size = 20971520
[EDIT 2]
Nothing worked for timeout except this:
doctrine:
dbal:
default_connection: default
connections:
default: ...
mssql:
driver_class: \Lsw\DoctrinePdoDblib\Doctrine\DBAL\Driver\PDODblib\Driver
host: mssql_freetds
port: "%stage_database_port%"
dbname: "%stage_database_name%"
user: "%stage_database_user%"
password: "%stage_database_password%"
charset: UTF8
options:
2: 600 # 2 is the equivalent of \PDO::ATTR_TIMEOUT
But the store procedure should end in 17seconds, it waited 10 minutes ending with the same result. I don't know why. It seems to be a FREETDS/SQL SERVER bug or something like that...
[EDIT 3]
Found that the problem was a "NOT IN" clause in the SQL of the stored procedure. I had to replace it with a LEFT JOIN combined with a IS NULL clause in WHERE statement. There's also a "IN" clause but it works. I don't know if it's a FREETDS or a LswDoctrinePdoDblib issue.

Resources