flyway encoding problems with utf8 - flyway

I am trying to use Flyway to migrate a MySQL database and everything works well except that I have encoding problems with cyrilic symbols that are replaced by '?' sign.
I am using the following Maven plugin.
<plugin>
<groupId>org.flywaydb</groupId>
<artifactId>flyway-maven-plugin</artifactId>
<version>4.2.0</version>
<configuration>
<user>mysql</user>
<password>password</password>
<url>jdbc:mysql://192.168.1.6/schem2</url>
<encoding>UTF-8</encoding>
<schemas>
<schema>schem2</schema>
</schemas>
</configuration>
</plugin>
in datasource bean ...
jdbc:mysql://192.168.1.6/schem2?useUnicode=true&characterEncoding=UTF-8
It seems that everywhere the encoding is specified.
Script...
-- MySQL dump 10.13 Distrib 5.7.12, for Win64 (x86_64)
--
-- Host: 192.168.1.6 Database: schem2
-- ------------------------------------------------------
-- Server version 5.5.50-MariaDB
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET #OLD_TIME_ZONE=##TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET #OLD_UNIQUE_CHECKS=##UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET #OLD_FOREIGN_KEY_CHECKS=##FOREIGN_KEY_CHECKS,
FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET #OLD_SQL_MODE=##SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET #OLD_SQL_NOTES=##SQL_NOTES, SQL_NOTES=0 */;
--
-- Table structure for table `instbl`
--
DROP TABLE IF EXISTS `instbl`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `instbl` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`INN` varchar(12) DEFAULT NULL,
`OGRN` varchar(13) DEFAULT NULL,
`FULLNAME` varchar(255) DEFAULT NULL,
`ADRESS` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = #saved_cs_client */;
--
-- Dumping data for table `instbl`
--
LOCK TABLES `instbl` WRITE;
/*!40000 ALTER TABLE `instbl` DISABLE KEYS */;
INSERT INTO `instbl` VALUES (1,'11111111111','10000000000','компания1','634
пр.Ленина д.1'),(2,'564441111111','1111','frfrf','wrfwerfw3'),
(3,'4545454','4565662','Полное Имя 1','Адресс 1');
/*!40000 ALTER TABLE `instbl` ENABLE KEYS */;
UNLOCK TABLES;
/*!40103 SET TIME_ZONE=#OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=#OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=#OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=#OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=#OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=#OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=#OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=#OLD_SQL_NOTES */;

I had the same problem with Turkish characters, they were all converted to "?" char. As #saw303 said, I modified flyway.url from
jdbc:mysql://192.168.11.7:3306/test_db?autoreconnect=true
to
jdbc:mysql://192.168.11.7:3306/test_db?autoreconnect=true&useUnicode=true&characterEncoding=UTF-8
This modification solved the problem.

Related

Deleting a 3GB table took over two hours on MariaDB

As posted on another forum, when upgrading XWiki from v7.0.1 to v13.10.9, a non-critical database table xwikistatsvisit, for the user visiting statistics, was preventing the after-upgrade migrations. It contained over seven million records and sized 3GB in total. As a workaround, we had to delete all records in the table, but the SQL command delete from table xwikistatsvisit took over two hours.
I have verified from ER diagram that the table is stand-alone without any foreign key referring to or from other tables. And the database is MariaDB v10.9.2 installed on the same host.
The host under test is a medium virtual machine with SSD, 4 CPUs of Intel i9 and 8GB of RAM, running MariaDB v10.9.2. Also, the hypervisor needs to enable “PAE/NX” and “Nested VT-x/ADM-V” for higher performance; otherwise, the computing task will get stuck forever.
My Questions:
Why the SQL command took so long? Is there any way to proceed faster? E.g. disabling the keys and restrictions, etc.; but I am unfamiliar with this area.
I will highly appreciate any hints or suggestions.
The definition of the table xwikistatsvisit:
--
-- Table structure for table `xwikistatsvisit`
--
DROP TABLE IF EXISTS `xwikistatsvisit`;
/*!40101 SET #saved_cs_client = ##character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `xwikistatsvisit` (
`XWV_ID` bigint(20) NOT NULL,
`XWV_NUMBER` int(11) DEFAULT NULL,
`XWV_NAME` varchar(255) NOT NULL,
`XWV_CLASSNAME` varchar(255) DEFAULT NULL,
`XWV_IP` varchar(255) NOT NULL,
`XWV_USER_AGENT` longtext NOT NULL,
`XWV_COOKIE` longtext NOT NULL,
`XWV_UNIQUE_ID` varchar(255) NOT NULL,
`XWV_PAGE_VIEWS` int(11) DEFAULT NULL,
`XWV_PAGE_SAVES` int(11) DEFAULT NULL,
`XWV_DOWNLOADS` int(11) DEFAULT NULL,
`XWV_START_DATE` datetime DEFAULT NULL,
`XWV_END_DATE` datetime DEFAULT NULL,
PRIMARY KEY (`XWV_ID`),
KEY `XWVS_END_DATE` (`XWV_END_DATE`),
KEY `XWVS_UNIQUE_ID` (`XWV_UNIQUE_ID`),
KEY `XWVS_PAGE_VIEWS` (`XWV_PAGE_VIEWS`),
KEY `XWVS_START_DATE` (`XWV_START_DATE`),
KEY `XWVS_NAME` (`XWV_NAME`),
KEY `XWVS_PAGE_SAVES` (`XWV_PAGE_SAVES`),
KEY `XWVS_DOWNLOADS` (`XWV_DOWNLOADS`),
KEY `XWVS_IP` (`XWV_IP`),
KEY `xwv_user_agent` (`XWV_USER_AGENT`(255)),
KEY `xwv_classname` (`XWV_CLASSNAME`),
KEY `xwv_number` (`XWV_NUMBER`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = #saved_cs_client */;
The DELETE command deletes all the matching rows one at a time. If you want to delete all the records from such a big table, TRUNCATE should be a lot faster, since it empties the full table at once.
TRUNCATE TABLE xwikistatsvisit;
Difference between DELETE and TRUNCATE.

Mariadb Can not drop System versioning

Mariadb 10.3
MariaDB [demo]> show create table people;
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| people | CREATE TABLE `people` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`address_id` int(11) DEFAULT NULL,
`civility_id` int(11) DEFAULT NULL,
`marital_status_id` int(11) DEFAULT NULL,
`language_id` int(11) DEFAULT NULL,
`country_id` int(11) DEFAULT NULL,
`import_timestamp` int(11) DEFAULT NULL WITHOUT SYSTEM VERSIONING,
PRIMARY KEY (`id`),
FULLTEXT KEY `indexPeopleMultiWords` (`lastname`,`lastname_of_birth`,`firstname`)
) ENGINE=InnoDB AUTO_INCREMENT=5818 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci WITH SYSTEM VERSIONING
PARTITION BY SYSTEM_TIME
(PARTITION `p_hist` HISTORY ENGINE = InnoDB,
PARTITION `p_cur` CURRENT ENGINE = InnoDB) |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)
But when I try to drop SYSTEM VERSIONING, I got this message error :
MariaDB [nemo]> ALTER TABLE people DROP SYSTEM VERSIONING;
ERROR 4131 (HY000): Can not DROP SYSTEM VERSIONING for table `people` partitioned BY SYSTEM_TIME
Question:
How to drop system versioning in my case ? Thank you
ref: dbfiddle
PARITITION BY SYSTEM TIME is a system versioned construct.
Per the error message you received, you need to remove the partitioning in order to removing the system versioning:
ALTER TABLE people REMOVE PARTITIONING
ALTER TABLE people DROP SYSTEM VERSIONING

mariadb utf8mb4 with dynamic column & procedure

I have two questions for mariadb utf8mb4 with dynamic column.
Above all, I use mariadb version 10.0 and connect by jdbc.
For saving emoji characters, I modified mariadb as follow that,
Edited in /etc/my.cnf
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
Edited DB Table Charset.
CREATE TABLE `MEMBER` (
`name` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`regdate` datetime DEFAULT NULL,
`sso_json` blob,
..(skip)..
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Remove characterEncoding parameter from JDBC String
deleted particle : characterEncoding=utf-8
So, It's perfect, emoji character was saved in a varchar column ㅁccurately. But not in a dynamic column. In both Ad-Hoc Query and procedure, column_create() save question mark instead of emoji.
As follow, procedure sample.
CREATE DEFINER=`sample`#`%` PROCEDURE `SP_INSERT`(
inName varchar(500) CHARACTER SET utf8mb4
)
BEGIN
SET #pSql = CONCAT( ' INSERT INTO SAMPLE_TBL ( '
, ' name, sso_json '
, ' ) VALUES ( '
, ' ?, COLUMN_CREATE(?, ?) '
, ' ) '
);
-- variables bind
SET #pName = inName;
SET #pKey = 'title';
-- prepare stmt
PREPARE pstmt FROM #pSql;
EXECUTE pstmt USING #pName, #pKey, #pName;
END
Procedure Result : {'title', '?????'}.
And In a Ad-Hoc query,
set names utf8mb4 collate 'utf8mb4_unicode_ci';
select 'test😄🍉🚓🚅', column_json(column_create('name','test😄🍉🚓🚅'));
Result :
test😄🍉🚓🚅' || {\"name\":\"test????\"}
result column is accurately but column_json no.
set names utf8;
select 'test😄🍉🚓🚅', column_json(column_create('name','test😄🍉🚓🚅'));
Result :
test😄🍉🚓🚅 || {\"name\":\"test😄🍉🚓🚅\"}
I don't know why. Help me, please.
sso_json blob acquires the table's DEFAULT CHARACTER SET utf8; you need utf8mb4 for Emoji, as you did with name.

ALTERing a sqlite table to add a timestamp column with default value

It isn't possible to ALTER table ADD column foo timestamp default CURRENT_TIMESTAMP in sqlite, but are there any clever workarounds?
SQLite does not actually modify the table contents when adding a column, so the default must be a value that does not require reading the table.
To work around this, you could use a trigger instead of a default value:
Add the column without the default value:
ALTER TABLE MyTable ADD COLUMN foo timestamp DEFAULT NULL;
Use a trigger to set the default value:
ADD TRIGGER MyTable_foo_default
AFTER INSERT ON MyTable
FOR EACH ROW
WHEN NEW.foo IS NULL
BEGIN
UPDATE MyTable
SET foo = CURRENT_TIMESTAMP
WHERE rowid = NEW.rowid;
END;
Alternatively, modify the table contents first so that all rows have a value, then set the default value:
Add the column without the default value:
ALTER TABLE MyTable ADD COLUMN foo timestamp DEFAULT NULL /* replace me */;
Set the column value to something in all rows (the actual value does not matter, the important thing is that all rows now have the new column):
UPDATE MyTable SET foo = CURRENT_TIMESTAMP;
Change the default value (documentation: PRAGMA writable_schema, sqlite_master):
PRAGMA writable_schema = on;
UPDATE sqlite_master
SET sql = replace(sql, 'DEFAULT NULL /* replace me */',
'DEFAULT CURRENT_TIMESTAMP')
WHERE type = 'table'
AND name = 'MyTable';
PRAGMA writable_schema = off;
Reopen the database (otherwise, SQLite won't know about the new default value).

sqlite3 - Column constraint not working

Why is my last insert into the Printer table not returning an error? I thought (hoped) it would be rejected due to referential integrity.
Code
CREATE TABLE IF NOT EXISTS
Client (
hostName TEXT PRIMARY KEY,
MACAddress TEXT NOT NULL,
deviceType TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS
Printer (
hostName TEXT NOT NULL REFERENCES Client(hostName),
outputType TEXT NOT NULL,
printerName TEXT NOT NULL,
PRIMARY KEY (hostname, outputType),
FOREIGN KEY (hostName) REFERENCES Client(hostName)
ON DELETE CASCADE
ON UPDATE CASCADE
);
insert into Client (hostName, MACAddress, deviceType)
values ('billlaptop.private.ycc', 'bc:ea:c5:13:b3:09','Client');
insert into Printer (hostName, outputType, printerName)
values ('billlaptop.private.ycc', 'Receipt', 'Thermal');
insert into Printer (hostName, outputType, printerName)
values ('xxxx.private.ycc', 'Receipt', 'Thermal');
select * from Printer;
Output
+--------------------------------------------------+
| hostName outputType printerName |
+--------------------------------------------------+
| billlaptop.private.ycc Receipt Thermal |
| xxxx.private.ycc Receipt Thermal |
+--------------------------------------------------+
Foreign key constraints are disabled by default per documenation. Use the PRAGMA foreign_keys command to enable:
sqlite> PRAGMA foreign_keys = ON;
Fiddle with expected error.
Per the SQLite3 pragma page, as of SQLite version 3.6.19, the default setting for foreign key enforcement is OFF.
You must issue a statement
PRAGMA foreign_keys = ON;
to enable checking.

Resources