How to create a federated table in MariaDB without specifying columns - mariadb

I am trying to use the Federated engine of MariaDB 10.1.12 to create tables that are based on tables in a remote database. Following the MariaDB instructions about how to use the FederatedX implementation, in database db1 I create a table as
CREATE TABLE test_table (
id int(20) NOT NULL auto_increment,
name varchar(32) NOT NULL default '',
other int(20) NOT NULL default '0',
PRIMARY KEY (id),
KEY name (name),
KEY other_key (other))
DEFAULT CHARSET=latin1;
Now when I want to see this table in a second database db2 using the Federated engine, I can issue
CREATE TABLE test_table (
id int(20) NOT NULL auto_increment,
name varchar(32) NOT NULL default '',
other int(20) NOT NULL default '0',
PRIMARY KEY (id),
KEY name (name),
KEY other_key (other)
) ENGINE=FEDERATED
DEFAULT CHARSET=latin1
CONNECTION='mysql://user_x:pass_y#localhost/db1/test_table';
All this is copied from the MariaDB documentation and works well. However, if I try to create the table without explicitly duplicating the definition of the table structure - an example given in the same documentation
CREATE TABLE test_table ENGINE=FEDERATED DEFAULT CHARSET=latin1
CONNECTION='mysql://user_x:pass_y#localhost/db1/test_table';
MariaDB responds with an error
ERROR 1113 (42000): A table must have at least 1 column
Am I missing something or is it not possible to use federated tables without specifying the individual columns?

It seems that with the standard installation of MariaDB on Ubuntu 14.04, the old Federated engine is active, not the new FederatedX variant. Only the latter supports auto-discovery of columns. In order to correct this, I took the following steps:
uninstall soname 'ha_federated';
to remove the federated plugin. After a server restart, the correct one is loaded with
install plugin federated soname 'ha_federatedx';
This loads the new FederatedX implementation supporting auto-discovery of columns. To view the installed engines, one can list them with
show engines;
If all is correct, there should be this line in the output:
| FEDERATED | YES | FederatedX pluggable storage engine |

Related

MariaDB constraint is incorrectly formed although columns are of the same type

I've got a problem while creating (altering) foreign key.
I have two tables in my DB (created via flyway migrations):
connector (migration)
create table if not exists connector
(
id char(36) not null,
# other fields omitted
primary key (id)
) DEFAULT CHARACTER SET 'utf8mb4'
COLLATE 'utf8mb4_unicode_520_ci';
connector_preset (migration)
CREATE TABLE IF NOT EXISTS connector_preset
(
id CHAR(36) NOT NULL,
# other fields omitted
PRIMARY KEY (id)
);
I want to create a link between connector and connector_preset, so I created another migration like this:
ALTER IGNORE TABLE `connector`
# `connector_preset`.`id`
ADD COLUMN IF NOT EXISTS `preset_id` CHAR(36),
ADD CONSTRAINT `fk_connector_preset_id` FOREIGN KEY (`preset_id`) REFERENCES `connector_preset` (`id`);
but it fails with the following error:
SQL State : HY000
Error Code : 1005
Message : (conn=4) Can't create table `test`.`connector` (errno: 150 "Foreign key constraint is incorrectly formed")
Location : db/migration/...
Line : 34
Statement : ALTER IGNORE TABLE `connector`
# `connector_preset`.`id`
ADD COLUMN IF NOT EXISTS `preset_id` CHAR(36),
ADD CONSTRAINT `fk_connector_preset_id` FOREIGN KEY (`preset_id`) REFERENCES `connector_preset` (`id`),
The columns seem to be of the same type. Also, for some reason it works in local k8s cluster (10.3.29-MariaDB), but fails in integration tests (testcontainers, MariaDB 10.6.11). Also fails in GH Actions which use 10.3.29, which is strange since it's working locally.
UPD: If I set mariadb version in testcontainers to 10.3.29 - it still fails.
UPD: Tried altering connector_preset table to use the same charset and collation:
ALTER IGNORE TABLE `connector_preset` DEFAULT CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_520_ci';
Still doesn't work
Table connector was created with character set utf8mb4 and collation utf8mb4_unicode_520_ci, while connector_preset was created with servers default character set and collation, which obviously differs.
So you have to convert your table to the right character set.
Your attempt to change it with ALTER TABLE ... DEFAULT CHARACTER SET 'utf8mb4' sets the default character set for the table, but not for the columns. Instead you have to convert it with:
ALTER TABLE connector_preset CONVERT TO CHARSET utf8mb4 COLLATE utf8mb4_unicode_520_ci

MariaDB does not finish query for table with foreign key

In a MariaDB data base I've set up I cannot change tables with foreign keys at all. Querys involving alter table and drop table never finish, I don't even get an error message. Same with repair table.
All I can do is hit ctrl+c. There are no apparent errors indicated in the InnoDB monitor output.
I'm quite new to relational data bases, so it's probably a user error. I just can't see what it might be. Any help greatly appreciated!
OS: Windows 10 Enterprise
MariaDB: 10.8
using both the client and a plugin in Visual Studio Code, doesn't matter.
I can alter other tables.
I set foreign_key_checks to off
The table with the foreign key looks like below. participant_id is the foreign key.
Field
Type
Null
Key
Default
Extra
trial_id
smallint(6)
NO
PRI
NULL
begin_trial
datetime
NO
UNI
NULL
participant_id
tinyint(4)
NO
MUL
NULL
The referenced table looks like this, id is participant_id.
Field
Type
Null
Key
Default
Extra
id
tinyint(4)
NO
PRI
NULL
auto_increment
code
char(6)
NO
UNI
NULL
day
date
NO
0000-00-00
Yes, user error. I had two connections open, one using the MariaDB client, another one using the VSC plugin. Thus there must have been some kind of lock on write operationsm but not on read operations.
Thanks for being my rubber duck, Stackoverflow.

SQLite3 insertion into foreign key table doesn't work

I have an SQLite3 database running on a Python Flask server. I am trying to insert values into foreign key database but the values for the foreign key are set at Null.
In my Flask application.py file:
from flask import Flask
from cs50 import SQL
app = Flask(__name__)
db = SQL("sqlite:///test.db")
db.execute("CREATE TABLE 'people' ('id' integer, 'name' varchar(100), PRIMARY KEY(id))")
db.execute("CREATE TABLE 'friendships' (person_id INTEGER,'id' smallint, 'friends' boolean, FOREIGN KEY(person_id) REFERENCES people(id))")
At some point in the code I want to insert a 'friendship' into the referenced database, assuming I have two people with ids 1 and 2 in the 'people' table. Therefore, I run the code:
db.execute("INSERT INTO friendships (person_id, id, friends) VALUES (1, 2, 1)")
This leaves me with a null value for this inserted row in the person_id field, and have no clue why it has happened. I'm also unable to find anything on similar problems online, so I would really appreciate any help.
Edit:
The code works on itself but I tried to insert an input from a non-existent form input value of person_id from the client-side to the server. This resulted in null values being submitted to the server, consequently ending up in the database.

Why is my MariaDb not adding a column to a large table using the INSTANT algorithm

I have a huge table in a MariaDb (10.4.10-MariaDB-1:10.4.10+maria~bionic) and I am adding a new column using
alter table Appointment add column responsible_organization varchar(256);
The existing table is this:
CREATE TABLE `Appointment` (
`id` VARCHAR(256) NOT NULL,
`version` VARCHAR(24) NOT NULL,
`repetition_ref` VARCHAR(256) NULL DEFAULT NULL,
`type` VARCHAR(256) NULL DEFAULT NULL,
`comment` VARCHAR(2048) NULL DEFAULT NULL,
`description` VARCHAR(2048) NULL DEFAULT NULL,
`end` DATETIME NULL DEFAULT NULL,
`start` DATETIME NULL DEFAULT NULL,
`status` VARCHAR(256) NULL DEFAULT NULL,
`statuschangedate` DATETIME NULL DEFAULT NULL,
`deliverystatus` VARCHAR(256) NULL DEFAULT NULL,
`reasoncancelled` VARCHAR(256) NULL DEFAULT NULL,
`visit_type` VARCHAR(256) NULL DEFAULT NULL,
`modified_db_time` TIMESTAMP(3) NOT NULL DEFAULT current_timestamp(3) ON UPDATE current_timestamp(3),
`markedasdeleted` DATETIME NULL DEFAULT NULL,
PRIMARY KEY (`id`, `version`),
INDEX `FKc4f6e4y3ftaya162pwf7v4uj4` (`deliverystatus`),
INDEX `FKeuhxsh83rlweegn404penommb` (`reasoncancelled`),
INDEX `FK5xmurewn61wf4n3of5yx2nsmg` (`visit_type`),
INDEX `modified_db_time` (`modified_db_time`),
CONSTRAINT `FK5xmurewn61wf4n3of5yx2nsmg` FOREIGN KEY (`visittype`) REFERENCES `Coding` (`id`),
CONSTRAINT `FKc4f6e4y3ftaya162pwf7v4uj4` FOREIGN KEY (`deliverystatus`) REFERENCES `CodeableConcept` (`id`),
CONSTRAINT `FKeuhxsh83rlweegn404penommb` FOREIGN KEY (`reasoncancelled`) REFERENCES `CodingDt` (`id`)
)
;
As far as I read the MariaDb documentation it should choose the most efficient algorithm if I don't specify any. I would expect it to use INPLACE at the minimum. But when I run it i can see in the process list that it is running with the state "Copy to tmp table". So this is the COPY algorithm right?
I then tried to force it to use INSTANT as sugested by #o-jones. That gave me this output:
MariaDB [mydb]> alter table Appointment add column responsibleorganisation varchar(256), ALGORITHM=INSTANT;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Cannot change column type. Try ALGORITHM=COPY
Weird, since I am adding a column.
I am wondering if it has something to do with the table being created on an older version of MariaDb and not having been rebuilt recently. I have found references to this being an issue for tables with old style temporal columns.
The variables old_alter_table and alter_algorithm both have the value DEFAULT
My table has 110+ million rows, so I would have liked to find a way to optimize this.
Any ideas?
MariaDB Server 10.4.10 was released over 2 years ago, in November 2019. Is this repeatable with a more recent version? There have been some fixes to ALTER TABLE since then.
If the problem is repeatable with the latest version in the 10.4 series, I would suggest that you file a bug report at https://jira.mariadb.org with a minimal reproducible test case (CREATE TABLE and ALTER TABLE statements).
With a more recent version, the answer could have been that MDEV-20590 introduced a way to disable instant operations that involve changing the data file format. SET GLOBAL innodb_instant_alter_column_allowed=never; would make ADD COLUMN always rebuild the table, like it did before MariaDB Server 10.3.
I have seen this in cases where a database was created in an earlier version of MariaDB. Sometimes the storage format for some columns has changed, but the upgrade did not rewrite all of the data. Instead it waits until certain schema changes are made and then it modifies that underlying data format. In my cases it was because some of the columns were date/datetime and the temporal storage format had changed in 10.3, but many of my tables were still in the old format. Any schema change I made in 10.4 wanted to update the format for all of those columns, which was a pain for tables with many millions of rows.
If you rebuild the table first using "ALTER TABLE tab_name ENGINE = InnoDB;" then the column add would be able to be added "instantly". Although the alter will take as long as the non-instant column add would. However, there are circumstances where doing the processes separately might be advantageous.
In my case I had to do a manual table rebuild because I could not allow the live table to be locked for as long as the rebuild would take. So I wrote a stored procedure to take the table and create a copy of it gradually, and keep the copy up-to-date. The copy process was throttled to avoid putting to much strain on the db and then once the copy was ready I was able to swap the old and new tables quickly with renames so that the down time was measured in seconds instead of hours. Once the new table was in place I was able to do instant ddl operations on it like normal.

MariaDB won't let me make column unique

I recently duplicated an existing MariaDB table (for use on Amazon RDS). The two tables are basically identical, data included, but I noticed the new version was missing all the unique constraints (kept the primary keys just fine). When I tried to add back the uniques, I got an error:
BLOB/TEXT column 'url' used in key specification without a key length
Others have had this problem, and people have claimed that MySQL/MariaDB simply doesn't let you make text columns unique. But my original MariaDB database (on my local machine) has plenty of text columns with unique keys, so that can't be it. I tried their solution anyway (switching to varchar) but MariaDB wouldn't let me do that either because my data has too many characters. Any ideas? Many thanks.
MariaDB 10.4 supports a UNIQUE KEY on a TEXT column because it automatically converts the index to a HASH index:
mysql> create table t (id serial primary key, t text, unique key(t));
Query OK, 0 rows affected (0.01 sec)
mysql> show create table t\G
*************************** 1. row ***************************
Table: t
Create Table: CREATE TABLE `t` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`t` text DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `t` (`t`) USING HASH
) ENGINE=InnoDB DEFAULT CHARSET=latin1
Unfortunately, Amazon RDS for MariaDB does not support MariaDB 10.4 yet (as of 2020-03-19), according to https://aws.amazon.com/rds/mariadb/:
Amazon RDS supports MariaDB Server versions 10.0, 10.1, 10.2, and 10.3 which means that the code, applications, and tools you already use today can be used with Amazon RDS.
The HASH index feature is not supported in MariaDB 10.3 or earlier:
mysql> create table t (id serial primary key, t text, unique key(t));
ERROR 1170 (42000): BLOB/TEXT column 't' used in key specification without a key length
You can create a unique key on a prefix of the text column:
mysql> create table t (id serial primary key, t text, unique key(t(1000)));
Query OK, 0 rows affected (0.01 sec)
mysql> show create table t\G
*************************** 1. row ***************************
Table: t
Create Table: CREATE TABLE `t` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`t` text DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `t` (`t`(1000))
) ENGINE=InnoDB DEFAULT CHARSET=latin1
This means the unique key will enforce uniqueness in the first 1000 characters. You won't be allowed to store two rows with string that have the same first 1000 characters, but differ in the 1001st character or later.

Resources