How to update/insert Cyrillic data in MySQL DB (Windows 10, XAMPP CP v3.2.4)? - mariadb

I observe a strange situation in Windows 10 with XAMPP Control Panel v3.2.4
Terminal window is launched and connected to Mysql DB.
Note: prior in terminal window issued command 'chcp 65001' to support UTF8 encoding.
Now when I attempt to update some table with value which is in Cyrillic then MySQL complains about not closed quote symbol. If I replace Cyrillic input to English then command is accepted.
MariaDB [youtube]> update episodes set name='Катя' where id=11;
'>
If I attempt to insert a new record into DB same situation happens
MariaDB [youtube]> insert into episodes (youtube_id,series_id,season,episode,name) values (12345678904,1,0,1,'Катя');
'>
If double quotes are used situation is the same
MariaDB [youtube]> insert into episodes (youtube_id,series_id,season,episode,title) values (12345678904,1,0,1,"Катя");
">
What a magic touch required to make it work through terminal window?
Update:
John suggested to look into configuration file of MariaDB for UTF8 settings.
The settings was changed to the following and the problem still persists
# The MySQL server
default-character-set=utf8mb4
[mysqld]
init-connect=\'SET NAMES utf8\'
character_set_server=utf8
collation_server=utf8_unicode_ci
skip-character-set-client-handshake
character_sets-dir="C:/bin/XAMPP/App/xampp/mysql/share/charsets"
Initially settings was
# The MySQL server
default-character-set=utf8mb4
[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
Server status report
MariaDB [youtube]> \s
--------------
mysql Ver 15.1 Distrib 10.4.10-MariaDB, for Win64 (AMD64), source revision c24ec3cece6d8bf70dac7519b6fd397c464f7a82
Connection id: 17
Current database: youtube
Current user: root#localhost
SSL: Not in use
Using delimiter: ;
Server: MariaDB
Server version: 10.4.10-MariaDB mariadb.org binary distribution
Protocol version: 10
Connection: localhost via TCP/IP
Server characterset: utf8
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
TCP port: 3306
Uptime: 11 min 12 sec
Threads: 7 Questions: 59 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 16 Queries per second avg: 0.087
--------------
MariaDB Documentation has reference to an option --default-character-set=name.
An attempt to use --default-character-set=utf8mb4 on command line had no effect on behavior of insert/update record in terminal client.
mysql -u root -p --default-character-set=utf8mb4 youtube
....
MariaDB [youtube]> update episodes set title='Катя' where id=11;
'>

I highly recommend getting a copy of the freeware program HeidiSQL. It's not perfect and even crashes occasionally though compared to everything else I've worked with? Oh boy, totally worth my time.
Secondly you want to make sure that you're using the following:
Database Character Set: utf8mb4
Database Collation: utf8mb4_unicode_520_ci
These have the greatest UTF-8 support from what I've read from Stackoverflow member Rick James, via his website. He's a database superstar here on SO so if you ever hire him dump buckets of money on his face. His site has a comparison chart. In fact it's been a while and 520 might have been superseded since I last checked.
To set/change the Database Character Set you will need to change the my.cnf configuration file for MariaDB, I recommend using Notepad++ for code editing. This should make any newly created databases use the correct encoding however you may have to go through and manually update the character sets and collations for databases, tables and table columns so do not forget to be thorough!
[client]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_520_ci
Once you do that your queries with Russian/Greek/etc should work normally as they do with English. Keyword: should.
Since I only speak two languages (English and bad English) I encode all characters past a certain Unicode numeric point just to be certain. It'll take up a bit more space however there are sometimes characters added for languages to Unicode after the majority of the language has been defined thus potentially fragmenting language support. If you're interested comment and I'll go find that code for you.
I'm at a moderate level of comprehension and I'm no Rick James though I have about two or three dozen translation pages (use the search feature and search for 'translation') on the site in my profile if you want to see the output. After I did these things I stopped having the data get corrupted. I hope this helps!

Related

Why is equals involving MariaDB JSON not reflexive?

The following MariaDB statement uses JSON_EXTRACT to turn the JSON-escaped string a into a JSON-typed value, and then this value is compared to itself. The comparison comes up as not equal. I thought that equality was reflexive (barring tricky things involving NULL and NaN), that is, a value is always equal to itself. What am I misunderstanding?
SELECT
JSON_EXTRACT('"a"', '$'),
JSON_EXTRACT('"a"', '$') =
JSON_EXTRACT('"a"', '$');
Server info:
Server: Localhost via UNIX socket
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.6.7-MariaDB-2ubuntu1-log - Ubuntu 22.04
Protocol version: 10
User: phpmyadmin#localhost
Server charset: UTF-8 Unicode (utf8mb4)
Apparently the solution is to use JSON_EQUALS(), which was added in MariaDB 10.7. I don't have an instance of MariaDB 10.7 so I can't test it, and dbfiddle only goes up to MariaDB 10.6.
You can, however, unquote the JSON to extract the string value, and test for equality.
SELECT
JSON_UNQUOTE(JSON_EXTRACT('"a"', '$')) AS a,
JSON_UNQUOTE(JSON_EXTRACT('"a"', '$')) =
JSON_UNQUOTE(JSON_EXTRACT('"a"', '$')) AS `a=a`;
a
a=a
a
1
https://dbfiddle.uk/?rdbms=mariadb_10.6&fiddle=6fa8c7156e6fe9213bfbf44dd57e2c63

mariadb slow query not logged

As the title suggests, no log is recorded in the log file even though the related settings have been completed.
slow_query_log_file = /var/log/mysql/mariadb-slow.log
slow_query_log = 1
long_query_time = 1
log_slow_rate_limit = 1000
log_slow_verbosity = query_plan
log-queries-not-using-indexes
This is mariadb's conf content.
When you open the log file, only the basics exist.
Tcp port: 3306 Unix socket: /run/mysqld/mysqld.sock
Time Id Command Argument
logrotate seems to work fine.
After connecting to mysql, I used select sleep(); but it did not work properly.
The result after using the command is 0, which seems to be normal, but the log is not recorded.
Why wouldn't it work?
The new settings will apply only if the MariaDB server instance is restarted. Therefore, the solution, as mentioned in the comment, is to restart the MariaDB server instance in order to apply the new settings.

Why would a database work on one server, and only partially on another?

Sorry if this is a newbie question. I have been searching and can't seem to figure this out.
I am trying to move a PHP webapp from Plesk, to RunCloud. Both are hosted on a Digital Ocean server.
I exported the database from Plesk, and imported it into RunCloud. After updating the config, it only partially works. Some of the tasks read data, a few seem to write, but a majority just break it.
The strange thing is, if I point the config file to the Plesk server, the app works great using it as a remote database.
Here is from the PHPMyAdmin. The only thing that looks fishy is UNIX, vs TCP. But I cannot find a way to change RunCloud to UNIX.
I would copy the errors, but they are across the board. Any ideas? Thank you so much!
PHPMyAdmin Plesk Server:
Server: Localhost via UNIX socket
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.1.48-MariaDB-0ubuntu0.18.04.1 - Ubuntu 18.04
Protocol version: 10
User: prod#localhost
Server charset: UTF-8 Unicode (utf8mb4)
From RunCloud Server:
Server: 127.0.0.1 via TCP/IP
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.4.21-MariaDB-1:10.4.21+maria~focal-log - mariadb.org binary distribution
Protocol version: 10
User: prod#localhost
Server charset: UTF-8 Unicode (utf8)
As a newbie, I learned that 10.1 and 10.4 were rather different. I then went through each error to see if I could find the change on the MariaDB update guide (which is great). Somehow, I came across this post: error code 1292 incorrect date value mysql. It was the zero date. I just added the line in my.cnf that it suggested. Fixed!!!

Mysterious mariadb 10.4.1 ram ussage

i have upgraded from mariadb 10.1.36 to 10.4.8 and i can see mysterious increasing ram ussage on that new version. I also edited innodb_buffer_pool_size ant seems there is no effect if its set to 15M or 4G, ram is just slowly increasing. After while it eat whole ram and oom killer kills mariadb and this is repeating.
My server has 8GB RAM and its increasing like 60-150MB per day. Its not terrible but i have around 150 database servers so its huge problem.
I can temporary fix problem by restarting mariadb and its start again.
Info about database server:
databases: 200+
tables: 28200(141 per database)
average active connections: 100-200
size of stored data: 100-350GB
cpu: 4
ram: 8GB
there is my config:
server-id=101
datadir=/opt/mysql/
socket=/var/lib/mysql/mysql.sock
tmpdir=/tmp/
gtid-ignore-duplicates=True
log_bin=mysql-bin
expire_logs_days=4
wait_timeout=360
thread_cache_size=16
sql_mode="ALLOW_INVALID_DATES"
long_query_time=0.8
slow_query_log=1
slow_query_log_file=/opt/log/slow.log
log_output=TABLE
userstat = 1
user=mysql
symbolic-links=0
binlog_format=STATEMENT
default_storage_engine=InnoDB
slave_skip_errors=1062,1396,1690innodb_autoinc_lock_mode=2
innodb_buffer_pool_size=4G
innodb_buffer_pool_instances=5
innodb_log_file_size=1G
innodb_log_buffer_size=196M
innodb_flush_log_at_trx_commit=1
innodb_thread_concurrency=24
innodb_file_per_table
innodb_write_io_threads=24
innodb_read_io_threads=24
innodb_adaptive_flushing=1
innodb_purge_threads=5
innodb_adaptive_hash_index=64
innodb_flush_neighbors=0
innodb_flush_method=O_DIRECT
innodb_io_capacity=10000
innodb_io_capacity_max=16000
innodb_lru_scan_depth=1024
innodb_sort_buffer_size=32M
innodb_ft_cache_size=70M
innodb_ft_total_cache_size=1G
innodb_lock_wait_timeout=300
slave_parallel_threads=5
slave_parallel_mode=optimistic
slave_parallel_max_queued=10000000
log_slave_updates=on
performance_schema=on
skip-name-resolve
max_allowed_packet = 512M
query_cache_type=0
query_cache_size = 0
query_cache_limit = 1M
query_cache_min_res_unit=1K
max_connections = 1500
table_open_cache=64K
innodb_open_files=64K
table_definition_cache=64K
open_files_limit=1020000
collation-server = utf8_general_ci
character-set-server = utf8
log-error=/opt/log/error.log
log-error=/opt/log/error.log
pid-file=/var/run/mysqld/mysqld.pid
malloc-lib=/usr/lib64/libjemalloc.so.1
I solved it! The problem is memory allocation library.
If you do this SQL query:
SHOW VARIABLES LIKE 'version_malloc_library';
You must to get value "jemalloc" library. If you get only "system", you may have problems.
To change that, you need edit any .conf file in this directory:
/etc/systemd/system/mariadb.service.d/
There, add this line:
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1"
(this library file may be in other folder)
Then you must to restart mysqld
service mysqld stop && systemctl daemon-reload && service mysqld start
You got carried away in increasing values in my.cnf.
Many of the caches grow until hitting their limit, hence the memory growth you experienced.
What is the value from SHOW GLOBAL STATUS LIKE 'Max_used_connections';? Having a large max_connections accentuates several of the other values; lower it.
But perhaps the really bad one(s) involve table caches -- which have units of tables, not bytes. Crank these down a lot:
table_open_cache=64K
innodb_open_files=64K
table_definition_cache=64K
I have exactly the same problem. Is it due to a bad configuration? Or is it a bug of the new version?
mariadb 10.1 was updated to 10.3 just when I upgraded Debian 9 to Debian 10. I tried solve the problem with mariadb 10.4 but nothing changed.
I want to downgrade version but I think it's neccesary dump all database and restore it, and that means being hours without service.
I don't think Debian 10 has to do with the issue
Please read my previous comments about alternative memory allocators...
When jemalloc is used:
When default memory allocator used:
Try with tcmalloc
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4.5.3.1"

Although freetds.conf set to charset UTF-8, pyodbc fetchall returns in unicode?

I am using pyodbc, ODBC with freetds to access a remote MSSQL database. I recently migrated from Centos + Python 2.4 to Ubuntu + Python 2.7 and the two are behaving differently. My freetds.conf is:
[global]
# TDS protocol version
tds version = 4.2
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field.
# Try setting 'text size' to a more reasonable limit
text size = 64512
[server]
host = server
port = 1333
tds version = 8.0
client charset = UTF-8
My Python code is (simplified):
import pyodbc
msConn = pyodbc.connect('DSN=%s;UID=%s;PWD=%s' % (self.dsn, self.user, self.passwd))
msCur = msConn.cursor()
msQuery='''select ...'''
msCur.execute(msQuery % startUpdateStamp)
for row in msRows:
print rows
When I execute the code above on my Centos server (with python 2.4), any special characters are encoded in utf-8. However when executing the same code with same configuration on the Ubuntu server (python 2.7), the special characters are in unicode. It is as if the configuration client charset = UTF-8 is being completely ignored.
However I've tried setting client charset to other values, in return the server does not allow the connection. Which means the client charset setting is getting through. I have also tried changing the tds version = 8.0 to no avail.
I've also looked here, but the answer does not explain why given the same database, two systems give two encodings.
How can I have the Ubuntu return UTF-8 as well?
I'm not proficient at MSSQL or ODBC. Any help is greatly appreciated.

Resources