MySQL 8 eats all CPU on Wordpress docker stack

MySQL 8 eats all CPU on Wordpress docker stack - wordpress

I have a standalone docker stack running a Wordpress 5.9.3 installation under Nginx 1.19 + PHP-FPM 8.1.6 with MySQL 8.0.29. The server has 16x AMD Opteron cores and 16GB RAM.
Here is my docker-compose.yaml file:
php:
image: php:8.1.6-fpm
restart: unless-stopped
volumes:
- ./services/app:/var/www
app:
image: nginx:1.19
restart: unless-stopped
volumes:
- ./services/app:/var/www
db:
image: mysql:8.0.29
hostname: db
restart: unless-stopped
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
security_opt:
- seccomp:unconfined
volumes:
- ./mysql/data:/var/lib/mysql
I have added security_opt: seccomp:unconfined in order to solve multiple mbind: Operation not permitted messages in the log, as per How to fix "mbind: Operation not permitted" in mysql error log.
Problem is, after a very short while, the mysql (db) container eats up the entire CPU forcing me to restart it to get it back working. Memory seems to be doing just fine.
MySQL container logs doesn't really show anything, except normal startup messages (at least for my level of knowledge):
db_1 | 2022-09-19 13:41:54+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.29-1debian10 started.
db_1 | 2022-09-19 13:41:55+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
db_1 | 2022-09-19 13:41:55+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.29-1debian10 started.
db_1 | 2022-09-19T13:41:57.995026Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
db_1 | 2022-09-19T13:41:57.995303Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 1
db_1 | 2022-09-19T13:41:58.091072Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
db_1 | 2022-09-19T13:42:23.116277Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
db_1 | 2022-09-19T13:42:26.278541Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
db_1 | 2022-09-19T13:42:26.278898Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
db_1 | 2022-09-19T13:42:26.290448Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
db_1 | 2022-09-19T13:42:26.494786Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
db_1 | 2022-09-19T13:42:26.495108Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
After reading several articles in here, this is what I ended up adding the following to my cnf file:
[mysqld]
innodb_buffer_pool_size = 35G
innodb_buffer_pool_instances = 8
internal_tmp_mem_storage_engine = MEMORY
innodb_lru_scan_depth = 100
innodb_flush_neighbors = 2
thread_cache_size = 100
I have recreated the containers after every change along the way, but still no luck.
Also tried to limit the container CPU using the deploy policy on docker-compose.yaml, but all it does is to make it fail faster.
Now running SHOW ENGINE INNODB STATUS during one of the freezes gave me this. Which is probably a good thing, except I have no idea how to make any sense from it:
Type: InnoDB
Name: Empty
Status:
=====================================
2022-09-19 13:51:22 140202165667584 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 44 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 390 srv_active, 0 srv_shutdown, 143 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 154
OS WAIT ARRAY INFO: signal count 350
RW-shared spins 0, rounds 0, OS waits 0
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 153365
Purge done for trx's n:o < 153353 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421699164044336, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164043528, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164041912, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164040296, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164037872, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164035448, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164033024, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164039488, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164041104, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164042720, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164034640, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164032216, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164036256, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164037064, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164033832, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164038680, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164031408, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION 421699164030600, not started
0 lock struct(s), heap size 1128, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
1148 OS file reads, 25434 OS file writes, 252 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 77.67 writes/s, 0.14 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 29, seg size 31, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 9296893, node heap has 1 buffer(s)
Hash table size 9296893, node heap has 1 buffer(s)
Hash table size 9296893, node heap has 0 buffer(s)
Hash table size 9296893, node heap has 0 buffer(s)
Hash table size 9296893, node heap has 1 buffer(s)
Hash table size 9296893, node heap has 1 buffer(s)
Hash table size 9296893, node heap has 10 buffer(s)
Hash table size 9296893, node heap has 4 buffer(s)
44.54 hash searches/s, 7.30 non-hash searches/s
---
LOG
---
Log sequence number 116624909
Log buffer assigned up to 116624909
Log buffer completed up to 116624909
Log written up to 116624909
Log flushed up to 116624909
Added dirty pages up to 116624909
Pages flushed up to 116624909
Last checkpoint at 116624909
127 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 0
Dictionary memory allocated 618620
Buffer pool size 2293553
Free buffers 2266335
Database pages 27200
Old database pages 9932
Modified db pages 15335
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 34168, not young 984
16.20 youngs/s, 0.14 non-youngs/s
Pages read 1105, created 140335, written 25100
0.00 reads/s, 348.34 creates/s, 77.53 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 27200, unzip_LRU len: 0
I/O sum[27192]:cur[264], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 286694
Free buffers 283337
Database pages 3355
Old database pages 1245
Modified db pages 1807
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4159, not young 122
3.03 youngs/s, 0.00 non-youngs/s
Pages read 186, created 17392, written 3096
0.00 reads/s, 41.16 creates/s, 9.67 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3355, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 286694
Free buffers 283351
Database pages 3340
Old database pages 1225
Modified db pages 1931
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 3945, not young 123
3.37 youngs/s, 0.14 non-youngs/s
Pages read 124, created 17413, written 3054
0.00 reads/s, 42.65 creates/s, 9.74 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3340, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 286698
Free buffers 283300
Database pages 3395
Old database pages 1237
Modified db pages 1974
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4487, not young 71
2.60 youngs/s, 0.00 non-youngs/s
Pages read 95, created 17538, written 3117
0.00 reads/s, 43.96 creates/s, 9.65 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3395, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 286694
Free buffers 283296
Database pages 3396
Old database pages 1233
Modified db pages 2012
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4769, not young 285
2.91 youngs/s, 0.00 non-youngs/s
Pages read 134, created 17706, written 3146
0.00 reads/s, 43.64 creates/s, 9.62 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3396, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 286694
Free buffers 283197
Database pages 3495
Old database pages 1275
Modified db pages 1988
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4297, not young 64
2.73 youngs/s, 0.00 non-youngs/s
Pages read 144, created 17832, written 3153
0.00 reads/s, 46.04 creates/s, 9.74 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 3 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3495, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 286691
Free buffers 283274
Database pages 3415
Old database pages 1247
Modified db pages 1959
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4439, not young 45
0.66 youngs/s, 0.00 non-youngs/s
Pages read 59, created 17666, written 3178
0.00 reads/s, 45.34 creates/s, 9.67 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3415, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 286698
Free buffers 283243
Database pages 3453
Old database pages 1254
Modified db pages 1845
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 3641, not young 66
0.50 youngs/s, 0.00 non-youngs/s
Pages read 163, created 17401, written 3212
0.00 reads/s, 43.03 creates/s, 9.74 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3453, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 286690
Free buffers 283337
Database pages 3351
Old database pages 1216
Modified db pages 1819
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4431, not young 208
0.41 youngs/s, 0.00 non-youngs/s
Pages read 200, created 17387, written 3144
0.00 reads/s, 42.51 creates/s, 9.71 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3351, unzip_LRU len: 0
I/O sum[3399]:cur[33], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=1, Main thread ID=140202194016000 , state=sleeping
Number of rows inserted 400936, updated 0, deleted 0, read 409667
1004.82 inserts/s, 0.00 updates/s, 0.00 deletes/s, 397.56 reads/s
Number of system rows inserted 0, updated 317, deleted 0, read 19911
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 13.20 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
I appreciate any help. Thanks.

Related

How should mysql my.cnf settings be? Innodb Mariadb 10 VestaCP

My.cnf
[mysql]
port = 3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1
datadir = /var/lib/mysql
max_allowed_packet = 1024M
max_connect_errors = 1000000
pid-file=/var/lib/mysql/mysql.pid
port = 3306
skip_name_resolve
socket=/var/lib/mysql/mysql.sock
tmpdir = /tmp
sql_mode = ""
thread_handling = pool-of-threads
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_size = 36G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 1G
innodb_sort_buffer_size = 4M # UPD - Defines how much data is read into memory for sorting operations before writing to disk (default is 1M / max is 64M)
innodb_stats_on_metadata = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_io_capacity = 2000 # Depends on the storage tech - use 2000 for SSD, more for NVMe
innodb_io_capacity_max = 4000 # Usually double the value of innodb_io_capacity
# === Connection Settings ===
max_connections = 100 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# === Buffer Settings ===
join_buffer_size = 4M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 60000 # UPD
max_heap_table_size = 128M # Increase to 256M or 512M if you have lots of temporary tables because of missing indices in JOINs
tmp_table_size = 128M # Use same value as max_heap_table_size
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Binary Logging ===
disable_log_bin = 1
# === Error & Slow Query Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 0 # Disabled on production
long_query_time = 5
slow_query_log = 0 # Disabled on production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
quick
quote_names
max_allowed_packet = 1024M
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
My ram & cpu
[root#localhost ~]# cat /proc/cpuinfo | grep processor | wc -l
24
[root#localhost ~]# free -m
total used free shared buff/cache available
Mem: 48125 6136 12446 50 29541 41490
Swap: 0 0 0
mysqltuner
[root#localhost ~]# ./mysqltuner.pl
>> MySQLTuner 1.9.8
* Jean-Marie Renouard <jmrenouard#gmail.com>
* Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.pl/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[!!] Your MySQL version 10.8.3-MariaDB is EOL software! Upgrade soon!
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/mysql_error.log exists
[--] Log file: /var/lib/mysql/mysql_error.log(70K)
[OK] Log file /var/lib/mysql/mysql_error.log is not empty
[OK] Log file /var/lib/mysql/mysql_error.log is smaller than 32 Mb
[OK] Log file /var/lib/mysql/mysql_error.log is readable.
[!!] /var/lib/mysql/mysql_error.log contains 339 warning(s).
[!!] /var/lib/mysql/mysql_error.log contains 51 error(s).
[--] 2 start(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:05 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 2) 2022-06-17 21:30:12 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 1 shutdown(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:04 0 [Note] /usr/sbin/mariadbd: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in Aria tables: 32.0K (Tables: 1)
[--] Data in InnoDB tables: 1.7G (Tables: 845)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Views Metrics -----------------------------------------------------------------------------
-------- Triggers Metrics --------------------------------------------------------------------------
-------- Routines Metrics --------------------------------------------------------------------------
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 4d 6h 58m 28s (166M q [448.474 qps], 937K conn, TX: 203G, RX: 12G)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory : 47.0G
[--] Max MySQL memory : 137.9G
[--] Other process memory: 0B
[--] Total buffers: 36.4G global + 1.0G per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 93.2G (198.36% of installed RAM)
[!!] Maximum possible memory usage: 137.9G (293.37% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (53/166M)
[OK] Highest usage of available connections: 56% (56/100)
[OK] Aborted connections: 0.01% (107/937871)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 874K sorts)
[!!] Joins performed without indexes: 46708
[OK] Temporary tables created on disk: 2% (6K on disk / 305K total)
[--] Thread cache not used with thread pool enabled
[OK] Table cache hit rate: 99% (77M hits / 77M requests)
[OK] table_definition_cache(40000) is upper than number of tables(1137)
[OK] Open file limit used: 0% (29/32K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)
-------- Performance schema ------------------------------------------------------------------------
[!!] Performance_schema should be activated.
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 24 thread(s).
[--] Using default value is good enough for your version (10.8.3-MariaDB)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] No MyISAM table(s) detected ....
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 36.0G/1.7G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (2.77777777777778 %): 1.0G * 1/36.0G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 64 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (36551426226 hits/ 36551507601 total)
[OK] InnoDB Write log efficiency: 90.84% (7736631 hits/ 8516998 total)
[OK] InnoDB log waits: 0.00% (0 waits / 780367 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/344.0K
[!!] Aria pagecache hit rate: 82.5% (38K cached / 6K reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
You are using n unsupported version for production environments
Upgrade as soon as possible to a supported version !
Check warning line(s) in /var/lib/mysql/mysql_error.log file
Check error line(s) in /var/lib/mysql/mysql_error.log file
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
Performance schema should be activated for better diagnostics
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this:
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
join_buffer_size (> 4.0M, or always use indexes with JOINs)
performance_schema=ON
innodb_log_file_size should be (=9G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
[root#localhost ~]#
Tuning Primer
[root#localhost ~]# ./tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.8.3-MariaDB x86_64
Uptime = 4 days 9 hrs 29 min 40 sec
Avg. qps = 454
Total Questions = 172718132
Threads Connected = 33
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.8/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 5.000000 sec.
You have 53 out of 172718244 that take longer than 5.000000 sec. to complete
Your long_query_time seems to be fine
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.8/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 0
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 100
Current threads_connected = 33
Historic max_used_connections = 56
The number of used connections is 56% of the configured maximum.
Your max_connections variable seems to be fine.
No InnoDB Support Enabled!
MEMORY USAGE
Max Memory Ever Allocated : 36.97 G
Configured Max Per-thread Buffers : 1.48 G
Configured Max Global Buffers : 36.14 G
Configured Max Memory Limit : 37.62 G
Physical Memory : 46.99 G
Max memory limit seem to be within acceptable norms
KEY BUFFER
No key reads?!
Seriously look into using some indexes
Current MyISAM index space = 0 bytes
Current key_buffer_size = 128 M
Key cache miss rate is 1 : 0
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 1 M
Current query_cache_used = 16 K
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 1.65 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
Current join_buffer_size = 4.00 M
You have had 48458 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 16319 tables
Current table_definition_cache = 40000 tables
You have a total of 957 tables
You have 1124 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 315112 temp tables, 2% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 155 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 0 : 172720142
Your table locking seems to be fine
[root#localhost ~]#
I can allocate 40gb ram to mariadb server.
I need 8gb ram. We can use all 24 processors.
Please help me how to do the settings.
The site will be used for a web-based game.
The tables were all using MyISAM.
I converted all of them to innodb and it works fine.
If you need additional information, let me know in the comments.

These seem excessive:
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
Will you have thousands of tables? Why? Even if you do have that many, these are caches, hence they don't have to be big enough to handle everything.
For better performance turn on the slowlog with a low value such as long_query_time = 1. After a few hours, use pt-query-digest. More on the SlowLog
For a deeper analysis of the settings: http://mysql.rjweb.org/doc.php/mysql_analysis#tuning

Rate Per Second = RPS
Suggestions to consider for your my.cnf [mysqld] section to improve performance
read_rnd_buffer_size=32K # from 4M to reduce handler_read_rnd_next RPS of 20,873
read_buffer_size=1M # from 3M to reduce handler_read_next RPS of 20,918
innodb_buffer_pool_size=4G # from 36G because you only have 1.7G of data and indexes
net_buffer_length=96K # from 16K to accommodate sending 700MB data per hr
Let us know how your system is performing in a few days, please. View profile for contact information and we do have FREE Utility Scripts to assist with performance tuning.
This is only the beginning of improving performance for your installation. As you get deeper into the environment you will find many opportunities to improve response time and reduce system overhead. We are available to assist.

MariaDB Server 10.6.7 stuck on INSERT query after computer resumes from sleep

I have a recent updated MariaDB installation that suddenly hangs on a every-minute-cronjob after my computer resumes from sleep. Configuration worked fine before latest apt upgrade.
Process list shows an insert query from a ~100 insert batch operation.
I tried cancelling all running scripts before sending the computer to sleep, but at the next wakeup it will still fail. I have to kill -9 the database to even get it to restart.
Id? User? Host? db? Command? Time? State? Info? Progress?
52 xxxxxx localhost xxxxxx Execute 4 Update INSERT INTO xxxxxxx
and it will never finish. This also happens if i wait ~30 seconds after wakeup and start the script manually, so i kinda guess MariaDB handles sleep/resume state not too well.
Is this a known bug? Any way around having my PC run 24/7 to get around this?
Server variables too much for this post, only changed
innodb_buffer_pool_size = 64G
innodb_log_file_size = 16G
As suggested, i checked SHOW ENGINE INNODB STATUS while the lock is happening, cannot post this as a comment due to length
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 4664 srv_idle
srv_master_thread log flush and writes: 4663
----------
SEMAPHORES
----------
------------
TRANSACTIONS
------------
Trx id counter 314781977
Purge done for trx's n:o < 314781976 undo n:o < 0 state: running
History list length 142
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 314781976, ACTIVE 106 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 1
MariaDB thread id 57, OS thread handle 140131630528064, query id 273212 localhost xxx Update
INSERT INTO xyzxyz (`xyz`, `xyz`, `time`, `xyz`, `xyz`) VALUES (?,?,?,?,?)
--------
FILE I/O
--------
Pending flushes (fsync) log: 0; buffer pool: 0
19307 OS file reads, 24298 OS file writes, 24302 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 932, free list len 2162, seg size 3095, 1 merges
merged operations:
insert 11, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 44207022268
Log flushed up to 44207022268
Pages flushed up to 44197540796
Last checkpoint at 44197540784
0 pending log flushes, 0 pending chkp writes
24306 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 68753031168
Dictionary memory allocated 424873192
Buffer pool size 4153344
Free buffers 4133333
Database pages 20011
Old database pages 7367
Modified db pages 3024
Percent of dirty pages(LRU & free pages): 0.073
Max dirty pages percent: 90.000
Pending reads 1
Pending writes: LRU 0, flush list 0
Pages made young 32, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 19243, created 741, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 20011, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 read views open inside InnoDB
Process ID=0, Main thread ID=0, state: sleeping
Number of rows inserted 24161, updated 113, deleted 0, read 623966
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
I checked whether it's a deadlock due to a bad update-query, but it's and INSERT-query with State update which just means mariadb is about to update the table (but never does). I use mariadb 10.6.7 on debian testing - i may need to use a stable release to sort this out. Sleeping happens from seconds (still happening) to hours.

I tried:
Removed PRIMARY and UNIQUE-key with the same indizes. Same problem
Change innodb_flush_log_at_trx_commit to zero to not have it update log after every insert - still stuck on first insert.

MariaDB sometime very long request does not appears in slow logs

I am using MariaDB 10.4.12-MariaDB-1:10.4.12+maria~stretch-log with
innodb on a debian stretch.
I am facing a problem of very slow insert/update/delete queries that
take more than 10 seconds but does not appear in the slow query log.
These long time running requests happen ONLY when the server receive
more requests as usual ( about 20/s ).
The variables for logging slow requests are as follow :
MariaDB [(none)]> show variables like 'long_query_time';
+-----------------+-----------+
| Variable_name | Value |
+-----------------+-----------+
| long_query_time | 10.000000 |
+-----------------+-----------+
1 row in set (0.001 sec)
For instance, the following update statement is very simple and use the primary key of the table :
delete from WEDMATTP Where AttRef like 'EDM-%-FOLDER' and DocId in( 1638486)
There is actually NO records with AttRef like 'EDM-%-FOLDER' and there was only one for DocId in( 1638486)
The EXPLAIN command says :
+------+-------------+----------+-------+-------------------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+----------+-------+-------------------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | WEDMATTP | range | PRIMARY,WEDMATTP_DocNum | PRIMARY | 70 | NULL | 1 | Using where |
+------+-------------+----------+-------+-------------------------+---------+---------+------+------+-------------+
1 row in set (0.001 sec)
The table was created as follow :
MariaDB [abngmadbprod]> show create table wedmattp\G
*************************** 1. row ***************************
Table: wedmattp
Create Table: CREATE TABLE "wedmattp" (
"DocId" int(11) NOT NULL,
"AttRef" varchar(64) NOT NULL,
"AttDate" varchar(14) NOT NULL,
"AttUser" varchar(20) DEFAULT NULL,
"AttMode" varchar(1) DEFAULT NULL,
"UsrId" int(11) DEFAULT NULL,
"AttEndDate" varchar(14) DEFAULT NULL,
PRIMARY KEY ("DocId","AttRef"),
KEY "WEDMATTP_DocNum" ("AttRef"),
KEY "WEDMATTP_AttDate" ("AttDate"),
KEY "WEDMATTP_Usr" ("UsrId"),
KEY "WEDMATTP_AttUser" ("AttUser"),
CONSTRAINT "FK_DocAtt" FOREIGN KEY ("DocId") REFERENCES "wedmdocp" ("DocId")
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.483 sec)
BUT when I look at the processes while this request is running :
Id User Host db Command Time State Info Progress
1 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
2 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
3 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
5 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
13 repl srvdatae.antemeta.lan:46504 NULL Binlog Dump 105901 Master has sent all binlog to slave; waiting for binlog to be updated NULL 0.000
325584 wyxuser_g_abn srvappd:39384 abnwyxdbprod Sleep 30 NULL 0.000
325589 wyxuser_g_abn srvappe:57448 abnwyxdbprod Sleep 30 NULL 0.000
325663 wyxuser_ph_edm srvappe:57526 abnwyxdbprod Query 14 Updating delete from WEDMATTP Where AttRef like 'EDM-%-FOLDER' and DocId in( 1638486) 0.000
325673 wyxuser_ph_edm srvappd:39468 abnwyxdbprod Query 14 Waiting for table level lock INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode ) \n VALUES (1638487, 'WYX-10701/GLI-448398-DAT', '20211129081046', '-NESTOR-BURMA-', 'P') 0.000
326176 wyxuser_g_abn srvappe:58170 abnwyxdbprod Sleep 8 NULL 0.000
326199 wyxuser_g_abn srvappe:58208 abnwyxdbprod Sleep 7 NULL 0.000
326290 wyxuser_ph_edm srvappe:58302 abnwyxdbprod Query 3 Waiting for table level lock insert into WEDMITMP(StkId,ItmId,ItmDate,ItmOrder) values( 7503,1638488,\n'20211129081058', 0 ) 0.000
326308 wyxuser_ph_edm srvappe:58330 abnwyxdbprod Query 3 Waiting for table level lock insert into WEDMITMP(StkId,ItmId,ItmDate,ItmOrder) values( 7503,1638489,\n'20211129081058', 0 ) 0.000
326441 wyxuser_g_oauth srvappd:40056 oauth2prod Sleep 0 NULL 0.000
326457 wyxuser_g_admin srvappd:40088 adminwyxdbprod Sleep 0 NULL 0.000
326458 root localhost NULL Query 0 Init show full processlist 0.000
The request was running for 14 seconds when I looked at the process but it does not appears in the slow logs.
It usually run very fast but not at that moment. And the serveur does not seems to be very loaded (only 4 requests running).
Some other common request that run fast (<1s) most of the time may take more than 60s in these kind of situation.
I suspected a dead-lock but in the case of the show processlist above, could it be possible to have a deadlock if the delete request is currently in "Updating" state ?
Sorry to reply very late. I was waiting for the error to occur again to be able to provide the innodb status.
Dimension of the table wedmattp select count(*) from wedmattp more than 1700000 rows.
I can't tell the result of SELECT COUNT(*) FROM wedmattp WHERE DocId in( 1638486) for this specific value of Docid but 99% of Docid have less than 7 rows.
All our client connections are explicitely in latin1.
Here the new log I got with the innodb status while the error (long request) occurs. First the show processlist
=== mercredi 1 décembre 2021, 07:03:01 (UTC+0100)
Id User Host db Command Time State Info Progress
1 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000
2 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
3 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000
5 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000
13 repl srvdatae.antemeta.lan:46504 NULL Binlog Dump 271021 Master has sent all binlog to slave; waiting for binlog to be updated NULL 0.000
1714646 wyxuser_g_abn srvappe:42218 abnwyxdbprod Sleep 33 NULL 0.000
1714673 wyxuser_g_abn srvappe:42252 abnwyxdbprod Sleep 31 NULL 0.000
1714677 wyxuser_ph_edm srvappe:42260 abnwyxdbprod Sleep 30 NULL 0.000
1714682 wyxuser_g_abn srvappe:42270 abnwyxdbprod Sleep 30 NULL 0.000
1714685 wyxuser_ph_edm srvappe:42276 abnwyxdbprod Query 29 Waiting for table level lock INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode ) \n VALUES (1648830, 'WYX-16665/GLI-845449-DAT', '20211201060231', '-NESTOR-BURMA-', 'P') 0.000
1714695 wyxuser_ph_edm srvappe:42288 abnwyxdbprod Query 29 Update INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode ) \n VALUES (1648831, 'WYX-3315/251985-CHR', '20211201060231', '-NESTOR-BURMA-', 'P') 0.000
1714702 wyxuser_g_abn srvappe:42302 abnwyxdbprod Sleep 26 NULL 0.000
1714711 wyxuser_ph_edm srvappe:42320 abnwyxdbprod Query 25 Waiting for table level lock INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode ) \n VALUES (1648832, 'WYX-10012/FAC-72539-DAT', '20211201060236', 'DR7FFC49', 'P') 0.000
1714784 root localhost NULL Query 0 Init show full processlist 0.000
The ps for the mariadb process (I got 2 MariaDB serveurs on the VM).
PID TID PRI PSR %CPU STAT COMMAND
33006 33006 19 0 4.7 Ssl /usr/sbin/mysqld
33176 33176 19 0 0.9 Ssl /usr/sbin/mysqld --defaults-file=/etc/mysql/my.prep.cnf
then the show engine innodb status
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2021-12-01 07:03:01 0x7f0f045ba700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 60 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 64402 srv_active, 0 srv_shutdown, 206431 srv_idle
srv_master_thread log flush and writes: 270771
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2448713
OS WAIT ARRAY INFO: signal count 1451239
RW-shared spins 0, rounds 93744, OS waits 30846
RW-excl spins 0, rounds 71000, OS waits 1448
RW-sx spins 640, rounds 15776, OS waits 386
Spin rounds per wait: 93744.00 RW-shared, 71000.00 RW-excl, 24.65 RW-sx
------------------------
LATEST FOREIGN KEY ERROR
------------------------
2021-11-29 17:23:12 0x7f10a063b700 Transaction:
TRANSACTION 48868328, ACTIVE 1 sec updating or deleting
mysql tables in use 2, locked 2
36 lock struct(s), heap size 3520, 24 row lock(s), undo log entries 7
MySQL thread id 712238, OS thread handle 139709387093760, query id 5527984 srvappe 10.100.15.22 wyxuser_g_abn Updating
Delete from exercice where
AbnId=13118
and ExrId=712495
and AbnId = 13118
Foreign key constraint fails for table "abnwyxdbprod"."documentcomptable":
,
CONSTRAINT "FK_DCTEXR" FOREIGN KEY ("AbnId", "ExrId") REFERENCES "exercice" ("ABNID", "EXRID")
Trying to delete or update in parent table, in index PRIMARY tuple:
DATA TUPLE: 19 fields;
0: len 8; hex 800000000000333e; asc 3>;;
1: len 4; hex 800adf2f; asc /;;
2: len 6; hex 000002e9abe8; asc ;;
3: len 7; hex 7300001d6c2404; asc s l$ ;;
4: len 4; hex 800934d9; asc 4 ;;
5: len 3; hex 8fcb1f; asc ;;
6: SQL NULL;
7: SQL NULL;
8: SQL NULL;
9: SQL NULL;
10: SQL NULL;
11: SQL NULL;
12: SQL NULL;
13: len 3; hex 8fc921; asc !;;
14: len 5; hex 99aa84cbe4; asc ;;
15: SQL NULL;
16: len 8; hex 8000000000000ff0; asc ;;
17: SQL NULL;
18: SQL NULL;
But in child table "abnwyxdbprod"."documentcomptable", in index FK_DCTEXR, there is a record:
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 8; hex 800000000000333e; asc 3>;;
1: len 4; hex 800adf2f; asc /;;
2: len 4; hex 80008c0c; asc ;;
------------
TRANSACTIONS
------------
Trx id counter 49129983
Purge done for trx's n:o < 49129967 undo n:o < 0 state: running but idle
History list length 6
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421184365078104, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421184365073848, not started
mysql tables in use 2, locked 2
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421184365069592, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 49129968, ACTIVE 30 sec inserting
mysql tables in use 2, locked 2
LOCK WAIT 4 lock struct(s), heap size 1136, 2 row lock(s)
MySQL thread id 1714695, OS thread handle 139702474045184, query id 13382588 srvappe 10.100.15.22 wyxuser_ph_edm Update
INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode )
VALUES (1648831, 'WYX-3315/251985-CHR', '20211201060231', '-NESTOR-BURMA-', 'P')
------- TRX HAS BEEN WAITING 30 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2184 page no 35846 n bits 88 index PRIMARY of table "abnwyxdbprod"."wedmattp" trx id 49129968 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
------------------
---TRANSACTION 49129964, ACTIVE 30 sec
mysql tables in use 2, locked 2
9 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 5
MySQL thread id 1714685, OS thread handle 139702475888384, query id 13382604 srvappe 10.100.15.22 wyxuser_ph_edm Waiting for table level lock
INSERT ignore INTO WEDMATTP (DocId, AttRef, AttDate,AttUser, AttMode )
VALUES (1648830, 'WYX-16665/GLI-845449-DAT', '20211201060231', '-NESTOR-BURMA-', 'P')
Trx read view will not see trx with id >= 49129964, sees < 49129964
---TRANSACTION 49129965, ACTIVE 30 sec
5 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 2
MySQL thread id 1714682, OS thread handle 139702634723072, query id 13382505 srvappe 10.100.15.22 wyxuser_g_abn
Trx read view will not see trx with id >= 49129964, sees < 49129964
---TRANSACTION 421184365056824, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421184365048312, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421184365044056, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421184365039800, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
239885 OS file reads, 1261322 OS file writes, 792802 OS fsyncs
0.03 reads/s, 16384 avg bytes/read, 23.67 writes/s, 11.38 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 2465, seg size 2467, 5023 merges
merged operations:
insert 7935, delete mark 538, delete 64
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 1593833, node heap has 5997 buffer(s)
Hash table size 1593833, node heap has 672 buffer(s)
Hash table size 1593833, node heap has 1167 buffer(s)
Hash table size 1593833, node heap has 3436 buffer(s)
Hash table size 1593833, node heap has 856 buffer(s)
Hash table size 1593833, node heap has 3387 buffer(s)
Hash table size 1593833, node heap has 603 buffer(s)
Hash table size 1593833, node heap has 1399 buffer(s)
13150.30 hash searches/s, 639.77 non-hash searches/s
---
LOG
---
Log sequence number 35793553330
Log flushed up to 35793553330
Pages flushed up to 35793553330
Last checkpoint at 35793553321
0 pending log flushes, 0 pending chkp writes
323373 log i/o's done, 4.05 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 6476005376
Dictionary memory allocated 729456
Buffer pool size 384720
Free buffers 114421
Database pages 252782
Old database pages 93407
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 983, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 239608, created 13174, written 794983
0.03 reads/s, 0.03 creates/s, 18.28 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 252782, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 48090
Free buffers 14073
Database pages 31811
Old database pages 11759
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 112, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 30092, created 1719, written 131533
0.02 reads/s, 0.02 creates/s, 4.47 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31811, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 48090
Free buffers 14285
Database pages 31624
Old database pages 11692
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 111, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 29985, created 1639, written 95837
0.00 reads/s, 0.00 creates/s, 0.93 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31624, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 48090
Free buffers 14213
Database pages 31657
Old database pages 11705
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 119, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 30089, created 1568, written 56444
0.00 reads/s, 0.00 creates/s, 1.60 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31657, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 48090
Free buffers 15270
Database pages 30638
Old database pages 11323
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 136, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 28829, created 1809, written 172234
0.02 reads/s, 0.00 creates/s, 4.33 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 30638, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 48090
Free buffers 14058
Database pages 31870
Old database pages 11768
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 137, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 30401, created 1469, written 111736
0.00 reads/s, 0.02 creates/s, 2.68 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31870, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 48090
Free buffers 13700
Database pages 32215
Old database pages 11911
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 131, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 30630, created 1585, written 91920
0.00 reads/s, 0.00 creates/s, 1.95 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 32215, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 48090
Free buffers 14818
Database pages 31074
Old database pages 11489
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 116, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 29606, created 1468, written 67979
0.00 reads/s, 0.00 creates/s, 1.57 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31074, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 48090
Free buffers 14004
Database pages 31893
Old database pages 11760
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 121, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 29976, created 1917, written 67300
0.00 reads/s, 0.00 creates/s, 0.75 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 31893, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
2 read views open inside InnoDB
Process ID=33006, Main thread ID=139702735435520, state: sleeping
Number of rows inserted 296865, updated 190576, deleted 9070, read 8255654063
2.97 inserts/s, 1.15 updates/s, 0.03 deletes/s, 59423.73 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 5378
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

If mysqld crashes before a query finishes, that query does not get written to the slowlog. This is an unfortunate fact -- sometimes a long-running query is a factor in causing a crash.
If it did not crash, then we will look deeper.
Please provide SELECT COUNT(*) FROM wedmattp WHERE DocId in( 1638486).
And... SHOW ENGINE=InnoDB STATUS; during and/or after the Delete is run.
It is not obvious what is causing "Waiting for table level lock", but the Delete is implicated.
What CHARACTER SET is the connection (for the Delete) using when connecting?
Meanwhile, I recommend lowering long_query_time to 1. (It won't help the current issue, but will help you find more slow queries.)
More
The EXPLAIN command says
Was that EXPLAIN UPDATE... or EXPLAIN against an equivalent Select?
Please turn on Explains in the slow log for when we can get something showing there. (I think it is something like log_slow_verbosity=explain)
Meanwhile, do SHOW EXPLAIN to get info on the running query.

innodb table level lock

We face the table level lock issues, almost every day on our test servers.
TRANSACTION 0, not started
mysql tables in use 97, locked 97
MySQL thread id 429, OS thread handle 0x2aff6ff59700, query id 24900 ec2-*-*-*-*.compute-1.amazonaws.com *.*.*.* sminq cleaning up
---TRANSACTION 10631403, not started
MySQL thread id 321, OS thread handle 0x2aff7b359700, query id 24901 115.112.140.139 sminq init
show engine innodb status
---TRANSACTION 10632661, not started
MySQL thread id 13, OS thread handle 0x2aff4e39a700, query id 24817 localhost 127.0.0.1 rdsadmin cleaning up
---TRANSACTION 10632664, not started
MySQL thread id 6, OS thread handle 0x2aff396c5700, query id 24873 ec2-*-*-*-*.ap-southeast-1.compute.amazonaws.com *.*.*.* sminq cleaning up
---TRANSACTION 10632655, not started
MySQL thread id 7, OS thread handle 0x2aff39706700, query id 24783 ec2-*-*-*-*.ap-southeast-1.compute.amazonaws.com *.*.*.* sminq cleaning up
---TRANSACTION 10632652, not started
MySQL thread id 3, OS thread handle 0x2aff37d28700, query id 24745 ec2-*-*-*-*.ap-southeast-1.compute.amazonaws.com *.*.*.* sminq cleaning up
---TRANSACTION 10627075, not started
MySQL thread id 1, OS thread handle 0x2aff37ca6700, query id 0 Waiting for background binlog tasks
---TRANSACTION 10632663, ACTIVE 7 sec
mysql tables in use 1, locked 1
MySQL thread id 431, OS thread handle 0x2aff37daa700, query id 24863 172.31.3.120 sminq Waiting for table level lock
insert into `sminq`.`Queue_token` (`token_queue_id`, `total_process_time`, `token_user`, `created_on`, `join_date`, `join_time`, `app_type`, `token_user_group`, `uuid`) values (13, 10, 87, '2016-07-21 04:47:04.157000', '2016-07-21 10:17:04', '10:10:00', 1, NULL, 'D<??BY??7?gk?Uo')
Trx #rec lock waits 0 #table lock waits 0
Trx total rec lock wait time 0 SEC
Trx total table lock wait time 0 SEC
---TRANSACTION 10632646, ACTIVE 45 sec
These keep occurring for Inserts only, we have never faced the issue with Updates or deletes.
I have an Isolation level READ-COMMITTED, along with innodb_autoinc_lock_mode = 2
mysqltuner output for the same server
[--] Up for: 2h 11m 55s (25K q [3.230 qps], 478 conn, TX: 3M, RX: 1M)
[--] Reads / Writes: 82% / 18%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Total buffers: 1.5G global + 17.0M per thread (100 max threads)
[!!] Maximum reached memory usage: 3.0G (152.35% of installed RAM)
[!!] Maximum possible memory usage: 3.1G (156.50% of installed RAM)
[OK] Slow queries: 0% (0/25K)
[!!] Highest connection usage: 95% (95/100)
[OK] Aborted connections: 0.00% (0/478)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 1K sorts)
[OK] Temporary tables created on disk: 24% (424 on disk / 1K total)
[OK] Thread cache hit rate: 80% (95 created / 478 connections)
[OK] Table cache hit rate: 129% (291 open / 224 opened)
[OK] Open file limit used: 0% (64/65K)
[OK] Table locks acquired immediately: 99% (6K immediate / 6K locks)
[OK] Binlog cache memory access: 100.00% ( 1618 Memory / 1618 Total)
-------- MyISAM Metrics -----------------------------------------------------
[!!] Key buffer used: 18.5% (1M used / 8M cache)
[OK] Key buffer size / total MyISAM indexes: 8.0M/2.4M
[!!] Read Key buffer hit rate: 82.2% (90 cached / 16 reads)
-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 1.3G/29.0M
[!!] InnoDB buffer pool instances: 8
[!!] InnoDB Used buffer: 1.32% (1139 used/ 86584 total)
[OK] InnoDB Read buffer efficiency: 99.86% (713109 hits/ 714137 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 4915 writes)
Since this is a test server we are running the tests on a t2.small

1.3G buffer_pool in 2GB of RAM? This probably leads to lots of swapping, which is really bad for performance.
For 2GB of RAM and only 29M of data, let's set innodb_buffer_pool_size = 100M. That should enough for now, and safe for later (as the data grows).
(The recommendation of 70% or 80% only applies for machines with at least 4GB of RAM.)
Fix that. If the problem continues, then update the question with newer values, plus SHOW CREATE TABLE for the table(s) involved.

An AUTO-INC lock is a special table-level lock taken by transactions inserting into tables with AUTO_INCREMENT columns. In the simplest case, if one transaction is inserting values into the table, any other transactions must wait to do their own inserts into that table, so that rows inserted by the first transaction receive consecutive primary key values.
The innodb_autoinc_lock_mode configuration option controls the algorithm used for auto-increment locking. It allows you to choose how to trade off between predictable sequences of auto-increment values and maximum concurrency for insert operations.
The permissible values are 0, 1, or 2, for “traditional”, “consecutive”, or “interleaved” lock mode
innodb_autoinc_lock_mode = 2 (“interleaved” lock mode)
In this lock mode, no “INSERT-like” statements use the table-level AUTO-INC lock, and multiple statements can execute at the same time. This is the fastest and most scalable lock mode, but it is not safe when using statement-based replication or recovery scenarios when SQL statements are replayed from the binary log.
In this lock mode, auto-increment values are guaranteed to be unique and monotonically increasing across all concurrently executing “INSERT-like” statements. However, because multiple statements can be generating numbers at the same time (that is, allocation of numbers is interleaved across statements), the values generated for the rows inserted by any given statement may not be consecutive.
If the only statements executing are “simple inserts” where the number of rows to be inserted is known ahead of time, there will be no gaps in the numbers generated for a single statement, except for “mixed-mode inserts”. However, when “bulk inserts” are executed, there may be gaps in the auto-increment values assigned by any given statement.
Refer the source Locking and modify the configuration based on your requirement.

Error establishing a database connection EC2 Amazon

I hope you can help me. I can not stand having to keep restarting my ec2 instance on Amazon.
I have two wordpress sites hosted there. My sites have always worked well until two months ago, one of them started having this problem. I tried all ways pack up, and the only solution was to reconfigure.
Now that all was right with the two. The second site started the same problem. I think Amazon is clowning me.
I am using a free micro instance. If anyone knows what the problem is, please help me!

Your issue will be the limited memory that is allocated to the T1 Micro instances in EC2. I'm assuming you are using ANI Linux in this case and if an alternate version of Linux is used then you may have different locations for your log and config files.
Make sure you are the root user.
Have a look at your MySQL logs in the following location:
/var/log/mysqld.log
If you see repeated instances of the following it's pretty certain that the 0.6GB of memory allocated to the micro instance is not cutting it.
150714 22:13:33 InnoDB: Initializing buffer pool, size = 12.0M
InnoDB: mmap(12877824 bytes) failed; errno 12
150714 22:13:33 InnoDB: Completed initialization of buffer pool
150714 22:13:33 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150714 22:13:33 [ERROR] Plugin 'InnoDB' init function returned error.
150714 22:13:33 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150714 22:13:33 [ERROR] Unknown/unsupported storage engine: InnoDB
150714 22:13:33 [ERROR] Aborting
You will notice in the log excerpt above that my buffer pool size is set to 12MB. This can be configured by adding the line innodb_buffer_pool_size = 12M to your MySQL config file /etc/my.cnf.
A pretty good way to deal with InnoDB chewing up your memory is to create a swap file.
Start by checking the status of your memory:
free -m
You will most probably see that your swap is not doing much:
total used free shared buffers cached
Mem: 592 574 17 0 15 235
-/+ buffers/cache: 323 268
Swap: 0 0 0
To start ensure you are logged in as the root user and run the following command:
dd if=/dev/zero of=/swapfile bs=1M count=1024
Wait for a bit as the command is not verbose but you should see the following response after about 15 seconds when the process is complete:
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 31.505 s, 34.1 MB/s
Next set up the swapspace with:
mkswap /swapfile
Now set up the swap event:
swapon /swapfile
If you get a permissions response you can ignore it or address the swap file by changing the permissions to 600 with the chmod command.
chmod 600 /swapfile
Now add the following line to /etc/fstab to create the swap spaces on server start:
/swapfile swap swap defaults 0 0
Restart your MySQL instance:
service mysqld restart
Finally check to see if your swap file is working correctly with the free -m command.
You should see something like:
total used free shared buffers cached
Mem: 592 575 16 0 16 235
-/+ buffers/cache: 323 269
Swap: 1023 0 1023
Hope this helps.

Develop Reference

r css asp.net wordpress firebase qt symfony nginx http apache-flex