How should mysql my.cnf settings be? Innodb Mariadb 10 VestaCP - mariadb

Question

My.cnf
[mysql]
port = 3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1
datadir = /var/lib/mysql
max_allowed_packet = 1024M
max_connect_errors = 1000000
pid-file=/var/lib/mysql/mysql.pid
port = 3306
skip_name_resolve
socket=/var/lib/mysql/mysql.sock
tmpdir = /tmp
sql_mode = ""
thread_handling = pool-of-threads
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_size = 36G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 1G
innodb_sort_buffer_size = 4M # UPD - Defines how much data is read into memory for sorting operations before writing to disk (default is 1M / max is 64M)
innodb_stats_on_metadata = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_io_capacity = 2000 # Depends on the storage tech - use 2000 for SSD, more for NVMe
innodb_io_capacity_max = 4000 # Usually double the value of innodb_io_capacity
# === Connection Settings ===
max_connections = 100 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# === Buffer Settings ===
join_buffer_size = 4M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 60000 # UPD
max_heap_table_size = 128M # Increase to 256M or 512M if you have lots of temporary tables because of missing indices in JOINs
tmp_table_size = 128M # Use same value as max_heap_table_size
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Binary Logging ===
disable_log_bin = 1
# === Error & Slow Query Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 0 # Disabled on production
long_query_time = 5
slow_query_log = 0 # Disabled on production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
quick
quote_names
max_allowed_packet = 1024M
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
My ram & cpu
[root#localhost ~]# cat /proc/cpuinfo | grep processor | wc -l
24
[root#localhost ~]# free -m
total used free shared buff/cache available
Mem: 48125 6136 12446 50 29541 41490
Swap: 0 0 0
mysqltuner
[root#localhost ~]# ./mysqltuner.pl
>> MySQLTuner 1.9.8
* Jean-Marie Renouard <jmrenouard#gmail.com>
* Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.pl/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[!!] Your MySQL version 10.8.3-MariaDB is EOL software! Upgrade soon!
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/mysql_error.log exists
[--] Log file: /var/lib/mysql/mysql_error.log(70K)
[OK] Log file /var/lib/mysql/mysql_error.log is not empty
[OK] Log file /var/lib/mysql/mysql_error.log is smaller than 32 Mb
[OK] Log file /var/lib/mysql/mysql_error.log is readable.
[!!] /var/lib/mysql/mysql_error.log contains 339 warning(s).
[!!] /var/lib/mysql/mysql_error.log contains 51 error(s).
[--] 2 start(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:05 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 2) 2022-06-17 21:30:12 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 1 shutdown(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:04 0 [Note] /usr/sbin/mariadbd: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in Aria tables: 32.0K (Tables: 1)
[--] Data in InnoDB tables: 1.7G (Tables: 845)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Views Metrics -----------------------------------------------------------------------------
-------- Triggers Metrics --------------------------------------------------------------------------
-------- Routines Metrics --------------------------------------------------------------------------
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 4d 6h 58m 28s (166M q [448.474 qps], 937K conn, TX: 203G, RX: 12G)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory : 47.0G
[--] Max MySQL memory : 137.9G
[--] Other process memory: 0B
[--] Total buffers: 36.4G global + 1.0G per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 93.2G (198.36% of installed RAM)
[!!] Maximum possible memory usage: 137.9G (293.37% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (53/166M)
[OK] Highest usage of available connections: 56% (56/100)
[OK] Aborted connections: 0.01% (107/937871)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 874K sorts)
[!!] Joins performed without indexes: 46708
[OK] Temporary tables created on disk: 2% (6K on disk / 305K total)
[--] Thread cache not used with thread pool enabled
[OK] Table cache hit rate: 99% (77M hits / 77M requests)
[OK] table_definition_cache(40000) is upper than number of tables(1137)
[OK] Open file limit used: 0% (29/32K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)
-------- Performance schema ------------------------------------------------------------------------
[!!] Performance_schema should be activated.
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 24 thread(s).
[--] Using default value is good enough for your version (10.8.3-MariaDB)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] No MyISAM table(s) detected ....
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 36.0G/1.7G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (2.77777777777778 %): 1.0G * 1/36.0G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 64 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (36551426226 hits/ 36551507601 total)
[OK] InnoDB Write log efficiency: 90.84% (7736631 hits/ 8516998 total)
[OK] InnoDB log waits: 0.00% (0 waits / 780367 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/344.0K
[!!] Aria pagecache hit rate: 82.5% (38K cached / 6K reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
You are using n unsupported version for production environments
Upgrade as soon as possible to a supported version !
Check warning line(s) in /var/lib/mysql/mysql_error.log file
Check error line(s) in /var/lib/mysql/mysql_error.log file
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
Performance schema should be activated for better diagnostics
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this:
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
join_buffer_size (> 4.0M, or always use indexes with JOINs)
performance_schema=ON
innodb_log_file_size should be (=9G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
[root#localhost ~]#
Tuning Primer
[root#localhost ~]# ./tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.8.3-MariaDB x86_64
Uptime = 4 days 9 hrs 29 min 40 sec
Avg. qps = 454
Total Questions = 172718132
Threads Connected = 33
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.8/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 5.000000 sec.
You have 53 out of 172718244 that take longer than 5.000000 sec. to complete
Your long_query_time seems to be fine
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.8/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 0
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 100
Current threads_connected = 33
Historic max_used_connections = 56
The number of used connections is 56% of the configured maximum.
Your max_connections variable seems to be fine.
No InnoDB Support Enabled!
MEMORY USAGE
Max Memory Ever Allocated : 36.97 G
Configured Max Per-thread Buffers : 1.48 G
Configured Max Global Buffers : 36.14 G
Configured Max Memory Limit : 37.62 G
Physical Memory : 46.99 G
Max memory limit seem to be within acceptable norms
KEY BUFFER
No key reads?!
Seriously look into using some indexes
Current MyISAM index space = 0 bytes
Current key_buffer_size = 128 M
Key cache miss rate is 1 : 0
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 1 M
Current query_cache_used = 16 K
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 1.65 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
Current join_buffer_size = 4.00 M
You have had 48458 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 16319 tables
Current table_definition_cache = 40000 tables
You have a total of 957 tables
You have 1124 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 315112 temp tables, 2% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 155 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 0 : 172720142
Your table locking seems to be fine
[root#localhost ~]#
I can allocate 40gb ram to mariadb server.
I need 8gb ram. We can use all 24 processors.
Please help me how to do the settings.
The site will be used for a web-based game.
The tables were all using MyISAM.
I converted all of them to innodb and it works fine.
If you need additional information, let me know in the comments.

Answer 1

These seem excessive:
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
Will you have thousands of tables? Why? Even if you do have that many, these are caches, hence they don't have to be big enough to handle everything.
For better performance turn on the slowlog with a low value such as long_query_time = 1. After a few hours, use pt-query-digest. More on the SlowLog
For a deeper analysis of the settings: http://mysql.rjweb.org/doc.php/mysql_analysis#tuning

Answer 2

Rate Per Second = RPS
Suggestions to consider for your my.cnf [mysqld] section to improve performance
read_rnd_buffer_size=32K # from 4M to reduce handler_read_rnd_next RPS of 20,873
read_buffer_size=1M # from 3M to reduce handler_read_next RPS of 20,918
innodb_buffer_pool_size=4G # from 36G because you only have 1.7G of data and indexes
net_buffer_length=96K # from 16K to accommodate sending 700MB data per hr
Let us know how your system is performing in a few days, please. View profile for contact information and we do have FREE Utility Scripts to assist with performance tuning.
This is only the beginning of improving performance for your installation. As you get deeper into the environment you will find many opportunities to improve response time and reduce system overhead. We are available to assist.