I have three node Galera cluster setup with MariaDB, I see a repeating message for one of the node where it desyncs itself from cluster and resyncs within few seconds. This message is repeating every hour. Though it is just a notice, I would like to understand the reason. Configuration on all the servers is same. Does anyone has idea why this happens ? or if anybody faced similar problem of this sort. Below are the details.
MariaDB version 10.3.12
Galera version: 25.3.23-1
OS: RHEL 7.4
State transfer mechanism: default
Below is my Galera configuration on the node which has above mentioned observation.
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options="gcache.size=300M; gcache.page_size=300M; pc.ignore_sb=false; pc.ignore_quorum=false"
#wsrep_cluster_address defines members of the cluster
wsrep_cluster_address=gcomm://192.168.1.1,192.168.1.2,192.168.1.3
wsrep_cluster_name="mariadb-cluster"
#wsrep_node_address passed to replication pluggin
wsrep_node_address=192.168.1.1
#wsrep_node_incoming_address is used for replicationclient connections
wsrep_node_incoming_address=192.168.1.1
wsrep_debug=OFF
#
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
Error Message is as given below
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 0 [Note] WSREP: Member 2.0 (maria1) desyncs itself from group
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 7430165)
Aug 28 16:38:42 maria1 mysqld: 2020-08-28 16:38:42 452 [Note] WSREP: Provider paused at 08e2494b-12ff-11e9-9785-4fa29d91e9c5:7430165 (22186)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 452 [Note] WSREP: resuming provider at 22186
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 452 [Note] WSREP: Provider resumed.
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Member 2.0 (maria1) resyncs itself to group
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 7430165)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Member 2.0 (maria1) synced with group.
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 7430165)
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 2 [Note] WSREP: Synchronized with group, ready for connections
Aug 28 16:38:50 maria1 mysqld: 2020-08-28 16:38:50 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
I could find out, I missed the fact that periodic backup was running on the node via Mariabackup tool. During that time node de-syncs itself from cluster for consistency.
I've VPS on Digitalocean Centos 7.
But it gives an error when I try to share my wordpress posts.
Here's my logs.
170302 09:00:03 mysqld_safe Number of processes running now: 0
170302 09:00:03 mysqld_safe mysqld restarted
170302 9:00:03 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 31156 ...
170302 9:00:03 InnoDB: The InnoDB memory heap is disabled
170302 9:00:03 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170302 9:00:03 InnoDB: Compressed tables use zlib 1.2.7
170302 9:00:03 InnoDB: Using Linux native AIO
170302 9:00:03 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
170302 9:00:03 InnoDB: Completed initialization of buffer pool
170302 9:00:03 InnoDB: Fatal error: cannot allocate memory for the buffer pool
170302 9:00:03 [ERROR] Plugin 'InnoDB' init function returned error.
170302 9:00:03 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
170302 9:00:03 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
170302 9:00:03 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
170302 9:00:03 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
170302 9:00:03 [Note] Plugin 'FEEDBACK' is disabled.
170302 9:00:03 [ERROR] Unknown/unsupported storage engine: InnoDB
170302 9:00:03 [ERROR] Aborting
170302 9:00:03 [Note] /usr/libexec/mysqld: Shutdown complete
170302 09:00:03 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
I recieve this error message when attempting to install Wordpress:
mamp has quit because of an unknown error. Please check the mamp pro error log inside the logs directory for more information.
Also, when attempting to change the MySQL password under the MySQL tab I recieve this error message:
mamp could not update mysql password. please check if mysql is running, check your configuration and try again.
Here are my error logs:
----apache----
[Thu Aug 11 14:54:25 2016] [warn] pid file C:/MAMP/bin/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Aug 11 14:54:25 2016] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 11 14:54:25 2016] [notice] Digest: done
[Thu Aug 11 14:54:26 2016] [notice] Apache/2.2.31 (Win32) DAV/2 mod_ssl/2.2.31 OpenSSL/1.0.2e mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.6 PHP/7.0.6 mod_perl/2.0.8 Perl/v5.16.3 configured -- resuming normal operations
[Thu Aug 11 14:54:26 2016] [notice] Server built: May 6 2016 10:19:53
[Thu Aug 11 14:54:26 2016] [notice] Parent: Created child process 11620
[Thu Aug 11 14:54:27 2016] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 11 14:54:27 2016] [notice] Digest: done
[Thu Aug 11 14:54:28 2016] [notice] Child 11620: Child process is running
[Thu Aug 11 14:54:28 2016] [notice] Child 11620: Acquired the start mutex.
[Thu Aug 11 14:54:28 2016] [notice] Child 11620: Starting 64 worker threads.
[Thu Aug 11 14:54:28 2016] [notice] Child 11620: Starting thread to listen on port 8888.
[Thu Aug 11 14:55:34 2016] [warn] pid file C:/MAMP/bin/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Aug 11 14:55:34 2016] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 11 14:55:34 2016] [notice] Digest: done
[Thu Aug 11 14:55:35 2016] [notice] Apache/2.2.31 (Win32) DAV/2 mod_ssl/2.2.31 OpenSSL/1.0.2e mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.6 PHP/7.0.6 mod_perl/2.0.8 Perl/v5.16.3 configured -- resuming normal operations
[Thu Aug 11 14:55:35 2016] [notice] Server built: May 6 2016 10:19:53
[Thu Aug 11 14:55:35 2016] [notice] Parent: Created child process 4176
[Thu Aug 11 14:55:36 2016] [notice] Digest: generating secret for digest authentication ...
[Thu Aug 11 14:55:36 2016] [notice] Digest: done
[Thu Aug 11 14:55:37 2016] [notice] Child 4176: Child process is running
[Thu Aug 11 14:55:37 2016] [notice] Child 4176: Acquired the start mutex.
[Thu Aug 11 14:55:37 2016] [notice] Child 4176: Starting 64 worker threads.
[Thu Aug 11 14:55:37 2016] [notice] Child 4176: Starting thread to listen on port 80.
--------
----mysql----
160811 14:54:28 [Note] Plugin 'FEDERATED' is disabled.
160811 14:54:28 InnoDB: The InnoDB memory heap is disabled
160811 14:54:28 InnoDB: Mutexes and rw_locks use Windows interlocked functions
160811 14:54:28 InnoDB: Compressed tables use zlib 1.2.3
160811 14:54:28 InnoDB: Initializing buffer pool, size = 128.0M
160811 14:54:28 InnoDB: Completed initialization of buffer pool
160811 14:54:28 InnoDB: highest supported file format is Barracuda.
160811 14:54:28 InnoDB: Waiting for the background threads to start
160811 14:54:29 InnoDB: 5.5.49 started; log sequence number 1595675
160811 14:54:29 [Note] Server hostname (bind-address): '0.0.0.0'; port: 8889
160811 14:54:29 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160811 14:54:29 [Note] Server socket created on IP: '0.0.0.0'.
160811 14:54:29 [Note] Event Scheduler: Loaded 0 events
160811 14:54:29 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: ready for connections.
Version: '5.5.49-log' socket: '' port: 8889 MySQL Community Server (GPL)
160811 14:55:30 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: Normal shutdown
160811 14:55:30 [Note] Event Scheduler: Purging the queue. 0 events
160811 14:55:30 InnoDB: Starting shutdown...
160811 14:55:31 InnoDB: Shutdown completed; log sequence number 1595675
160811 14:55:31 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: Shutdown complete
160811 14:55:37 [Note] Plugin 'FEDERATED' is disabled.
160811 14:55:37 InnoDB: The InnoDB memory heap is disabled
160811 14:55:37 InnoDB: Mutexes and rw_locks use Windows interlocked functions
160811 14:55:37 InnoDB: Compressed tables use zlib 1.2.3
160811 14:55:37 InnoDB: Initializing buffer pool, size = 128.0M
160811 14:55:37 InnoDB: Completed initialization of buffer pool
160811 14:55:37 InnoDB: highest supported file format is Barracuda.
160811 14:55:37 InnoDB: Waiting for the background threads to start
160811 14:55:38 InnoDB: 5.5.49 started; log sequence number 1595675
160811 14:55:38 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
160811 14:55:38 [Note] - '0.0.0.0' resolves to '0.0.0.0';
160811 14:55:38 [Note] Server socket created on IP: '0.0.0.0'.
160811 14:55:38 [Note] Event Scheduler: Loaded 0 events
160811 14:55:38 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: ready for connections.
Version: '5.5.49-log' socket: '' port: 3306 MySQL Community Server (GPL)
160811 15:01:46 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: Normal shutdown
160811 15:01:46 [Note] Event Scheduler: Purging the queue. 0 events
160811 15:01:46 InnoDB: Starting shutdown...
160811 15:01:47 InnoDB: Shutdown completed; log sequence number 1595675
160811 15:01:47 [Note] C:\MAMP\bin\mysql\bin\mysqld.exe: Shutdown complete
--------
If you would be able to provide me with a solution to this, it would be much appreciated.
Event if MAMP quits, sometimes MySQL is running in the background. Could you kill all MySQL background process and try again?
Quit MAMP
Open the terminal and type: killall -9 mysqld
Restart MAMP
I have installed ubuntu 16.04 on a VMware VM. Then I added the repo from mariadb website and I installed the latest version of mariadb 10.1.
The installation works fine
I have then created a file
sudo nano /etc/mysql/conf.d/cluster.cnf
with the following
[mysqld]
# Cluster node configurations
wsrep_cluster_address="gcomm://20.0.1.51"
wsrep_node_address="20.0.1.51"
innodb_buffer_pool_size=800M
# Mandatory settings to enable Galera
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
binlog_format=ROW
default-storage-engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
query_cache_size=0
bind-address=0.0.0.0
# Galera synchronisation configuration
wsrep_sst_method=rsync
I want only to bootstrap the first node and then add new nodes
So I run
sudo service mysql bootstrap
But I get this error
Jul 03 02:38:07 db1 mysqld[14779]: at gcomm/src/pc.cpp:connect():162
Jul 03 02:38:07 db1 mysqld[14779]: 2016-07-03 2:38:07 140418873596160 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed ou
Jul 03 02:38:07 db1 mysqld[14779]: 2016-07-03 2:38:07 140418873596160 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel 'my_wsrep_cluster' at 'gcomm://20.0.1.51': -11
Jul 03 02:38:07 db1 mysqld[14779]: 2016-07-03 2:38:07 140418873596160 [ERROR] WSREP: gcs connect failed: Connection timed out
Jul 03 02:38:07 db1 mysqld[14779]: 2016-07-03 2:38:07 140418873596160 [ERROR] WSREP: wsrep::connect(gcomm://20.0.1.51) failed: 7
Jul 03 02:38:07 db1 mysqld[14779]: 2016-07-03 2:38:07 140418873596160 [ERROR] Aborting
Jul 03 02:38:08 db1 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jul 03 02:38:08 db1 systemd[1]: Failed to start MariaDB database server.
Jul 03 02:38:08 db1 systemd[1]: mariadb.service: Unit entered failed state.
Jul 03 02:38:08 db1 systemd[1]: mariadb.service: Failed with result 'exit-code'.
What am I doing wrong??
Its the systemd thing. On systems that has systemd, one should use galera_new_cluster to bootstrap a new cluster.
https://mariadb.com/kb/en/mariadb/systemd/
I am trying to setup mariadb galera cluster on Debian Wheezy 7.5. I have found many different instructions, all a bit different, but none have worked thus far.
I am trying to setup a two node cluster.
On the primary node, I am using the default my.cnf, with these additional settings in conf.d/cluster.cnf:
[mysqld]
#mysql settings
bind-address=10.1.1.139
query_cache_size=0
query_cache_type=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
#galery settings
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://10.1.1.139,10.1.1.140"
wsrep_sst_method=rsync
wsrep_cluster_name="sql_cluster"
wsrep_node_incoming_address=10.1.1.139
wsrep_sst_receive_address=10.1.1.139
wsrep_sst_auth=cluster:password
wsrep_node_address='10.1.1.139'
wsrep_node_name='sql1'
wsrep_on=ON
Created the cluster user, gave that user all the required permissions, started the server successfully with
service mysql start --wsrep-new-cluster
The cluster starts up, I can see cluster_size=1;
On the second node, I am using the default my.cnf, with these additional settings in conf.d/cluster.cnf:
[mysqld]
#mysql settings
bind-address=10.1.1.140
query_cache_size=0
query_cache_type=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
#galery settings
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://10.1.1.139,10.1.1.140"
wsrep_sst_method=rsync
wsrep_cluster_name="sql_cluster"
wsrep_node_incoming_address=10.1.1.140
wsrep_sst_receive_address=10.1.1.140
wsrep_sst_auth=cluster:password
wsrep_node_address='10.1.1.140'
wsrep_node_name='sql1'
wsrep_on=ON
I also replaced debian.cnf on the secondary node with the one from the primary node as per this suggestion:
http://docs.openstack.org/high-availability-guide/content/ha-aa-db-mysql-galera.html and granted the appropriate permissions (this was also suggested in other places, don't have the links right now).
Contents of debian.cnf on both nodes:
[client]
host = localhost
user = debian-sys-maint
password = <password>
socket = /var/run/mysqld/mysqld.sock
[mysql_upgrade]
host = localhost
user = debian-sys-maint
password = <password>
socket = /var/run/mysqld/mysqld.sock
basedir = /usr
When I try to start the second node with:
service mysql start
It fails, and I get this in /var/log/syslog:
May 7 19:45:30 ns514282 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
May 7 19:45:30 ns514282 mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.s6Uwyc' --pid-file='/var/lib/mysql/ns514282.ip-167-114-159.net-recover.pid'
May 7 19:45:33 ns514282 mysqld_safe: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Read nil XID from storage engines, skipping position init
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: wsrep_load(): Galera 3.9(rXXXX) by Codership Oy <info#codership.com> loaded successfully.
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: CRC-32C: using hardware acceleration.
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Passing config to GCS: base_host = 10.1.1.142; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npv
May 7 19:45:33 ns514282 mysqld: o = false; pc.recovery
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Service thread queue flushed.
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: wsrep_sst_grab()
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Start replication
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: protonet asio version 0
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: Using CRC-32C for message checksums.
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: backend: asio
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: restore pc from disk successfully
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: GMCast version 0
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: EVS version 0
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: gcomm: connecting to group 'bfm_cluster', peer '10.1.1.141:,10.1.1.142:'
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Warning] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') address 'tcp://10.1.1.142:4567' points to own listening address, blacklisting
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') address 'tcp://10.1.1.142:4567' pointing to uuid 66b559a2 is blacklisted, skipping
May 7 19:45:33 ns514282 mysqld: 150507 19:45:33 [Note] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: declaring dc2b490d at tcp://10.1.1.141:4567 stable
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: re-bootstrapping prim from partitioned components
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: view(view_id(PRIM,66b559a2,12) memb {
May 7 19:45:34 ns514282 mysqld: #01166b559a2,0
May 7 19:45:34 ns514282 mysqld: #011dc2b490d,0
May 7 19:45:34 ns514282 mysqld: } joined {
May 7 19:45:34 ns514282 mysqld: } left {
May 7 19:45:34 ns514282 mysqld: } partitioned {
May 7 19:45:34 ns514282 mysqld: })
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: save pc into disk
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: clear restored view
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: gcomm: connected
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Opened channel 'bfm_cluster'
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Waiting for SST to complete.
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 279db665-f513-11e4-9149-aa318d13ebc4
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: STATE EXCHANGE: sent state msg: 279db665-f513-11e4-9149-aa318d13ebc4
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: STATE EXCHANGE: got state msg: 279db665-f513-11e4-9149-aa318d13ebc4 from 0 (sql1)
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: STATE EXCHANGE: got state msg: 279db665-f513-11e4-9149-aa318d13ebc4 from 1 (sql3)
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Warning] WSREP: Quorum: No node with complete state:
May 7 19:45:34 ns514282 mysqld:
May 7 19:45:34 ns514282 mysqld:
May 7 19:45:34 ns514282 mysqld: #011Version : 3
May 7 19:45:34 ns514282 mysqld: #011Flags : 0x1
May 7 19:45:34 ns514282 mysqld: #011Protocols : 0 / 7 / 3
May 7 19:45:34 ns514282 mysqld: #011State : NON-PRIMARY
May 7 19:45:34 ns514282 mysqld: #011Prim state : NON-PRIMARY
May 7 19:45:34 ns514282 mysqld: #011Prim UUID : 00000000-0000-0000-0000-000000000000
May 7 19:45:34 ns514282 mysqld: #011Prim seqno : -1
May 7 19:45:34 ns514282 mysqld: #011First seqno : -1
May 7 19:45:34 ns514282 mysqld: #011Last seqno : -1
May 7 19:45:34 ns514282 mysqld: #011Prim JOINED : 0
May 7 19:45:34 ns514282 mysqld: #011State UUID : 279db665-f513-11e4-9149-aa318d13ebc4
May 7 19:45:34 ns514282 mysqld: #011Group UUID : 00000000-0000-0000-0000-000000000000
May 7 19:45:34 ns514282 mysqld: #011Name : 'sql1'
May 7 19:45:34 ns514282 mysqld: #011Incoming addr: '10.1.1.142:3306'
May 7 19:45:34 ns514282 mysqld:
May 7 19:45:34 ns514282 mysqld: #011Version : 3
May 7 19:45:34 ns514282 mysqld: #011Flags : 0x2
May 7 19:45:34 ns514282 mysqld: #011Protocols : 0 / 7 / 3
May 7 19:45:34 ns514282 mysqld: #011State : NON-PRIMARY
May 7 19:45:34 ns514282 mysqld: #011Prim state : SYNCED
May 7 19:45:34 ns514282 mysqld: #011Prim UUID : b65a0277-f50f-11e4-a916-dbeff5b65a2e
May 7 19:45:34 ns514282 mysqld: #011Prim seqno : 8
May 7 19:45:34 ns514282 mysqld: #011First seqno : -1
May 7 19:45:34 ns514282 mysqld: #011Last seqno : 0
May 7 19:45:34 ns514282 mysqld: #011Prim JOINED : 1
May 7 19:45:34 ns514282 mysqld: #011State UUID : 279db665-f513-11e4-9149-aa318d13ebc4
May 7 19:45:34 ns514282 mysqld: #011Group UUID : dc2be55b-f506-11e4-8748-4bd7f3fc795c
May 7 19:45:34 ns514282 mysqld: #011Name : 'sql3'
May 7 19:45:34 ns514282 mysqld: #011Incoming addr: '10.1.1.141:3306'
May 7 19:45:34 ns514282 mysqld:
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Full re-merge of primary b65a0277-f50f-11e4-a916-dbeff5b65a2e found: 1 of 1.
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Quorum results:
May 7 19:45:34 ns514282 mysqld: #011version = 3,
May 7 19:45:34 ns514282 mysqld: #011component = PRIMARY,
May 7 19:45:34 ns514282 mysqld: #011conf_id = 8,
May 7 19:45:34 ns514282 mysqld: #011members = 1/2 (joined/total),
May 7 19:45:34 ns514282 mysqld: #011act_id = 0,
May 7 19:45:34 ns514282 mysqld: #011last_appl. = -1,
May 7 19:45:34 ns514282 mysqld: #011protocols = 0/7/3 (gcs/repl/appl),
May 7 19:45:34 ns514282 mysqld: #011group UUID = dc2be55b-f506-11e4-8748-4bd7f3fc795c
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Flow-control interval: [23, 23]
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: State transfer required:
May 7 19:45:34 ns514282 mysqld: #011Group state: dc2be55b-f506-11e4-8748-4bd7f3fc795c:0
May 7 19:45:34 ns514282 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: New cluster view: global state: dc2be55b-f506-11e4-8748-4bd7f3fc795c:0, view# 9: Primary, number of nodes: 2, my index: 0, protocol version 3
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Warning] WSREP: Gap in state sequence. Need state transfer.
May 7 19:45:34 ns514282 mysqld: 150507 19:45:34 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.1.1.142' --auth 'cluster:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '12278' --binlog '/var/log/mysql/mariadb-bin' '
May 7 19:45:34 ns514282 rsyncd[12428]: rsyncd version 3.0.9 starting, listening on port 4444
May 7 19:45:37 ns514282 mysqld: 150507 19:45:37 [Note] WSREP: (66b559a2, 'tcp://0.0.0.0:4567') turning message relay requesting off
May 7 19:45:47 ns514282 /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing
May 7 19:45:57 ns514282 /usr/sbin/irqbalance: Load average increasing, re-enabling all cpus for irq balancing
May 7 19:46:02 ns514282 /USR/SBIN/CRON[16491]: (root) CMD (/usr/local/rtm/bin/rtm 50 > /dev/null 2> /dev/null)
May 7 19:46:03 ns514282 /etc/init.d/mysql[16711]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
May 7 19:46:03 ns514282 /etc/init.d/mysql[16711]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
May 7 19:46:03 ns514282 /etc/init.d/mysql[16711]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
May 7 19:46:03 ns514282 /etc/init.d/mysql[16711]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
This question has countless threads all over the internet. Some with no answers. Some of the ones that do have answers
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) - my disk space is not full
Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' - no answer. But as per comments, the mysql.sock does exist and has mysql.mysql ownership.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) - server is installed, again socket is present in the right location
I have also read that this might be a permissions issue on /var/run/mysqld, but I have checked this and gave it mysql.mysql ownership.
If nothing else, this is an attempt to revive this issue. Any direction is really appreciated.
Thank you,
Update: my.cnf for both nodes. It is the default my.cnf. The only change is commenting out the bind-address=127.0.0.1 line.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
# bind-address = 127.0.0.1
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size = 16M
tmp_table_size = 32M
max_heap_table_size = 32M
myisam_recover = BACKUP
key_buffer_size = 128M
table_open_cache = 400
myisam_sort_buffer_size = 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
query_cache_limit = 128K
query_cache_size = 64M
log_warnings = 2
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
log_slow_verbosity = query_plan
log_bin = /var/log/mysql/mariadb-bin
log_bin_index = /var/log/mysql/mariadb-bin.index
expire_logs_days = 10
max_binlog_size = 100M
default_storage_engine = InnoDB
innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/
Update
Also, I tested, and If I attempt to start the node regularly on its own, (without the cluster, with no extra settings, just defaults) it works.
Is the "wsrep_node_name" correct?
[mysqld]
#mysql settings
bind-address=10.1.1.140
query_cache_size=0
query_cache_type=0
binlog_format=ROW
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_doublewrite=1
#galery settings
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://10.1.1.139,10.1.1.140"
wsrep_sst_method=rsync
wsrep_cluster_name="sql_cluster"
wsrep_node_incoming_address=10.1.1.140
wsrep_sst_receive_address=10.1.1.140
wsrep_sst_auth=cluster:password
wsrep_node_address='10.1.1.140'
wsrep_node_name='sql1' <== ???
wsrep_on=ON
I recommend restarting the server.
if you keep having problems ...
backup, uninstall and install again ...
with me it worked like this ...
Verify that all IPS used are BIND compliant