A galera node went down and started reporting error innodb as unknown or unsupported engine - innodb

One of my node was down and it was also the donor for the node A, so from node A config I removed the down node as donor. This is so that node A should not have to wait for the node to come up.
As the node that originally went down was on slow storage and slow network connection I decided to remove it. As I removed it and restarted the cluster even the node A didn't come up, on restart it started reporting error:
● mariadb.service - MariaDB 10.1.47 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-12-13 22:06:50 GMT; 12h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 32653 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 17439 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_STA
Process: 17436 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 17432 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 32653 (code=exited, status=1/FAILURE)
CPU: 801ms
Dec 13 22:06:50 rockpi sh[17439]: InnoDB: http://dev.mysql.com/doc/refman/5.6/en/error-creating-innodb.html
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Plugin 'InnoDB' init function returned error.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [Note] Plugin 'FEEDBACK' is disabled.
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Unknown/unsupported storage engine: innodb
Dec 13 22:06:50 rockpi sh[17439]: 2020-12-13 22:06:50 548039303184 [ERROR] Aborting'
Dec 13 22:06:50 rockpi systemd[1]: mariadb.service: Control process exited, code=exited status=1
Dec 13 22:06:50 rockpi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 13 22:06:50 rockpi systemd[1]: Failed to start MariaDB 10.1.47 database server.
Why it started reporting Unknown/unsupported storage engine innodb now when it was working all this time?
Incidentally, I used another third node to bootstrap (start Galera_new_cluster).

Related

mariadb won't start suddenly

I have some trouble with my MariaDB Server, it was working ok but it won't start anymore. When I try to start the server it failed:
root#vps45223599:/var/log# /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
failed!
root#vps45223599:/var/log# systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.41 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-01-10 21:20:58 UTC; 1min 15s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1349 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 1274 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR
Process: 1272 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1271 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1349 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Jan 10 21:20:55 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
Jan 10 21:20:55 vps45223599.local mysqld[1349]: 2023-01-10 21:20:55 140599894461824 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1349 ...
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:20:58 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:43 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:15:43 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:15:43 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:43 vps45223599.local sudo[1040]: pam_unix(sudo:session): session closed for user root
Jan 10 21:15:50 vps45223599.local sudo[1146]: root : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/bin/systemctl start mariadb
Jan 10 21:15:50 vps45223599.local sudo[1146]: pam_unix(sudo:session): session opened for user root by cbarca(uid=0)
Jan 10 21:15:50 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:15:50 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:15:50 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Jan 10 21:15:51 vps45223599.local mysqld[1227]: 2023-01-10 21:15:51 139968422329728 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1227 ...
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:15:54 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:54 vps45223599.local sudo[1146]: pam_unix(sudo:session): session closed for user root
Jan 10 21:20:55 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:20:55 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:20:55 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Jan 10 21:20:55 vps45223599.local mysqld[1349]: 2023-01-10 21:20:55 140599894461824 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1349 ...
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:20:58 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
I don't know what happened. I also try
mysqlcheck
root#vps45223599:/var/log# mysqlcheck --all-databases -p
Enter password:
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") when trying to connect
I don't know what else I should try, can anyone helpe me, please?
Cheers
As #obe said this is likely a proxmox or whatever is instigating systemd without sufficient privileges is the issue.
The devices.allow error is probably requested by the PrivateDevices=true (seems confirmed based on MDEV-13207 which failed to provide more info) aspect of the systemd service file for MariaDB.
PrivateDevices=true allows:
/dev/null
/dev/zero
/dev/random
Based on this answer for a different device, doing the equivalent for these devices would be:
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:3 rwm'
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:8 rwm'
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:5 rwm'
Major/minor device code determined by:
$ ls -la /dev/zero /dev/random /dev/null
crw-rw-rw-. 1 root root 1, 3 Jan 8 22:20 /dev/null
crw-rw-rw-. 1 root root 1, 8 Jan 8 22:20 /dev/random
crw-rw-rw-. 1 root root 1, 5 Jan 8 22:20 /dev/zero
Thanks all for the answer.
#danblack Nothing change, suddenly stop working, but reading this forum I found the solution in this thread.
Can't start MariaDB on debian 10
Basically the solution is ...
"Solved it by deleting/renaming the tc.log mv -vi /var/lib/mysql/tc.log /root And restarting the database service mysql restart"
And mariaddb start again.

radiusd.service: Control process exited, code=exited status=1

I am very new to linux and freeradius and want to learn about this for my work, however I encounter some challenges.
I have installed freeradius on Centos 8. I installed it and it worked just fine but after installing mariaDB, issues came out when trying to start the service.
[root#FPM-FreeRad01 ~]# systemctl start radiusd.service
Job for radiusd.service failed because the control process exited with error code.
See "systemctl status radiusd.service" and "journalctl -xe" for details.
[root#FPM-FreeRad01 ~]# systemctl status radiusd.service
â radiusd.service - FreeRADIUS high performance RADIUS server.
Loaded: loaded (/usr/lib/systemd/system/radiusd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2022-06-23 09:13:53 PST; 9s ago
Process: 2900 ExecStart=/usr/sbin/radiusd -d /etc/raddb (code=exited, status=1/FAILURE)
Process: 2897 ExecStartPre=/usr/sbin/radiusd -C (code=exited, status=0/SUCCESS)
Process: 2891 ExecStartPre=/bin/sh /etc/raddb/certs/bootstrap (code=exited, status=0/SUCCESS)
Process: 2889 ExecStartPre=/bin/chown -R radiusd.radiusd /var/run/radiusd (code=exited, status=0/SUCCESS)
Main PID: 1789 (code=exited, status=0/SUCCESS)
Jun 23 09:13:53 FPM-FreeRad01.int.ahinet.ph systemd[1]: Starting FreeRADIUS high performance RADIUS server....
Jun 23 09:13:53 FPM-FreeRad01.int.ahinet.ph systemd[1]: radiusd.service: Control process exited, code=exited status=1
Jun 23 09:13:53 FPM-FreeRad01.int.ahinet.ph systemd[1]: radiusd.service: Failed with result 'exit-code'.
Jun 23 09:13:53 FPM-FreeRad01.int.ahinet.ph systemd[1]: Failed to start FreeRADIUS high performance RADIUS server..
I've been dealing with this since yesterday and I tried searching for other related problems but cannot find some answers so I try asking for help here. Any help would be appreciated.
Thank you

MariaDB server is down after reboot

I needed to know my mariadb version but there's a typo mysqld -version (I installed it via apt, it was working with cli login, but not mysqli). The command jammed and I forced reboot the VPS from the panel. When it went up, I could not start mariadb.
● mariadb.service - MariaDB 10.3.34 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2022-03-12 16:43:54 CET; 23min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 563 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 598 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 628 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 692 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Main PID: 692 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Mar 12 16:43:53 myvps systemd[1]: Starting MariaDB 10.3.34 database server...
Mar 12 16:43:53 myvps mysqld[692]: 2022-03-12 16:43:53 0 [Note] /usr/sbin/mysqld (mysqld 10.3.34-MariaDB-0ubuntu0.20.04.1) starting as process 692 ... Mar 12 16:43:54 myvps systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Mar 12 16:43:54 myvps systemd[1]: mariadb.service: Failed with result 'exit-code'.
Mar 12 16:43:54 myvps systemd[1]: Failed to start MariaDB 10.3.34 database server
I saw this:
[ERROR] InnoDB: Missing MLOG_CHECKPOINT at 1630019 between the checkpoint 1630019 and the end 1630028.
I read this.
https://dba.stackexchange.com/questions/163445/innodb-ignoring-the-redo-log-due-to-missing-mlog-checkpoint
I did this
rm /var/lib/mysql/ib_logfile*
And worked for me.

MariaDB Won't Start After Installing

I'm running an Ubuntu 20.4 server and I've recently removed mysql-server and mysql-client and am trying to get MariaDB up and running. I've installed both mariadb-server and mariadb-client. I stop and then try to restart maria db. I get the general error message:
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
I then run systenctl status mariadb.service and get the below:
mariadb.service - MariaDB 10.6.4 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Thu 2021-09-23 13:26:03 CDT; 8s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 97366 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 97377 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 97379 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] &&>
Process: 97387 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Main PID: 97387 (code=exited, status=1/FAILURE)
Sep 23 13:26:03 notised.xyz systemd[1]: Starting MariaDB 10.6.4 database server...
Sep 23 13:26:03 notised.xyz sh[97384]: [114B blob data]
Sep 23 13:26:03 notised.xyz sh[97384]: Fatal error in defaults handling. Program aborted
Sep 23 13:26:03 notised.xyz mariadbd[97387]: [106B blob data]
Sep 23 13:26:03 notised.xyz mariadbd[97387]: Fatal error in defaults handling. Program aborted
Sep 23 13:26:03 notised.xyz systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 13:26:03 notised.xyz systemd[1]: mariadb.service: Failed with result 'exit-code'.
Sep 23 13:26:03 notised.xyz systemd[1]: Failed to start MariaDB 10.6.4 database server.
I've tried purging everything related to mysql and mariadb and reinstalled mariadb multiple times, but run into the same issue.

Cant start mariadb after switching Data Directory

Im trying to switch the data directory of mariadb to my HDD drive. But if i change datadir variable in the 50-server.cnf, mariadb wont start.
I have already modified the new directory with chmod and chown.
After switching the directory i get this message:
pi#raspberrypi:/etc/mysql/mariadb.conf.d $ sudo systemctl restart mariadb
Job for mariadb.service failed because the control process exited with error >code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
This is the Error:
● mariadb.service - MariaDB 10.1.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-08-10 13:10:25 CEST; 3min 26s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 5386 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 5309 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=/usr/bin/galera_recovery; [ $? -eq 0 ] &&
Process: 5305 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 5302 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 5386 (code=exited, status=1/FAILURE)
Status: "Starting Innodb crash recovery"
Aug 10 13:10:20 raspberrypi systemd[1]: Starting MariaDB 10.1.38 database server...
Aug 10 13:10:22 raspberrypi mysqld[5386]: 2019-08-10 13:10:22 1996119856 >[Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starti
Aug 10 13:10:25 raspberrypi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 10 13:10:25 raspberrypi systemd[1]: Failed to start MariaDB 10.1.38 database server.
Aug 10 13:10:25 raspberrypi systemd[1]: mariadb.service: Unit entered failed state.
Aug 10 13:10:25 raspberrypi systemd[1]: mariadb.service: Failed with result 'exit-code'.

Resources