mariadb won't start suddenly - mariadb

Question

I have some trouble with my MariaDB Server, it was working ok but it won't start anymore. When I try to start the server it failed:
root#vps45223599:/var/log# /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
failed!
root#vps45223599:/var/log# systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.41 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-01-10 21:20:58 UTC; 1min 15s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1349 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 1274 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR
Process: 1272 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 1271 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1349 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
Jan 10 21:20:55 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
Jan 10 21:20:55 vps45223599.local mysqld[1349]: 2023-01-10 21:20:55 140599894461824 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1349 ...
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:20:58 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:43 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:15:43 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:15:43 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:43 vps45223599.local sudo[1040]: pam_unix(sudo:session): session closed for user root
Jan 10 21:15:50 vps45223599.local sudo[1146]: root : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/bin/systemctl start mariadb
Jan 10 21:15:50 vps45223599.local sudo[1146]: pam_unix(sudo:session): session opened for user root by cbarca(uid=0)
Jan 10 21:15:50 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:15:50 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:15:50 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Jan 10 21:15:51 vps45223599.local mysqld[1227]: 2023-01-10 21:15:51 139968422329728 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1227 ...
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:15:54 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:15:54 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jan 10 21:15:54 vps45223599.local sudo[1146]: pam_unix(sudo:session): session closed for user root
Jan 10 21:20:55 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:20:55 vps45223599.local systemd[1]: Failed to set devices.allow on /system.slice/mariadb.service: Operation not permitted
Jan 10 21:20:55 vps45223599.local systemd[1]: Starting MariaDB 10.1.41 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has begun starting up.
Jan 10 21:20:55 vps45223599.local mysqld[1349]: 2023-01-10 21:20:55 140599894461824 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB-0+deb9u1) starting as process 1349 ...
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jan 10 21:20:58 vps45223599.local systemd[1]: Failed to start MariaDB 10.1.41 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Unit entered failed state.
Jan 10 21:20:58 vps45223599.local systemd[1]: mariadb.service: Failed with result 'exit-code'.
I don't know what happened. I also try
mysqlcheck
root#vps45223599:/var/log# mysqlcheck --all-databases -p
Enter password:
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") when trying to connect
I don't know what else I should try, can anyone helpe me, please?
Cheers

Answer 1

As #obe said this is likely a proxmox or whatever is instigating systemd without sufficient privileges is the issue.
The devices.allow error is probably requested by the PrivateDevices=true (seems confirmed based on MDEV-13207 which failed to provide more info) aspect of the systemd service file for MariaDB.
PrivateDevices=true allows:
/dev/null
/dev/zero
/dev/random
Based on this answer for a different device, doing the equivalent for these devices would be:
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:3 rwm'
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:8 rwm'
lxc config set mycontainer raw.lxc 'lxc.cgroup.devices.allow = c 1:5 rwm'
Major/minor device code determined by:
$ ls -la /dev/zero /dev/random /dev/null
crw-rw-rw-. 1 root root 1, 3 Jan 8 22:20 /dev/null
crw-rw-rw-. 1 root root 1, 8 Jan 8 22:20 /dev/random
crw-rw-rw-. 1 root root 1, 5 Jan 8 22:20 /dev/zero

Answer 2

Thanks all for the answer.
#danblack Nothing change, suddenly stop working, but reading this forum I found the solution in this thread.
Can't start MariaDB on debian 10
Basically the solution is ...
"Solved it by deleting/renaming the tc.log mv -vi /var/lib/mysql/tc.log /root And restarting the database service mysql restart"
And mariaddb start again.