Salt Error while bringing up minion for multi-master - salt-stack

Question

I'm getting some errors when I start salt minions:
Error while bringing up minion for multi-master. Is master at salt-server00001.nix.us.companyworld.company.com responding?
2019-10-16 11:57:27,935 [salt.minion ][ERROR ][34679] Error while bringing up minion for multi-master. Is master at usazrcncm10286.nix.us.kworld.kpmg.com responding?
2019-10-16 11:57:37,990 [salt.crypt ][ERROR ][34679] The masters sent its public-key signature, but signature verification is not enabled on the minion. Either enable signature verification on the minion or disable signing the public key on the master!
2019-10-16 11:57:37,990 [salt.crypt ][CRITICAL][34679] The Salt Master server's public key did not authenticate!
The master may need to be updated if it is a version of Salt lower than 2017.7.2, or
If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.
I am NOT setting up multi-master.
I have accepted the key for the server on the salt master.
I've opened up the salt ports on the firewall.
But the salt server still isn't responding.
This is my salt versions report:
salt --versions-report
Salt Version:
Salt: 2018.3.4
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8.1
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Mar 30 2018, 15:31:59)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 18.0.1
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.3.1
System Versions:
dist: redhat 6.10 Santiago
locale: UTF-8
machine: x86_64
release: 2.6.32-754.2.1.el6.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 6.10 Santiago
Why am I seeing this problem and how do I correct it?

Answer 1

Look for errors in the /var/log/salt/minion
Then, issue salt '*' service.status and check if the master is running.
Do a check of the salt-master service systemctl status salt-master
In my case, the salt-master failed to start:
● Salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/salt-master.service.d
└─salt-master-opts-managed.conf
Active: failed (Result: timeout) since Wed 2020-07-08 10:59:22 -03; 54min ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltstack.com/en/latest/contents.html
Process: 5780 ExecStart=/usr/bin/salt-master (code=killed, signal=TERM)
Main PID: 5780 (code=killed, signal=TERM)
Jul 08 10:57:52 v321p601 systemd[1]: Starting The Salt Master Server...
Jul 08 10:58:26 v321p601 salt-master[5780]: [INFO ] Setting up the Salt Master
Jul 08 10:59:22 v321p601 systemd[1]: salt-master.service start operation timed out. Terminating.
Jul 08 10:59:22 v321p601 systemd[1]: Failed to start The Salt Master Server.
Jul 08 10:59:22 v321p601 systemd[1]: Unit salt-master.service entered failed state.
Jul 08 10:59:22 v321p601 systemd[1]: salt-master.service failed.
Try restarting the service with systemctl start salt-master.
If the error persists, run netstat -lnp on the master and see if the Salt ports are there and listening.
In my case, I've reissued the command to start the salt-master service after a while and it went online:
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/salt-master.service.d
└─salt-master-opts-managed.conf
Active: active (running) since Wed 2020-07-08 12:19:31 -03; 4min 55s ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltstack.com/en/latest/contents.html
Main PID: 27609 (salt-master)
CGroup: /system.slice/salt-master.service
├─27609 /usr/bin/python /usr/bin/salt-master
├─27618 /usr/bin/python /usr/bin/salt-master
├─27820 /usr/bin/python /usr/bin/salt-master
├─27824 /usr/bin/python /usr/bin/salt-master
├─27839 /usr/bin/python /usr/bin/salt-master
├─27840 /usr/bin/python /usr/bin/salt-master
├─27841 /usr/bin/python /usr/bin/salt-master
├─27842 /usr/bin/python /usr/bin/salt-master
├─27849 /usr/bin/python /usr/bin/salt-master
├─27850 /usr/bin/python /usr/bin/salt-master
├─27851 /usr/bin/python /usr/bin/salt-master
├─27852 /usr/bin/python /usr/bin/salt-master
└─27853 /usr/bin/python /usr/bin/salt-master
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Authentication accepted from 4b11df09-8c57-4745-ac5a-1ab3da854566
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Authentication request from eeea44c1-5d2e-430d-8217-50f2517e587b
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Authentication accepted from eeea44c1-5d2e-430d-8217-50f2517e587b
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Authentication request from 48892d9e-78b1-43c2-a0d4-479ac4907432
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Authentication accepted from 48892d9e-78b1-43c2-a0d4-479ac4907432
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO ] Got return from 70a62d5b-31df-403d-b0f5-3e90d1b9fd7d for job 20200708122114319236
Jul 08 12:21:18 v321p601 salt-master[27609]: [INFO ] Authentication request from 39f50b14-89d4-427b-bd46-39b11cada7da
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO ] Authentication request from 81f9b89c-a7d3-4264-b156-f2b0470590c3
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO ] Authentication accepted from 39f50b14-89d4-427b-bd46-39b11cada7da
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO ] Authentication accepted from 81f9b89c-a7d3-4264-b156-f2b0470590c3
References:
saltstack cheatsheet
saltmaster not running properly?