How to Fix Minions that Can’t Find a New Replacement salt-master Server

Emergency Scenario: Need to reestablish security between salt master and minions. Too much tickering around and I’ve broken my salt-master server. No problem, right? Just spin up a raspberry, install the SaltStack master software on it and let the minions find the new server. All you have to do is make sure that the new raspberry has the same IP address as the old salt-master, let dns call the new server “salt” and you should be good, right? Wrong!

root@salt:~# salt 'web*' state.apply
webs1:
Minion did not return. [Not connected]
webs2:
Minion did not return. [Not connected]
webs3:
Minion did not return. [Not connected]
ERROR: Minions returned with non-zero exit code
root@salt:~#

saltstack has security involved to keep someone from taking over your enterprise, just by putting up a rogue salt master. Your replacement salt master will be essentially that, a rogue server — at least until you break the paired key on every minion. Sounds tricky, but it’s really not. Oh, I know what you’re thinking, “Just delete the keys and re-accept them on the master!”. Nope. saltstack is smarter than you think. Each minion stores a copy of the master’s key on itself. You’re going to have to remove that key from each minion and THEN re-accept a key, which will not only inform the master of the existance of the minion but also give the minion the opportunity to collect the master key.

SO, how do we make this happen? Just a couple of (very methodical) short steps:

  1. delete the keys on the master and stop the server
  2. delete the keys on the minions and restart the minions
  3. accept the new keys on the master

First, log in to your salt-master and remove the existing keys:

root@salt:~# salt-key -D
The following keys are going to be deleted:
Accepted Keys:
webs1
webs2
webs3
Proceed? [N/y] y
The following keys are going to be deleted:
Accepted Keys:
webs1
webs2
webs3
Proceed? [N/y] y
Key for minion webs1 deleted.
Key for minion webs2 deleted.
Key for minion webs3deleted.
root@salt:~# systemctl stop salt-master
root@salt:~#

Next, go to each minion and remove the master key that was stored there and restart:

root@webs1:~# rm /etc/salt/pki/minion/minion_master.pub
root@webs2:~# rm /etc/salt/pki/minion/minion_master.pub
root@webs3:~# rm /etc/salt/pki/minion/minion_master.pub

(Sometimes the file to delete is:
/var/lib/salt/pki/minion/minion_master.pub

Now back to the master and turn things on again:

root@salt:~# systemctl start salt-master

restart each of the minions:

root@webs1:~# systemctl restart salt-minion

root@webs2:~# systemctl restart salt-minion

root@webs3:~# systemctl restart salt-minion

and then, finally, Accept the keys key requests coming in from the minions when they restart:

root@salt:~# salt-key
Accepted Keys:
Denied Keys:
Unaccepted Keys:
webs1
webs2
webs3
Rejected Keys:
root@salt:~# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
webs1
webs2
webs3
Proceed? [n/Y] Y
Key for minion webs1 accepted.
Key for minion webs2 accepted.
Key for minion webs3 accepted.
root@salt:~#

You can accept them each with salt-key -a <minion name> or do what I did here and just accept them all with the “-A” switch.

Happy salting!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.