I'm migrating salt to salt multimaster.
So in minions config I have my master list, with some multimaster parameters.
I see each master have his connected minions and can talk only with its.
I actually have jobs which send salt commands to my master to run tasks on some minions.
With multimaster I will need to connect on each master and run command if I want to reach all desired minions.
Is there a way to run commands to all minions from only one host ?
You can use syndics and have another master on top.
https://docs.saltstack.com/en/latest/topics/topology/syndic.html#syndic
this way the minion will connect to your normal Master/s where Syndic/s is also installed and can failover to any of them. Syndic/s (another form of a special minion) will connect to MoM (Master of Masters) and you can push commands to all your masters. You can also have multiple MoMs which syndic/s are always connected to.
This offers HA for your minions, masters/syndics and masters of masters.
It has some performance impact if you plan to deploy a large number of minions and might require tuning many options.
Unfortunately the Syndic architecture is not officially supported by Saltstack the time that I'm writing this but they welcome community patches. The main reason is because they provide HA through their Enterprise product and want customers purchasing the HA option rather than getting it for free.
The latest is just my personal opinion based on PRs/Feature Requests I have seen being rejected or dismissed to be developed by the Saltstack team. I don't think they made any official announcement about this.
Related
Background: I have several servers which run a service I develop. All of them should have the same copy of the service.
To ensure deployment and up-to-dateness I use Ansible, with an idempotent playbook which deploys the service. Since the servers are on an unreliable network, I have to run the playbook periodically (in a cron job) to reach the servers which may not have been available before.
Problem: I was under the impression that the SaltStack philosophy is different: I though I could just "set a state, compile it and offer to a set of minions. These minions would then, at their own leisure, come to the master and get whatever they need to do".
This does not seem to be the case: the minions which were not available at deployment time are skipped.
Question: is there a mechanism which would allow for an asynchronous deployment, in the sense that a state set on the master one time only would then be pulled and applied by the minions (to themselves) once they are ready / can reach the master?
Specifically, without the need to continuously re-offer the same state to all minions, in the hope that the ones which were unavailable in the past are now capable to get the update.
Each time a minion connects to the master there is an event on the event bus which you can react upon.
Reactor
This is the main difference between Ansible and Saltstack.
In order to do what you want, I would react on each minion's reconnect and try to apply a state which is idempotent.
Idempotent
You could also setup a scheduled task in Saltstack that runs the state every X minutes and apply the desired configuration.
Scheduled task
The answer from Daniel Wallace (salt developper):
That is not possible.
The minions connect to the publish port/bus and the master puts new
jobs on that bus. Then the minion picks it up and runs the job, if the
minion is not connected when the job is published, then it will not
see the job.
Long story short - I need to use networking between projects to have separate billing for them.
I'd like to reach all the VMs in different projects from a single point that I will use for provisioning systems (let's call it coordinator node).
It looks like VPC network peering is a perfect solution to this. But unfortunately one of the existing networks is "legacy". Here's what google docs state about legacy networks.
About legacy networks
Note: Legacy networks are not recommended. Many newer GCP features are not supported in legacy networks.
OK, naturally the question arises: how do you migrate out of legacy network? Documentation does not address this topic. Is it not possible?
I have a bunch of VMs, and I'd be able to shutdown them one by one:
shutdown
change something
restart
unfortunately it does not seem possible to change network even when VM is down?
EDIT:
it has been suggested to recreate VMs keeping the same disks. I would still need a way to bridge legacy network with new VPC network to make migration fluent. Any thoughts on how to do that using GCE toolset?
One possible solution - for each VM in the legacy network:
Get VM parameters (API get method)
Delete VM without deleting PD (persistent disk)
Create VM in the new VPC network using parameters from step 1 (and existing persistent disk)
This way stop-change-start is not so different from delete-recreate-with-changes. It's possible to write a script to fully automate this (migration of a whole network). I wouldn't be surprised if someone already did that.
UDPATE
https://github.com/googleinterns/vm-network-migration tool automates the above process, plus it supports migration of a whole Instance Group or Load Balancer, etc. Check it out.
I am trying to run a multi-master setup in our dev environment.
The idea is that every dev team has their own salt master. However, all minions in the entire dev environment should be able to receive salt commands from all salt master servers.
Since not every team needs their salt master 24/7, most of them are turned off for several days during the week.
I'm running 2016.11.4 on the masters, as well as on the minions.
However, I run into the following problem: If one of the hosts that are listed in the mininons config file is shut down, the minion will not always report back on a 'test.ping' command (not even with -t 60)
My experience is, that the more master servers are offline, the longer the lag of the minion is to answer requests.
Especially if you execute a 'test.ping' on MasterX while the minions' log is at this point:
2017-05-19 08:31:44,819 [salt.minion ][DEBUG ][5336] Connecting to master. Attempt 4 (infinite attempts)
If I trigger a 'test.ping' at this point, chances are 50/50 that I will get a 'minion did not return' on my master.
Obviously though, I always want a return to my 'test.ping', regardless from which master I send it.
Can anybody tell me if what I try is feasible with salt? Because all the articles about salt multi master setup that I could find would only say: 'put a list of master servers into the minion config and that's it!'
The comment from gtmanfred solved my question:
That is not really the way multi master is meant to work. It is supposed to be used more for failover and not for separating out teams.
I just have a question I can't figure out about saltstack. It concerns the mater and the minion configuration.
Salt is even driven, but the documention says (and it works) we should only open port on the master, and that event are received on the master.
However it seems a little ambiguous as the salt command is run from master to execute task on minions. But I'm unsuccessfuly trying to understand how the master does that, and I can't a clear documention about it.
And we also have these statements in salt documention architechure :
More Salt Master facts:
Job publisher with pub/sub and reply channel;
Two open ports on master (default 4505 / 4506);
Salt Mine stores the most recent miniondata, cached data is visible
to other minions;
Salt Syndic passes jobs from a higher master for hierarchal system
management;
Multi-master for SaltStack high availability.
and this
More Salt Minion facts: Listens and receives jobs from a remote Salt
Master;
Creates and reports compressed events (job results, system alerts) to
the Salt Master;
No open ports, not chatty;
Shares data with other Salt Minions via the peer system;
Returners deliver minion data to any system (Salt Master by default)
that can receive the data.
I've highlighted what is ambiguous for me in the attached screenshot.
The question beeing how can we say the no port is to be opened on the minions and also say minions are listerning from master?
Minions listen on what? To what?
Thanks for clarifications.
Good question here. By default, Salt uses a zmq pub/sub interface. So there is a slight mismatch between what's literally happening on the network and most people's mental model of how Salt works.
The zmq connection just needs those 2 ports on the Salt master to allow for the pub/sub interface to work. The minion reaches out to the master on pub port. Zmq just handles all the necessary network communication for you. The Salt Master "publishes" jobs on the pub port.
As far as a mental model of how Salt works, it's helpful to think of the minion "listening" on the pub port and executing commands when the Salt Master publishes a job on the pub port when the minion matches the job target.
I am currently in the process of creating 3 Neo4j High Availability servers. My business logic leaves one server as a dedicated master, while the other two machines are dedicated slaves. My slaves exist in an entirely different datacenter than my master.
What is the best method to establish a link between the two applications? I've been able to establish connections using OpenVPN, but am curious if that would be better than like SSH port forwarding? I'm not entirely sure how Zookeeper needs to communicate with each other node. A VPN connection only creates a one-way connection, where my master, for example, can create a connection with slave, but could not create one with its master. (I think?)
How should I do this? Thanks!
PS: My master is using an embedded instance of Neo4j, while the slaves are stand-alone instances (if this matters).
So your setup is not about availability as the slaves cannot become masters anyway?
Just about replication to the other datacenter?
You also need to take the neo4j coordinator (zookeeper) into account which is usually needed for all cluster participants.
My colleague suggested that you might get away with just putting the zookeeper (perhaps even just a single one as you don't need master election) directly besides your master server.
Then the ability to connect into the masters' VPN should be enough for the slaves to pull updates.