I am trying to mount a repository with server config files (think nginx, mysql, etc) inside my salt fileserver in order to be able to distribute these files to my minions (Without having to do a checkout of the full repository on all my minions).
If I've understood correctly: All gitfs_remotes will be 'flattened' into one filesystem structure (I can confirm this when I run salt-run fileserver.file_list.
What worries me is that, as far as I know, this 'config file only' repository is now also being searched by Salt for state modules.
Is there some way to either:
Designate a gitfs mount as 'don't search for SLS files'.
Mount the actual salt state repository (which contains my top.sls and state modules) under a subdirectory of the salt fileserver and point salt to the top.sls therein?
I stand open to the possibility that this is a wrong approach entirely of course, my only requirement is that the server config files (again, nginx, mysql, etc) live in a separate repository, and that the entire high state (state modules, top file) lives in git.
master config:
fileserver_backend:
- gitfs
gitfs_remotes:
- git#github.com:MyOrg/salt-configs.git:
- git#github.com:MyOrg/server-config-files.git:
- mountpoint: config-files
Have you considered storing your configuration file in a pillar?
For example:
HostFiles:
LinuxBasic: |
192.168.1.1 server1
192.168.1.2 server2
And then in your state file, when you want to render the hostfile:
LinuxBasicHostFile:
file.managed:
- name: /etc/hosts
- contents_pillar: {{ HostFiles:LinuxBasic }}
You could also GPG that file if it was sensitive using the keys on your Salt master's server:
$ cat nginx.hostfile | sudo gpg --armor --batch --trust-model always --encrypt --homedir <salthomdir> -r <keyname>
Paste the output of that into your pillar:
HostFiles:
LinuxBasic: |
-----BEGIN PGP MESSAGE-----
Xks383...a bunch of encrypted text...BjAs0
-----END PGP MESSAGE-----
And inform your salt master that HostFiles contains GPG encrypted content in your master.conf, or better yet, in a local conf file in /etc/salt/master.d/decrypt.conf:
decrypt_pillar:
- 'HostFiles': gpg
Related
I am configuring a local Salt setup and I have hit a bit of a wall.
My setup is:
CentOS: Red Hat Enterprise Linux Server release 7.7 (Maipo)
Salt: salt 3000.1
I have a very basic configuration with nothing changed from default in the Master or Minion config.
My directory structure is as follows:
/srv/salt/apache/init.sls
/srv/salt/uptodate/common.sls
If I run the following:
salt '*' state.sls apache Test=true
It correctly applies the sls files inside the apache folder.
If I run:
salt '*' state.sls uptodate Test=true
It returns:
minion:
Data failed to compile:
----------
No matching sls found for 'uptodate' in env 'base'
I have no top.sls files configured and if I move common.sls into the apache directory it also does not get applied.
Does anyone have any idea what is going wrong here?
The init.sls can be compared to an index.html file on a webserver.
If you want to apply a statefile other than a init.sls you need to add the name of the state file.
This should work for you:
salt '*' state.sls uptodate.common test=True
I encrypted a bunch of files (certificates) using the following script
for i in $(find . -type f); do ansible-vault encrypt $i --vault-password-file ~/.vault && echo $i encrypted ; done
During rsyncing I run something like this
- name: Copy letsencrypt files
synchronize:
src: "{{ path }}/letsencrypt/"
dest: /etc/letsencrypt/
rsync_path: "sudo rsync"
rsync_opts:
- "--delete"
- "--checksum"
- "-a"
notify:
- Reload Nginx
The problem I’ve faced is that the files that moved still remained encrypted. I thought ansible was smart enough to detect if it was encrypted and decrypt like how I do here
- name: Copy deploy private key
copy:
content: "{{ private_key_content }}"
dest: "/home/deploy/.ssh/id_rsa"
owner: deploy
group: deploy
mode: 0600
no_log: true
Back to the earlier question, how do I make sure the files in the folder/files are decrypted before rsyncing?
Edit:
I tried using the copy module since it is encryption aware but the module seems to be hanging. Noticed some issues with copy module for directories on ansible github and I am back to synchronize.
I also tried the with_fileglob approach but that flattens the directory structure.
Edit 2:
I got encryption, decryption to work with the copy module but its horribly slow.
There is already an issue https://github.com/ansible/ansible/issues/45161 at the ansible site open and the conclusion is:
Synchronize is a wrapper around rsync, I doubt that you can hook into the
process like that. You might want to implement a custom module doing this
or use something, which supports it.
Is it mandatory to put files in /srv/salt folder of master to transfer file/directory from master to connected minions.
Can we transfer files without using salt file server
1) Can we transfer files without using salt file server?
2) Also the document says "You can't run interactive scripts" .
Does it mean there are some limitations to execute arbitrary linux commands with cmd.run eg. we can run salt "*" cmd.run ['ls -l /home'] .
Similarly can we run commands like scp,ssh with cmd.run.
You don't have to use the salt file server. You can also set the source to for example a http location. In that case though, you must also declare the hash of the file, for example:
/etc/nginx/sites-enabled/mysite:
file.managed:
- source: http://example.com/mysite
- source_hash: abc123....
The document suggests the default location as /srv/salt as mentioned here
I want to store --password-file option that comes with rsync. I don't want to use ssh public_private key encryption. I have tried this command:
rsync -avz --progress --password-file=pass.txt source destination
This says:
The --password-file option may only be used when accessing an rsync daemon.
So, I tried using:
rsync -avz --progress --password-file=pass.txt source destination rsyncd --daemon
But this return various errors like unknown options. Is my sytanx correct? How do I setup rsync daemon in my Debian machine.
That is correct,
--password-file is only applicable when connecting to a rsync daemon.
You probably haven't set it in the daemon itself though, the password you set and the one you use during that call must match.
Edit /etc/rsyncd.secrets, and set the owner/group of that file to root:root with world reading permissions.
#/etc/rsyncd.secrets
root:YourSecretestPassword
To connect to a rsync daemon, use a double colon followed by the module name, and the file or folder to synchronize (instead of a colon when using SSH),
RSYNC_PASSWORD="YourSecretestPassword"; rsync -rtv user#remotehost::module/source/ destination/
NOTE:
this implies abdicating SSH encryption, though the password itself is not sent across the network in plain text, your data is ...
this is already insecure as is, never as the the same password as any of your users account.
For a better understanding of its inner workings (how to give specific IPs/processes the ability to upload to specified areas of the filesystem without the need for a user account): http://transamrit.net/docs/rsync/
After trying a while, I got this to work. Since Im copying from my live server (and routers data) to my local server in my laptop as backup user no problem with password been unencrypted, its secured wired on my laptop at home. First you need to install sshpass if Centos with yum install sshpass then create a user backup and assign a temp password. I listed the -p option in case your ssh port is different than default.
sshpass -p 'password' rsync -vaurP -e 'ssh -p 2222' backup#???.your.ip.???:/somedir/public_data/temp/ /your/localdata/temp
Understand SSH RSA is a better permanente alternative and all that, but this is a quick alternative to backup and restore on the go. It works if you are not too concern about security but more concern about your data been backup locally as in an emergency o data recovery. Your user backup password you can change it once the backup is completed. Its a lot faster to setup when your servers change IPs, users, and its in constant modifications (as routers change config and non static IPs, also when routers are not local and you are backing up clients servers locally, where you dont have always access to do SSH. Some of my clients dont even have SSH installed and they dont want to hassle with creating public keys. On some servers only where you have access on a temporary basis. By the way, if you want to do the restore, just reverse the case. Dont need change much, from the same command shell you can do it reversing the order of target and source directories, and creating another backup user with same temp password on the target. After finish, you delete the backup user or change its passwords on target and/or source servers. You can protect even further, as I have done, replacing the password for a one line file using a bash script for multi server environment. Alternative is to use the -f option so the password does not show in the bash history -f "/path/to/passwordfile" Regards
NOTE: If you want to update only modified files then you should use this parameters -h -v -r -P -t as described here https://unix.stackexchange.com/questions/67539/how-to-rsync-only-new-files
rsync -arv -e \
"sshpass -f '/your/pass.txt' ssh -o StrictHostKeyChecking=no" \
--progress /your/source id#IP:/your/destination
Maybe you have to install "sshpass" if you not.
I understand preserving the permissions for rsync.
However in my case my local computer does not have the user the files need to under for the webserver. So when I rsync I need the owner and group to be apache on the webserver, but be my username on my local computer. Any suggestions?
I wanted to clarify to explain exactly what I need done.
My personal computer: named 'home' with the user account 'michael'
My web server: named 'server' with the user account 'remote' and user account 'apache'
Current situation: My website is on 'home' with the owner 'michael' and on 'server' with the owner 'apache'. 'home' needs to be using the user 'michael' and 'server' needs to be using the user 'apache'
Task: rsync my website on 'home' to 'server' but have all the files owner by 'apache' and the group 'apache'
Problem: rsync will preseve the permissions, owner, and group; however, I need all the files to be owner by apache. I know the not preserving the owner will put the owner of the user on 'server' but since that user is 'remote' then it uses that instead of 'apache'. I can not rsync with the user 'apache' (which would be nice), but a security risk I'm not willing to open up.
My only idea on how to solve: after each rsync manually chown -R and chgrp -R, but it's a huge system and this takes a long time, especially since this is going to production.
Does anyone know how to do this?
Current command I use to rsync:
rsync --progress -rltpDzC --force --delete -e "ssh -p22" ./ remote#server.com:/website
If you have access to rsync v.3.1.0 or later, use the --chown option:
rsync -og --chown=apache:apache [src] [dst]
More info in an answer from a similar question here: ServerFault: Rsync command issues, owner and group permissions doesn´t change
There are hacks you could put together on the receiving machine to get the ownership right -- run 'chmod -R apache /website' out of cron would be an effective but pretty kludgey option -- but instead, I'd recommend securely allowing rsync-over-ssh-as-apache.
You'd create a dedicated ssh keypair for this:
ssh-keygen -f ~/.ssh/apache-rsync
and then take ~/.ssh/apache-rsync.pub over to the webserver, where you'd put it into ~apache/.ssh/authorized_keys and carefully specify the allowed command, something like so, all on one line:
command="rsync --server -vlogDtprCz --delete . /website",from="IP.ADDR.OF.SENDER",no-port-forwarding,no-X11-forwarding,no-pty ssh-rsa AAABKEYPUBTEXTsVX9NjIK59wJ+fjDgTQtGwhATsfidQbO6u77dbAjTUmWCZjKAQ/fEFWZGSlqcO2yXXXXXXXXXXVd9DSS1tjE6vAQaRdnMXBggtn4M9rnePD2qlR5QOAUUwhyFPhm6U4VFhRoa3wLvoqCVtCV0cuirB6I45On96OPijOwvAuz3KIE3+W9offomzHsljUMXXXXXXXXXXMoYLywMG/GPrZ8supIDYk57waTQWymUyRohoQqFGMzuDNbq+U0JSRlvLFoVUZ5Piz+gKJwwiFwwAW2iNag/c4Mrb/BVDQAyEQ== comment#email.address
and then your rsync command on your "home" machine would be something like
rsync -av --delete -e 'ssh -i ~/.ssh/apache-rsync apache#server' ./ /website
There are other ways to skin this cat, but this is the clearest and involves the fewest workarounds, to my mind. It prevents getting a shell as apache, which is the biggest security concern, natch. If you're really deadset against allowing ssh as apache, there are other ways ... but this is how I've done it.
References here: http://ramblings.narrabilis.com/using-rsync-with-ssh, http://www.sakana.fr/blog/2008/05/07/securing-automated-rsync-over-ssh/
Last version (at least 3.1.1) of rsync allows you to specify the "remote ownership":
--usermap=tom:www-data
Changes tom ownership to www-data (aka PHP/Nginx). If you are using Mac as the client, use brew to upgrade to the last version. And on your server, download archives sources, then "make" it!
The solution using rsync --chown USER:GROUP [src] [dst] only works if the remote user has write access to the the destination directory which in most cases is not the case.
Here's another solution:
Overview
(srcmachine) (rsync) (destmachine)
srcuser -- SSH --> destuser
|
| sudo su jenkins
|
v
jenkins
Let's say that you want to rsync:
From:
Machine: srcmachine
User: srcuser
Directory: /var/lib/jenkins
To:
Machine: destmachine
User: destuser to establish the SSH connection.
Directory: /tmp
Final files owner: jenkins.
Solution
rsync --rsync-path 'sudo -u jenkins rsync' -avP --delete /var/lib/jenkins destuser#destmachine:/tmp
Read more here:
https://unix.stackexchange.com/a/546296/116861
rsync version 3.1.2
I mostly use windows in local, so this is the command line i use to sync files with the server (debian) :
user#user-PC /cygdrive/c/wamp64/www/projects
$ rsync -rptgoDvhnP --chown=www-data:www-data --exclude=.env --exclude=vendor --exclude=node_modules --exclude=.git --exclude=tests --exclude=.phpintel --exclude=storage ./website/ username#hostname:/var/www/html/website
-n : perform a trial run with no changes made, to really execute the command remove the -n option