I'm running a command to (re)create a remote file. I'd like to see a difference between the old and the new versions of the file -- and for the task to set changed: false, if there are no differences.
Is there a way to do this without doing it all by hand -- with multiple tasks (creating a backup, running the command, diff-ing the two, etc.)?
Ok, apparently, this is not currently possible. The best I could come up with was:
Run the command with output sent simply to stdout, rather than directly to a file. This is possible with many commands (such as with -o /dev/stdout), allows to declare it with changed_when: False and check_mode: ansible_check_mode. Save the results:
- name: Run the command to obtain content
command: foo -
register: foo
changed_when: False
check_mode: ansible_check_mode
...
Use the copy module to (re)write the file with the captured output:
- name: Write the content into file
copy:
dest: /where/ever/it/was
content: '{{ foo.stdout }}'
...
This is Ok for when your files aren't too large -- requiring "only" one additional task... One per file.
For everything else, I created this enhancement-request.
Related
I allow the user the option to enter 3 digit number for setting permission of files or folders being transferred say -e myperm: 775
I use ansible where I provide rsync_opts: --chmod:F775 (synchronize module) to change the permission of the transferred file/folder on the destination to 775
- name: sync file
synchronize:
src: /tmp/file.py
dest: /home/myuser/file.py
mode: push
rsync_opts:
- "--chmod=F0{{ myperm }}"
Above works fine for files; however, the same does not work for transferring folders say when src: /tmp/folder
I tried --chmod=D0{{ myperm }},F0{{ myperm }} in ansible but it translates to --chmod=D0775 F0775 and gives this error:
msg": "Unexpected remote arg: user#desthost:/tmp/folder\nrsync error: syntax or usage error (code 1) at main.c(1344) [sender=3.1.2]\n", "rc": 1}
Can you please suggest rsync_opts with variable myperm for changing permissions of both files and folders?
Any other solution will also be fine.
It seems a problem in parsing the comma separated argument when the module generates the rsync command line, however, since rsync allows multiple chmod options, you can rewrite your task as:
[..]
rsync_opts:
- "--chmod=F0{{ myperm }}"
- "--chmod=D0{{ myperm }}"
I want to set some parameters as defined here(https://github.com/nteract/papermill#python-version-support). The catch is, I want to be able to do this via UI. I have a JHub installed on my cluster and while opening it, I want certain parameters to be set by default.
Also, when I pass the parameters via papermill(the above script gets saved somewhere and then I will run it via papermill), I want the latter to override the former.
I tried looking into several topics in pure JuPyter notebooks but in vain.
For the user to access some parameters as soon as her notebook starts, ipython needs to know the startup cells. This can be done via the following commands in case of JuPyterHub:
proxy:
secretToken: "yada yada"
singleuser:
image:
name: some_acc_id.dkr.ecr.ap-south-1.amazonaws.com/demo
tag: 12h
lifecycleHooks:
postStart:
exec:
command: ["/bin/sh", "-c", 'ipython profile create; cd ~/.ipython/profile_default/startup; echo ''run_id = "sample" ''> aviral.py']
imagePullSecret:
enabled: true
registry: some_acc_id.dkr.ecr.ap-south-1.amazonaws.com
username: aws
email: aviral#abc.com
Make sure you are escaping the quotes in the yaml correctly, or simply follow what I have done above.
Once this is done, papermill will override the params but for that, you have to make sure that the cell is tagged as "parameters". For instance, in my jupyterhub, every notebook that starts has run_id variable with the value "sample".
Hello helpful friends,
We have quite a setup here of 100+ servers being managed by Salt states. With different roles in the organization executed by different people, I'd really like to have a possibility to "aggregate" some states. In this case: updating (yum) packages.
I would really like to have our sysadmins safely being able to execute a command like this on the master:
salt '*' state.apply update.packages
while maybe our developers would be able to execute:
salt 'dev-*' state.apply update.application
Of course we have a large set of sls files and the key to this issue is that I don't want all those states executed, but just a selected bunch of them.
To achieve this, I've tried to create an update/packages.sls state, containing:
update-packages:
test.nop
And then added to, for example the following existing state:
nagios-plugins-all:
pkg.latest:
- require:
- pkg: corepackages
a watch_in as follows:
nagios-plugins-all:
pkg.latest:
- require:
- pkg: corepackages
- watch_in:
- test: update-packages
Unfortunately, this is clearly not the way to go, as executing salt 'testserver001' state.apply update.packages now only returns:
testserver001:
----------
test_|-update-packages_|-update-packages_|-nop:
----------
__id__:
update-packages
__run_num__:
0
changes:
----------
comment:
Success!
duration:
0.946
name:
update-packages
result:
True
start_time:
12:10:46.035686
while I know for sure that updated packages are available. I can't include all the existing state files into the update/packages.sls file, as that would cause all states to be executed in those files and that's not what I want either. It would also become a very messy file.
I also don't want to just execute salt '*' pkg.upgrade as I have states depending on updates; i.e. if the package nagios is updated, the states concerning the up-to-date config files should be run and consequently a restart of the nagios service should be executed. All of that is configured in salt using watch and require arguments, so I'd like to use that also when updating my packages. Also, I want to be in control of which packages can be updated.
I don't know if I'm on the right path, or whether this is possible with Salt at all, but maybe someone here has a brilliant idea on how to achieve this behavior. I would be very thankful!
You might want to look at External Auth System of salt.
This way you can limit users and group to specific minions and commands, and even restrict the parameters.
I am writing a build process for a WordPress installation using Ansible. It doesn't have a application-level build system at the moment, and I've chosen Ansible so that it can cleanly integrate with server build scripts, so I can bring up a working server at the touch of a button.
Most of my WordPress plugins are being installed with the unarchive feature, pointing to versioned plugin builds on the official wordpress.org installation server. I've encountered a problem with just one of these, which is that it is always being marked as "changed" even though the files are exactly the same.
Having examined the state of ls -Rl before and after, I noticed that this plugin (WordPress HTTPS) is the only one to use internal sub-directories, and upon each decompression, the modification time of folders is getting bumped.
It may be useful to know that this is a project build script, with a connection of local. I guess therefore that means that SSH is not being used.
Here is a snippet of my playbook:
- name: Install the W3 Total Cache plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/w3-total-cache.0.9.4.1.zip
dest=wp-content/plugins
copy=no
- name: Install the WP DB Manager plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/wp-dbmanager.2.78.1.zip
dest=wp-content/plugins
copy=no
# #todo Since this has internal sub-folders, need to work out
# how to preserve timestamps of the original folders rather than
# re-writing them, which forces Ansible to record a change of
# server state.
- name: Install the WordPress HTTPS plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/wordpress-https.3.3.6.zip
dest=wp-content/plugins
copy=no
One hacky way of fixing this is to use ls -R before and after, using options to include file sizes but not timestamps, and then md5sum that output. I could then mark it as changed if there is a change in checksum. It'd work but it's not very elegant (and I'd want to do that for all plugins, for consistency).
Another approach is to abandon the task if a plugin file already exists, but that would cause problems when I bump the plugin version number to the latest copy.
Thus, ideally, I am looking for a switch to present to unarchive to say that I want the folder modification times from the zip file, not from playbook runtime. Is it possible?
Update: a commenter asked if the file contents could have changed in any way. To determine whether they have, I wrote this script, which creates a checksum for (1) all file contents and (2) all file/directory timestamps:
#!/bin/bash
# Save pwd and then change dir to root location
STARTDIR=`pwd`
cd `dirname $0`/../..
# Clear collation file
echo > /tmp/wp-checksum
# List all files recursively
find wp-content/plugins/wordpress-https/ -type f | while read file
do
#echo $file
cat $file >> /tmp/wp-checksum
done
# Get checksum of file contents
sha1sum /tmp/wp-checksum
# Get checksum of file sizes
ls -Rl wp-content/plugins/wordpress-https/ | sha1sum
# Go back to original dir
cd $STARTDIR
I ran this as part of my playbook (running it in isolation using tags) and received this:
PLAY [Set this playbook to run locally] ****************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [jonblog : Run checksum command] ******************************************
changed: [localhost]
TASK [jonblog : debug] *********************************************************
ok: [localhost] => {
"checksum_before.stdout_lines": [
"374fadc4df1578f78fd60b1be6758477c2c533fa /tmp/wp-checksum",
"10d66f7bdbbdd3af531d1b11a3db3059a5868838 -"
]
}
TASK [jonblog : Install the WordPress HTTPS plugin] ***************
changed: [localhost]
TASK [jonblog : Run checksum command] ******************************************
changed: [localhost]
TASK [jonblog : debug] *********************************************************
ok: [localhost] => {
"checksum_after.stdout_lines": [
"374fadc4df1578f78fd60b1be6758477c2c533fa /tmp/wp-checksum",
"719c9da94b525e723b1abe188ee9f5bbaf121f3f -"
]
}
PLAY RECAP *********************************************************************
localhost : ok=6 changed=3 unreachable=0 failed=0
The debug lines reflect the checksum hash of the contents of the files (this is identical) and then the checksum hash of ls -Rl of the file structure (this has changed). This is in keeping with my prior manual finding that directory checksums are changing.
So, what can I do next to track down why folder modification times are incorrectly flagging this operation as changed?
Rather than overwriting all files each time and find a way to keep the same modification datetime, you may want to use the creates option of the unarchive module.
As you maybe already know, this tells Ansible that a specific file/folder will be created as a result of the task. Thus, next time the task will not be run again if that file/folder already exists.
See http://docs.ansible.com/ansible/unarchive_module.html#options
My solution is to modify the checksum script and to make that a permanent feature of the Ansible process. It feels a bit hacky to do my own checksumming, when Ansible should do it for me, but it works.
New answers that explain that I am doing something wrong, or that a new version of Ansible fixes the problem, would be most welcome.
If I get a moment, I will raise this as a possible bug with the Ansible team. However I do sometimes wonder about the effort/reward ratio when raising bugs on a busy tracker - I already have one item outstanding, it has been waiting a while, and I've chosen to work around that too.
Update (18 months later)
This Ansible build system never made it into live. It felt like I was always working around something. Recently, when I decided I needed to move my blog to another server, I finally Dockerised it. This took several weeks (since there is a surprising amount of things to think about in a real WordPress installation) but in general I found the process much nicer than using orchestration tools.
I'm trying to use Salt to deploy an online tool to a new VPS. The process involves cloning a git repo and then various set-up commands - however there seems to be an issue with including other .sls files from within sub directories.
Here's a simplified version:
Master config file:
file_roots:
base:
- /srv/salt/saltstates
I have a a file in /srv/salt/saltstates/test/test.sls containing:
base:
'*':
- basic
The file /srv/salt/saltstates/test/basic.sls contains:
Europe/London:
timezone.system
However, when I run salt 'Minion1' state.sls test.test, an error is returned:
Minion1:
----------
ID: base
Function: *.basic
Result: False
Comment: State *.basic found in sls test.test is unavailable
Started:
Duration:
Changes:
OK, so you've confused several things here.
First of all the contents you've put in /srv/salt/saltstates/test/test.sls really is what is called a top file and should probably be moved to /srv/salt/saltstates/top.sls
The top.sls is only needed if you want to do a highstate, but since you're trying to run salt 'Minion1' state.sls test.test you don't really need the top.sls.
Now since you have your sls file here: /srv/salt/saltstates/test/basic.sls, then the command you want to run is the following:
salt 'Minion1' state.sls test.basic
The "dot" traverses down directories.