By default, salt-api uses glob method to match target minion. Is it possible to use other way(maybe grains, pcre and so on) to find minions.
I tried to pass tgt_type to salt-api, but it didn't work.
curl -s -H "Accept: application/json" -d username='user' -d password='pwd' -d eauth='pam' -d fun='test.ping' -d tgt='127.0.0.1' -d client='local' -d tgt_type='ipcidr' localhost:8000/run
expr_form is the option you're looking for.
expr_form='ipcidr'
This is found quite often in various parts of Salt.
Here's a legend with the different types of matching in Salt:
http://docs.saltstack.com/en/latest/topics/targeting/compound.html
tgt_type should work with latest versions of Salt API. As referred to as the type of tgt (Salt target matcher for minions).
Starting from version 2017.7.0, expr_form has been renamed to tgt_type.
tgt_type can be one of the following:
glob - Bash glob completion - Default
pcre - Perl style regular expression
list - Python list of hosts
grain - Match based on a grain comparison
grain_pcre - Grain comparison with a regex
pillar - Pillar data comparison
pillar_pcre - Pillar data comparison with a regex
nodegroup - Match on nodegroup
range - Use a Range server for matching
compound - Pass a compound match string
ipcidr - Match based on Subnet (CIDR notation) or IPv4 address.
For more information, please refer official client function documentation:
https://docs.saltstack.com/en/latest/ref/clients
Related
I want to know if I can have conflict in this scenario :
#!/usr/bin/env bash
curl -XPUT -d '{"bar":"baz"}' \
-H "Content-Type: application/json" \
http://127.0.0.1:8098/riak/obj/1
response=$(curl -I http://127.0.0.1:8098/riak/obj/1 | grep 'X-Riak-Vclock:' | egrep -o ' .*$')
curl -v -XPUT -d '{"bar":"foo"}' \
-H "Content-Type: application/json" \
-H "X-Riak-Vclock: $response" \
http://127.0.0.1:8098/riak/obj/1
In some words :
First I have no object for the key 1 I put the {"bar":"baz"} value with the PUT of the http api.
Then, I read the value with a get. And I store the vclock in variable.
And finaly I put a new value {"bar":"foo"} for the key 1
Is there a case where I can have {"bar":"baz"} for the key 1 ? If Riak has a conflict, it will be resolve with vclock ?
Thanks !
It depends how your Riak database is configured, either globally or if you changed the default configuration of the bucket you're using. If you keep the default config, your second PUT (with the vclock) might:
- fail, if someone updated the key behind your back (rare), and the vclock data you have is already obsolete. You need to re-read the value and update it. Best is to have a retry mechanism.
- fail, if the write consistency constrains you have is too strict, and too many nodes are down (rare). Usually the default read and write config are sane.
- succeed, if the vclock data is still valid for this key (most of the time)
In case it succeeds, it might be that the network topology was in a split-brain situation. In this case, Riak will solve the issue itself using v-clock data.
Is there a way to exclude minions from being targeted even if I run salt '*' state.apply on CLI?
Ideally the exclusion should be declared somewhere in top.sls
From CLI, you can exclude minion as follows,
salt -C 'not minion-id' test.ping
Above pattern is available since version 2015.8.0. If you are using older version then,
salt -C '* and not minion-id' test.ping
Please read more about Compound matchers here.
You want to use compound matching. Targetting all the minions for the webserver states except minion_id_1 can be done like this.
base:
'not minion_id_1':
- match: compound
- webserver
Documentation on compound matching can be found here: docs.saltstack.com/en/latest/topics/targeting/compound.html
When running the following command on OpenVAS 7 under root (where gb_iojs_detect_win.nasl is one of the OpenVAS plugins. other plugins cause the same error):
openvas-nasl gb_iojs_detect_win.nasl
causes this error:
base gpgme-Message: Setting GnuPG homedir to '/usr/local/var/lib/openvas/gnupg'
base gpgme-Message: Using OpenPGP engine version '1.4.16'
base gpgme-Message: Setting GnuPG sysconf homedir to '/usr/local/etc/openvas/gnupg'
gb_iojs_detect_win.nasl: bad or missing signature. Will not execute this script
Note that i did follow this page and set nasl_no_signature_check = yes, but this error still persists.
Thanks! Any help would be appreciated!
Execute the following and follow the prompts:
gpg --homedir=/usr/local/etc/openvas/gnupg --gen-key
Then execute:
wget http://www.openvas.org/OpenVAS_TI.asc -P /tmp
gpg --homedir=/usr/local/etc/openvas/gnupg --import /tmp/OpenVAS_TI.asc
gpg --homedir=/usr/local/etc/openvas/gnupg --list-keys
gpg --homedir=/usr/local/etc/openvas/gnupg --lsign-key 48DB4530
Source: http://www.openvas.org/trusted-nvts.html
Do what indicated above and update your feeds plugins
I also struggled with this, due to the documentation falling short.
You need to provide the -X flag, which means authenticated mode, which apparently means "Don't do the signature checks"
I was having this issue with OpenVAS 9, but based on the instructions from #D.W., I read the docs about trusted NVTs, they mention that they're going to start using a new key starting 12/1/18.
However, when I ran openvassd -f to see what it was doing, I noticed that there were only some plugin checksums that were failing, and I'm thinking they're already signing these with the new key.
checksum for /var/lib/openvas/plugins/gb_electrasoft_32bit_ftp_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_atlassian_confluence_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_synology_dsm_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_quixplorer_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_yealink_ip_phone_detect.nasl not matching
checksum for /var/lib/openvas/plugins/secpod_neon_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_torrent_trader_classic_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_simatic_s7_snmp_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_sophos_xg_detect_userportal.nasl not matching
checksum for /var/lib/openvas/plugins/gb_keycloak_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_wd_wdtv_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_get_http_banner.nasl not matching
checksum for /var/lib/openvas/plugins/gb_orion_nta_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_gather_windows_uptime.nasl not matching
checksum for /var/lib/openvas/plugins/gb_teleopti_wfm_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_asp_dotnet_core_detect_win.nasl not matching
checksum for /var/lib/openvas/plugins/eggdrop_detect.nasl not matching
checksum for /var/lib/openvas/plugins/gb_f5_linerate_lros_version.nasl not matching
checksum for /var/lib/openvas/plugins/gb_mikrotik_router_routeros_consolidation.nasl not matching
However, the fix was to just use the same instructions (had to modify paths for config directory for Ubuntu/Debian), but with the newer key. After that the checksums were validated OK:
wget https://www.greenbone.net/GBCommunitySigningKey.asc -P /tmp
gpg --homedir=/etc/openvas/gnupg --import /tmp/GBCommunitySigningKey.asc
gpg --homedir=/etc/openvas/gnupg --list-keys
gpg --homedir=/etc/openvas/gnupg --lsign-key 0ED1E580
systemctl restart openvas-scanner.service
Your command should be like this:
openvas-nasl -Xp gb_iojs_detect_win.nasl #for parsing
openvas-nasl -Xt IP gb_iojs_detect_win.nasl #for exec
note the additional switches.
For more details, visit this link
I need the equivalent of "egrep -a " option in solaris.
Currently this option works fine in Red Hat linux, but we wanted to migrate some code in another server which is in solaris flavour.
So need to know the equivalent of "egrep -a " in solaris.
Thanks.
The feature requested (see manual) is documented
-a
--text
Process a binary file as if it were text; this is equivalent to the --binary-files=text option.
--binary-files=type
If a file's allocation metadata, or if its data read before a line is selected for output, indicate that the file contains binary data, assume that the file is of type type. By default, type is ‘binary’, and grep normally outputs either a one-line message saying that a binary file matches, or no message if there is no match. When matching binary data, grep may treat non-text bytes as line terminators.
If type is ‘without-match’, grep assumes that a binary file does not match; this is equivalent to the -I option.
If type is ‘text’, grep processes a binary file as if it were text; this is equivalent to the -a option.
Warning: ‘--binary-files=text’ might output binary garbage, which can have nasty side effects if the output is a terminal and if the terminal driver interprets some of it as commands.
Solaris grep (see manual) has no such feature. Third-party packages are available, e.g., CSWggrep from OpenCSW.
The -a option is a GNU grep extension which has no POSIX equivalent.
If you have a full Solaris 10 installation, you probably already have GNU grep installed in /usr/sfw/bin/ggrep (the double g is not a typo).
You can then replace the egrep -a ... occurence(s) by:
if [ -x /usr/sfw/bin/ggrep ] ; then
EGREP_A="/usr/sfw/bin/ggrep -E"
else
EGREP_A="egrep -a"
fi
$EGREP_A ...
If not installed, you can easily install that package SUNWggrp from you installation media (CD/DVD).
If for some reason, you can't do it, you need to provide more details about what kind of binary file it is and what pattern you are searching in it.
There are certainly other ways to overcome this issue with standard Solaris tools.
I've about 50 or so files in various sub-directories that I'd like to push to a remote server. I figured rsync would be able to do this for me using the --include-from option. Without the --exclude="*" option, all the files in the directory are being synced, with the option, no files are.
rsync -avP -e ssh --include-from=deploy/rsync_include.txt --exclude=* ./ root#0.0.0.0:/var/www/ --dry-run
I'm running it as dry initially and 0.0.0.0 is obviously replaced by the IP of the remote server. The contents of rsync_include.txt is a new line separated list of relative paths to the files I want to upload.
Is there a better way of doing this that is escaping me on a Monday morning?
There is a flag --files-from that does exactly what you want. From man rsync:
--files-from=FILE
Using this option allows you to specify the exact list of files to transfer (as read from the specified FILE or - for standard input). It also tweaks the default behavior of rsync to make transferring just the specified files and directories easier:
The --relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use --no-relative or --no-R if you want to turn that off).
The --dirs (-d) option is implied, which will create directories specified in the list on the destination rather than noisily skipping them (use --no-dirs or --no-d if you want to turn that off).
The --archive (-a) option’s behavior does not imply --recursive (-r), so specify it explicitly, if you want it.
These side-effects change the default state of rsync, so the position of the --files-from option on the command-line has no bearing on how other options are parsed (e.g. -a works the same before or after --files-from, as does --no-R and all other options).
The filenames that are read from the FILE are all relative to the source dir -- any leading slashes are removed and no ".." references are allowed to go higher than the source dir. For example, take this command:
rsync -a --files-from=/tmp/foo /usr remote:/backup
If /tmp/foo contains the string "bin" (or even "/bin"), the /usr/bin directory will be created as /backup/bin on the remote host. If it contains "bin/" (note the trailing slash), the immediate contents of the directory would also be sent (without needing to be explicitly mentioned in the file -- this began in version 2.6.4). In both
cases, if the -r option was enabled, that dir’s entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a). Also note that the effect of the (enabled by default) --relative option is to duplicate only the path info that is read from the file -- it
does not force the duplication of the source-spec path (/usr in this case).
In addition, the --files-from file can be read from the remote host instead of the local host if you specify a "host:" in front of the file (the host must match one end of the transfer). As a short-cut, you can specify just a prefix of ":" to mean "use the remote end of the transfer". For example:
rsync -a --files-from=:/path/file-list src:/ /tmp/copy
This would copy all the files specified in the /path/file-list file that was located on the remote "src" host.
If the --iconv and --protect-args options are specified and the --files-from filenames are being sent from one host to another, the filenames will be translated from the sending host’s charset to the receiving host’s charset.
NOTE: sorting the list of files in the --files-from input helps rsync to be more efficient, as it will avoid re-visiting the path elements that are shared between adjacent entries. If the input is not sorted, some path elements (implied directories) may end up being scanned multiple times, and rsync will eventually unduplicate them after
they get turned into file-list elements.
For the record, none of the answers above helped except for one. To summarize, you can do the backup operation using --files-from= by using either:
rsync -aSvuc `cat rsync-src-files` /mnt/d/rsync_test/
OR
rsync -aSvuc --recursive --files-from=rsync-src-files . /mnt/d/rsync_test/
The former command is self explanatory, beside the content of the file rsync-src-files which I will elaborate down below. Now, if you want to use the latter version, you need to keep in mind the following four remarks:
Notice one needs to specify both --files-from and the source directory
One needs to explicitely specify --recursive.
The file rsync-src-files is a user created file and it was placed within the src directory for this test
The rsyn-src-files contain the files and folders to copy and they are taken relative to the source directory. IMPORTANT: Make sure there is not trailing spaces or blank lines in the file. In the example below, there are only two lines, not three (Figure it out by chance). Content of rsynch-src-files is:
folderName1
folderName2
--files-from= parameter needs trailing slash if you want to keep the absolute path intact. So your command would become something like below:
rsync -av --files-from=/path/to/file / /tmp/
This could be done like there are a large number of files and you want to copy all files to x path. So you would find the files and throw output to a file like below:
find /var/* -name *.log > file
$ date
Wed 24 Apr 2019 09:54:53 AM PDT
$ rsync --version
rsync version 3.1.3 protocol version 31
...
Syntax: rsync <args> <file_and_or_folder_list> <source_dir> <destination_dir/>
Folder names - WITH a trailing /; e.g. Cancer - Evolution/ - are provided in a file (e.g. my_folder_list):
# comment: /mnt/Vancouver/my_folder_list
# comment: 2019-04-24
some_file
another_file
Cancer/
Cancer - Evolution/
Cancer - Genomic Variants/
Cancer - Metastasis (EMT Transition ...)/
Cancer Pathways, Networks/
Catabolism - Autophagy; Phagosomes; Mitophagy/
so those are the "source" (files and/or) folders, to be rsync'd.
Note that if you don't include the trailing / shown above, rsync creates the target folders, but they are empty.
Those folder names provided in the <file_and_or_folder_list> are appended to the rest of their path: <src_dir> = /home/victoria/RESEARCH - NEWS (here, on a different partition), thus providing the complete folder path to rsync; e.g.: ... /home/victoria/RESEARCH - NEWS/Cancer - Evolution/ ...
[ I'm editing this answer some time later (2022-07), and I can't recall if the path provided to <src_dir> is /home/victoria/RESEARCH - NEWS or /home/victoria/RESEARCH - NEWS/ - providing the correct concatenated path. I believe it's the former; if it doesn't work, use the latter. ]
Note that you also need to use --files-from= ..., NOT --include-from= ...
Again the rsync syntax is:
rsync <args> <file_and_or_folder_list> <source_dir> <destination_dir/>
so,
rsync -aqP --delete --files-from=/mnt/Vancouver/my_folder_list "/home/victoria/RESEARCH - NEWS" $DEST_DIR/
where
<args> is -aqP --delete
<file_and_or_folder_list> is --files-from=/mnt/Vancouver/my_folder_list
<source_dir> is "/home/victoria/RESEARCH - NEWS"
<destination_dir/> is $DEST_DIR/ (note the trailing / added to the variable name)
In my BASH script, for coding flexibility I defined variable $DEST_DIR in two parts as follows.
BASEDIR="/mnt/Vancouver"
DEST_DIR=$BASEDIR/data
echo $DEST_DIR ## /mnt/Vancouver/data
## To clarify, here is $DEST_DIR with / appended to the variable name:
echo $DEST_DIR/ ## /mnt/Vancouver/data/
echo $DEST_DIR/apple/banana ## /mnt/Vancouver/data/apple/banana
However, you can more simply specify the destination path:
via a BASH variable: $DEST_DIR=/mnt/Vancouver/data
note that in the rsync expression above, / is appended to $DEST_DIR (i.e. $DEST_DIR/ is actually $DEST_DIR + /), giving the destination directory path /mnt/Vancouver/data/
explicitly state the destination path: /mnt/Vancouver/data/
rsync options used: ## man rsync or rsync -h
-a : archive: equals -rlptgoD (no -H,-A,-X)
-r : recursive
-l : copy symlinks as symlinks
-p : preserve permissions
-t : preserve modification times
-g : preserve group
-o : preserve owner (super-user only)
-D : same as --devices --specials
-P : same as --partial --progress
-q : quiet (https://serverfault.com/questions/547106/run-totally-silent-rsync)
--delete
This tells rsync to delete extraneous files from the RECEIVING SIDE (ones
that AREN’T ON THE SENDING SIDE), but only for the directories that are
being synchronized. You must have asked rsync to send the whole directory
(e.g. "dir" or "dir/") without using a wildcard for the directory’s contents
(e.g. "dir/*") since the wildcard is expanded by the shell and rsync thus
gets a request to transfer individual files, not the files’ parent directory.
Files that are excluded from the transfer are also excluded from being
deleted unless you use the --delete-excluded option or mark the rules as
only matching on the sending side (see the include/exclude modifiers in the
FILTER RULES section). ...
Edit: atp's answer below is better. Please use that one!
You might have an easier time, if you're looking for a specific list of files, putting them directly on the command line instead:
# rsync -avP -e ssh `cat deploy/rsync_include.txt` root#0.0.0.0:/var/www/
This is assuming, however, that your list isn't so long that the command line length will be a problem and that the rsync_include.txt file contains just real paths (i.e. no comments, and no regexps).
None of these answers worked for me, when all I had was a list of directories. Then I stumbled upon the solution! You have to add -r to --files-from because -a will not be recursive in this scenario (who knew?!).
rsync -aruRP --files-from=directory.list . ../new/location
I got similar task: to rsync all files modified after given date, but excluding some directories. It was difficult to build one liner all-in-one style, so I dived problem into smaller pieces.
Final solution:
find ~/sourceDIR -type f -newermt "DD MMM YYYY HH:MM:SS" | egrep -v "/\..|Downloads|FOO" > FileList.txt
rsync -v --files-from=FileList.txt ~/sourceDIR /Destination
First I use find -L ~/sourceDIR -type f -newermt "DD MMM YYYY HH:MM:SS". I tried to add regex to find line to exclude name patterns, however my flavor of Linux (Mint) seams not to understand negate regex in find. Tried number of regex flavors - non work as desired.
So I end up with egrep -v - option that excludes pattern easy way. My rsync is not copying directories like /.cache or /.config plus some other I explicitly named.
This answer is not the direct answer for the question.
But it should help you figure out which solution fits best for your problem.
When analysing the problem you should activate the debug option -vv
Then rsync will output which files are included or excluded by which pattern:
building file list ...
[sender] hiding file FILE1 because of pattern FILE1*
[sender] showing file FILE2 because of pattern *