Related
I have input files:
Bob_1.fastq.gz
Bob_2.fastq.gz
Bob_3.fastq.gz
Bob_4.fastq.gz
Ron_1.fastq.gz
Ron_2.fastq.gz
Ron_3.fastq.gz
Ron_4.fastq.gz
I am running demultiplexing and trimming steps in one snakefile, like this:
workdir: "/path/to/dir/"
(SAMPLES,) =glob_wildcards('/path/to/dir/raw/{sample}.fastq.gz')
rule all:
input:
expand("demulptiplex/{sample}.fastq.gz", sample=SAMPLES),
expand("trimmed/{sample}.trimmed.fastq.gz", sample=SAMPLES)
rule sabre:
input:
infile="/path/to/dir/raw/{sample}.fastq.gz",
barcodefile= "files/{sample}.txt"
output:
unknownfile=temp("demulptiplex/unknown_barcode_{sample}.fastq.gz"),
shell:
"""
/Tools/sabre-master2/sabre se -f {input.infile} -b {input.barcodefile} -u {output.unknownfile}
"""
rule trimmomatic_se:
input:
r="{sample}.fastq.gz"
output:
r="trimmed/{sample}.trimmed.fastq.gz"
threads: 10
shell:
"""java -jar /Tools/Trimmomatic-0.36/trimmomatic-0.36.jar SE -threads {threads} {input.r} {output.r} ILLUMINACLIP:/Tools/Trimmomatic-0.36/adapters/TruSeq3-SE.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 MINLEN:36"""
The demultiplex output files are like this:
Bob_1_CL1.fastq.gz.... Bob_1_CL345.fastq.gz
Bob_2_CL1.fastq.gz.... Bob_1_CL248.fastq.gz
Ron_1_dad1.fastq.gz... Ron_1_dad67.fastq.gz
and so on
So,if I do not specify the demultiplex output file the program would create it by itself. My problem is how to specify/introduce a new wildcard from the output of the previous rule in the next trimming step, as the wildcards are different from initial sample now.
Wildcards just need to be consistent in a rule, not across the workflow. The issue here is that you have a rule generating 'unknown' outputs that you need to process further. For that you need to use checkpoints.
Read through the second block of code about aggregating. Your checkpoint will be demultiplexing and if you don't have any other steps, all will be your aggregate step that calls checkpoints.demultiplex.get. If you search for checkpoint on stackoverflow you will find lots of examples; it's a hard feature to use at first!
I have a bunch of manifest/yaml files that may or may not have these key value pair duplicates:
...
app: activity-worker
app: activity-worker
...
I need to search through each of those files and find those duplicates so that I can remove one of them.
Note: I know that to replace a certain string (say, switch service: to app:) in all files of a directory (say, dev) I can run grep -l 'service:' dev/* | xargs sed -i "" 's/\service:/app:/g'. I'm looking for a relation between lines.
What you call YAML, is not YAML. The YAML specification
very explicitly states that
keys in a mapping must be unique, and your keys are not:
The content of a mapping node is an unordered set of key: value node
pairs, with the restriction that each of the keys is unique. YAML
places no further restrictions on the nodes. In particular, keys may
be arbitrary nodes, the same node may be used as the value of
several key: value pairs, and a mapping could even contain itself as
a key or a value (directly or indirectly).
On the other hand some libraries have implemented this incorrectly, choosing to overwrite
any previous value associated with a key, with a later value. In your case, since
the values are the same, which value would be taken doesn't really matter.
Also your block style representation is not the only way to represent key-value pairs of a
mapping in "YAML", these duplicates could also be represented in a mapping, as
{...., app: activity-worker, app: activity-worker, .... }
With the two occurences not necessarily being next to each, nor on the same line. The
following is also semantically equivalent "YAML" to your input:
{...., app: activity-worker, app:
activity-worker, .... }
If you have such faulty "YAML" files, the best way to clean them up is
using the round-trip capabilities of
ruamel.yaml (disclaimer: I
am the author of that package), and its ability to switch except/warn
on faulty input containing duplicate keys. You can install it for your
Python (virtual environment) using:
pip install ruamel.yaml
Assuming your file is called input.yaml and it contains:
a: 1 # some duplicate keys follow
app: activity-worker
app: activity-worker
b: "abc"
You can run the following one-liner:
python -c "import sys; from ruamel.yaml import YAML; yaml = YAML(); yaml.preserve_quotes=yaml.allow_duplicate_keys=True; yaml.dump(yaml.load(open('input.yaml')), sys.stdout)"
to get:
a: 1 # some duplicate keys follow
app: activity-worker
b: "abc"
and if your input were like:
{a: 1, app: activity-worker, app:
activity-worker, b: "abc"}
the output would be:
{a: 1, app: activity-worker, b: "abc"}
I'm trying to write a script which cleans up old builds in my generic file repository in Artifactory. I guess the first step would be to look in the repository and check which builds are in there.
Each build shows up as a subfolder of /foo, so for example I have folders /foo/123, /foo/124, /foo/125/, etc.
There doesn't seem to be a ls or dir command. So I tried the search command:
jfrog rt search my-repo/foo/*
But this recursively lists all files, which is not what I'm looking for. I just need the list of direct subfolders. I also tried
jfrog rt search my-repo/foo/* --recursive=false
but this doesn't return any results, because the search command only returns files, not folders.
How do I list the subfolders of a given folder in an Artifactory repository?
Just one more way to do it with curl and jq
curl -s http://myatifactory.domain:4567/artifactory/api/storage/myRepo/myFolder | jq -r '.children[] |select(.folder==true) |.uri'
Explanation: Curl is used to get the folder info and that is piped to JQ which then displays all the uri keys of the children array whose folder key has value true.
Just for easier understanding - the json that curl gets looks something like this (example from artifactory docs)
{
"uri": "http://localhost:8081/artifactory/api/storage/libs-release-local/org/acme",
"repo": "libs-release-local",
"path": "/org/acme",
"created": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ),
"createdBy": "userY",
"lastModified": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ),
"modifiedBy": "userX",
"lastUpdated": ISO8601 (yyyy-MM-dd'T'HH:mm:ss.SSSZ),
"children": [
{
"uri" : "/child1",
"folder" : "true"
},{
"uri" : "/child2",
"folder" : "false"
}
]
}
and for it the output of the command would be /child1.
Of course here it's assumed that artifactory repo myRepo allows anonymous read.
You should have a look to AQL (Artifactory Query Langage) here : https://www.jfrog.com/confluence/display/RTF/Artifactory+Query+Language
as an example the following AQL will retrieve all folders located in "my-repo" under "foo" folder and will display the result ordered by folder's name :
items.find(
{
"type":"folder",
"repo":{"$eq":"my-repo"},
"path":{"$eq":"foo"}
}
)
.include("name")
.sort({"$desc":["name"]})
For cleanup you can also have a look at the following example which gives a list of the 10 biggest artifacts created more than a month ago that have never been downloaded :
items.find(
{
"type":"file",
"repo":{"$eq":"my-repo"},
"created":{"$before":"1mo"},
"stat.downloads":{"$eq":null}
}
)
.include("size","name")
.sort({"$desc":["size"]})
.limit(10)
Based on jroquelaure's answer, I ended up with the following. The key thing that was still missing was that you have to convert the "items.find" call into JSON when putting it in a filespec. There is an example of that in the filespec documentation which I missed at first.
I put this JSON in a test.aql file:
{
"files":
[
{
"aql":
{
"items.find" :
{
"type":"folder",
"repo":{"$eq":"my-repo"},
"path":{"$eq":"foo"}
}
}
}
]
}
Then I call jfrog rt search --spec=test.aql.
The jfrog cli now includes the --include-dirs option for search.
The command:
jf rt search --recursive=false --include-dirs path/
will essentially act like an ls.
By default, it searches for files, if you want to list directories, add one more property --include-dirs
Refer the link for additional parameters. jfrog search
Here is the command.
jf rt search --recursive=false --include-dirs=true path/
Response:
[
{
"path": "artifactory-name/path",
"type": "folder",
"created": "",
"modified": ""
}
]
A cleaner approach is to tell Artifactory about builds, and let it discard old ones.
There are 3 parts to this. My examples are for the jfrog command line utility:
When uploading files with the "jfrog rt upload" command, use the --build-name someBuildName and --build-number someBuildNumber arguments. This links the uploaded files to a certain build.
After uploading files, publish the build with "jfrog rt build-publish someBuildName someBuildNumber"
To clean up all but the 3 latest builds, use "jfrog rt build-discard --max-builds=3 someBuildName"
I'm trying to set up a dictionary as a variable (so I can use it as a Resource and access its values from another file) and there is something that is driving me crazy.
Here is the code I have (just for testing purposes):
*** Settings ***
Documentation Suite description
Library Collections
*** Variables ***
&{SOME DICT} key1=value1 key2=value2
*** Test Cases ***
Dict Test # why $ instead of &?
${RANDOM VAR}= Get From Dictionary ${SOME DICT} key1
Log ${RANDOM VAR} WARN
If I run that, I got the expected result ([ WARN ] value1) BUT the IDE (PyCharm) is complaining about that ${SOME DICT} variable is not defined, and the dictionary declaration is not highlighted the same as variable or a list.
If I change that to &{SOME DICT} the IDE won't complain anymore, but the test fails with the following output:
Dict Test | FAIL |
Keyword 'Collections.Get From Dictionary' got positional argument after named arguments.
That is puzzling me to no end: why I have to use a $ instead of a & if it's a dictionary to make it work? Is there something I am doing wrong and it is just running by luck?
Thanks for any advice or guidance you may have!
Have a look into "Get from Dictionary" libdoc,looks like example is showing the same as your working snippet:
Name: Get From Dictionary
Source: Library (Collections)
Arguments: [dictionary, key]
Returns a value from the given ``dictionary`` based on the given ``key``.
If the given ``key`` cannot be found from the ``dictionary``, this
keyword fails.
The given dictionary is never altered by this keyword.
Example:
| ${value} = | Get From Dictionary | ${D3} | b |
=>
| ${value} = 2
Keyword implementation details are as follows:
try:
return dictionary[key]
except KeyError:
raise RuntimeError("Dictionary does not contain key '%s'." % key)
So indeed, Robot sends representation of dict content and not dict name thus value for key can be returned.
This is the same as direct call in python:
a = {u'key1': u'value1', u'key2': u'value2'}
print(a['key1'])
In the end, libdoc for that KW is not straightforward but your PyCharm plugin for Robot does not work properly in this case.
In RED Robot Editor (Eclipse based), proper case does not rise any warnings in editor, wrong-case provides error marker about arguments (better but still not clear what is exactly wrong. Blame minimalistic libdoc info).
ps. I am lead of RED project to be clear.
Simple Example to Use Key Value Variable in robot framework
Set value to dictionary
Get value from dictionary
&{initValues} Create Dictionary key1=value1 key2=value2
Set To Dictionary ${initValues} key1=newvalue1
Set To Dictionary ${initValues} key2=newvalue2
Set To Dictionary ${initValues} key3=newvalue3
${value} Get From Dictionary ${intialValues} key1
I have quite extensive salt config and I want to be able to see what has changed. If I just run salt '*' state.highstate I got the whole list with things that were present and not changed - like 3 to 4 screens of log. But I'd really like to see only things that changed in the last job.
It doesn't have to work for the salt call, it can also employ salt-run jobs.lookup_jid.
You can set state_verbose: False in /etc/salt/master or /etc/salt/minion.
If you want to shorten the output to one line per state, set state_output: terse.
You can also pass these filters on command line:
salt --state-output=terse '*' state.highstate
If you only want to see changes, you can use state-output=changes or state-output=mixed. The latter one will show more information on a failure.
See the following answers fore more detail: basepi, psarossy
We've also added state_output: mixed which will give you the same output as terse, except if there's a failure, in which case it will give you the more verbose output.
To actually answer the question, yes, there is an output filter for changes only:
salt '*' state.highstate --state-output=changes
This will display one liners for things that are in the right state and the proper output for the changes. ie:
<...>
Name: /etc/sudoers - Function: file.managed - Result: Clean
Name: /etc/timezone - Function: file.managed - Result: Clean
Name: /etc/pki/tls/certs/logstash-forwarder.crt - Function: file.managed - Result: Clean
Name: /etc/init.d/logstash-forwarder - Function: file.managed - Result: Clean
----------
ID: /etc/logstash-forwarder
Function: file.managed
Result: True
Comment: File /etc/logstash-forwarder updated
Started: 14:14:28.580950
Duration: 65.664 ms
Changes:
----------
diff:
---
+++
## -1,6 +1,6 ##
{
"network": {
- "servers": [ "10.0.0.104:5000" ],
+ "servers": [ "10.0.0.72:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
Name: deb http://packages.elasticsearch.org/logstashforwarder/debian stable main - Function: pkgrepo.managed - Result: Clean
Name: logstash-forwarder - Function: pkg.installed - Result: Clean
<...>
There are 2 options, first is to change the state_output in master's configuration file, like mentioned in the accepted answer, and it also possible to override the state output in command line, like:
salt --state-output=mixed \* test.version
As of the following PR that was merged into Salt 2015.8.0 (https://github.com/saltstack/salt/pull/26962) it is now possible to toggle the state_verbose flag from command line when running highstate. This overrides the config you can set in /etc/salt/master that was mentioned in previous answers.
The following command should now display only the changes and errors from a highstate run salt '*' state.highstate --state-verbose=False
You can use the below to shorten the output in one line and then filter that output to show only the changes:
salt -v 'minion' state.highstate test=True --state-output=terse --state-verbose=False