I am running a pipeline and was trying to optimize it by declaring the paths in a config file (config.yaml). The config.yaml file contains the path to find the scripts to run inside the pipeline, but when I expand the wildcard of the path, the pipeline does not run the script. The script itself runs fine.
To explain my problem:
rule with_script:
input: someinput
output: someoutput
script: expand("{script_path}/scriptfile", script_path = config[scriptpath])
input, output or rule all do not contain the script's path wildcard, so here is the first time I'm declaring it. The config.yaml line that contains the path looks like this:
scriptpath: /path/to/the/script
is there a way to maintain the wildcard and config file path (to make it easier for others to make changes if needed) and have the script work? Like this snakemake doesn't even enter the script file. Or maybe it is possible to declare global wildcards outside the rule all?
Thank you for your help!
P.S.: I'm sorry if this question has already been answered, but I couldn't find anything to help me with this.
You cannot define a function like expand() in the script section. Snakemake expects a path to your script.
Like the documentation states:
The script path is always relative to the Snakefile containing the directive (in contrast to the input and output file paths, which are relative to the working directory). It is recommended to put all scripts into a subfolder "scripts"
If you need to define different paths to your scripts, you can always do it in python outside of your rules. Don't forget, all python code outside of rules is executed before building the DAG. Thus, you can define all variables you want and use them in your rules.
SCRIPTSPATH = config["scriptpath"]
rule with_script:
input: someinput
output: someoutput
script: "{SCRIPTSPATH}/scriptfile"
Note:
Do not mix wildcards and "variables". In an expand function as
expand("{script_path}/scriptfile", script_path = config[scriptpath])
{script_path} is not a wildcard but just a placeholder for the values given in the second parameter of the function.
Related
I would like to use wildcards in Snakemake in a very simple way to start a script for two datasets. Unfortunately, I cannot find the proper way of doing it.
My data folder contains three files: gene_list.txt, expression_JGI.txt, expression_UBC.txt.
Here is what my snakefile looks like:
rule extract:
input:
genes="data/gene_list.txt",
expression="data/expression_{dataset}.txt"
output:
"data/expression_{dataset}_subset.txt"
shell:
"bash scripts/extract.sh {input.genes} {input.expression} {output}"
When I use snakemake -c1 extract I get the following error message:
Building DAG of jobs...
WorkflowError:
Target rules may not contain wildcards. Please specify concrete files or a rule without wildcards at the command line, or have a rule without wildcards at the very top of your workflow (e.g. the typical "rule all" which just collects all results you want to generate in the end).
I tried adding a rule all at the beginning of the snakefile with the desired result files as input without success:
rule all:
input:
"data/expression_JGI_subset.txt",
"data/expression_UBC_subset.txt"
I also tried with expand:
DATASETS = ["JGI", "UBC"]
rule all:
input:
expand("data/expression_{dataset}_subset.txt", dataset=DATASETS)
But I get the same error message.
The script works fine when I use it outside Snakemake.
How can I achieve what I want?
When you do snakemake -c1 extract you ask snakemake to execute only rule extract and its dependencies, if any. However, because extract contains wildcards snakemake doesn't know what to replace them with. (Note that rule all is not a dependency of extract).
So either execute snakemake -c1 to run the whole pipeline or specify the concrete files you want to generate, e.g.:
snakemake -c1 -- data/expression_JGI_subset.txt data/expression_UBC_subset.txt
I'm using the File Should Exist keyword to check for an existing file in a List.
This is the high-level format:
Variables:
#{Files} /Desktop/Other/scooby.txt
... /Desktop/DummyFiles/daffy.txt
... %{CURDIR}/mickey.txt
Test Case:
Given File Should Exist ${Files[0]}
[...]
Output:
Setup failed:
Path '/Desktop/Other/scooby.txt' does not exist.
I'm not sure why this happens. The file name and filepath are correct. I also tried a bunch of different combinations (I copied the file over to the subdirectory this script is running from but that also doesn't work). I tried making the filepath a scalar variable instead of a List/array. Adding ".." in front of the file path doesn't work either. I've looked into "Glob pattern" but I don't think that's the issue either.
Always use absolute paths if in doubt. For example - /Desktop/Other/scooby.txt Does not point to any "meaningful" path even on windows because its lacking the the drive. On windows, using C:/Users/$yourlocalusername/Desktop/Other/scooby.txt might be working (replace $yourlocalusername with correct value)
Relying on relative paths (like in your example, 2 first ones are relative even thought you start with /, because in windows you still have a drive at the start) - you will need to ensure that working directory is set to a specific directory before you run your suite. Now, if you run your suite via IDE, current working directory might not be what you expect it to be so before you are familiar with how relative & absolute paths work - prefer to use absolute paths. See https://www.computerhope.com/issues/ch001708.htm - maybe that will clear out your issue.
You can either use ${CURDIR} or do not start the relative path with '/'.
I guess when starting with '/' , RF does not take this as relative path input and try to map the address from root directory i.e. C:
In below example, I have demonstrated both the approach, and it works fine for me.
*** Settings ***
Library OperatingSystem
Documentation Demonstrating File Exist keywords
*** Variables ***
#{Files} Data/VariableData/ConditionalFlows.robot
... Data/VariableData/testdata.robot
#{Files2} ${CURDIR}/Data/VariableData/ConditionalFlows.robot
... ${CURDIR}/Data/VariableData/testdata.robot
*** Test Cases ***
TS002-Check For File Existence
[Documentation] File exists
${File} File Should Exist ${Files}[0]
${File} File Should Exist ${Files2}[0]
For Python there is an option to add custom CSS to a Dash app. The method seems quite straightforward, it says
Just create a folder named assets in the root of your app directory
and include your CSS and JavaScript files in that folder. Dash will
automatically serve all of the files that are included in this folder.
By default the url to request the assets will be /assets but you can
customize this with the assets_url_path argument to dash.Dash
source: https://dash.plotly.com/external-resources
However when I try to do so in Julia, nothing happens.
Is this feature a thing in Julia? If not, how can I do so?
Found a hack, have no idea if this is the correct way though...
Essentially tried to find the inputs to app = dash() by going methods(dash)
julia> methods(dash)
# 1 method for generic function "dash":
[1] dash(; external_stylesheets, external_scripts, url_base_pathname, requests_pathname_prefix, routes_pathname_prefix, assets_folder, assets_url_path, assets_ignore, serve_locally, suppress_callback_exceptions, prevent_initial_callbacks, eager_loading, meta_tags, index_string, assets_external_path, include_assets_files, show_undo_redo, compress, update_title) in Dash at C:\Users\<User>\.julia\packages\Dash\Weukk\src\app\dashapp.jl:291
where I noticed a arg assets_folder. Putting in the absolute path seems to work so the full code looks something like this
using Dash
app = dash(assets_folder="/absolute/path/to/assets")
I'm using Google Closure Compiler to minify my JS scripts: https://developers.google.com/closure/compiler/docs/gettingstarted_app?hl=en
The command I'm using is:
java -jar /home/user/compiler/compiler.jar --js $File::Find::name --create_source_map $File::Find::name.map --source_map_format=V3 --compilation_level=WHITESPACE_ONLY --js_output_file $minified --charset=Windows-1251 --output_wrapper '%output%\n//# sourceMappingURL=output.js.map'
Thats fine, apart from one thing - the .js.map file has the FULL path for the file, not the relative one:
"version":3,
"file":"/home/user/public_html/new_design/common37.min.js",
"lineCount":375,
....
I assume I can change this in the invocation of the compiler.jar script? Otherwise, I guess I will have to add some more code into my script (not something I want to do, if its possible "out of the box")
EDIT: I've done a little bit of a dirty hack in my Perl script:
# now open the map file one, and edit it to remove the full path.. needs to be relative
my $contents = File::Slurp::read_file("/home/user/public_html/$tmp.map");
$contents =~ s|/home/user/public_html||g;
File::Slurp::write_file("/home/user/public_html/$tmp.map",$contents);
That gets rid of the path info correctly. I've prefer if there were an option to use relative urls in the .map file (compared to the full path it currently puts in)
Thanks!
Specify sourcemap location transformations by using the --source_map_location_mapping flag. The flag expects a value formatted as:
--source_map_location_mapping=/filesystem/src/root|relative/source/root
How can I compile a LESS file to output a source map file (.css.map) in addition to a CSS file? Is there a way to do it on both command line (NodeJS's lessc) and on any GUI-based programs?
Update: New shortest answer
The docs have been updated! As new features hit LESS, sometimes the docs lag behind a bit, so if you're looking for bleeding-edge features, you're still probably better off running lessc (see longer answer) and checking what pops out of the help text.
http://lesscss.org/usage/
Short answer
You're looking for any number of the following options from the command line:
--source-map[=FILENAME] Outputs a v3 sourcemap to the filename (or output filename.map)
--source-map-rootpath=X adds this path onto the sourcemap filename and less file paths
--source-map-basepath=X Sets sourcemap base path, defaults to current working directory.
--source-map-less-inline puts the less files into the map instead of referencing them
--source-map-map-inline puts the map (and any less files) into the output css file
--source-map-url=URL the complete url and filename put in the less file
As I write this I'm not aware of any GUI options that generate maps (source maps were only added to LESS in the last few months) -- sorry to not have any better news. I'm sure they'll add support in as they update over the next year.
Longer answer
If you run lessc from the command line without any parameters it will give you all the options. (In my experience, this is more up to date than their documentation, so it'll at least get you pointed in the right direction.) with all the most recent map stuff included.
The easiest combo to use for dev is --source-map-less-inline --source-map-map-inline as that will give you your source maps embedded in your output css.
If you'd like to add a separate map file, you can use --source-map which, from my.less will output my.css and my.css.map
For reference: when I run my copy (v 1.6.1 at the moment) I get
usage: lessc [option option=parameter ...] <source> [destination]
If source is set to `-' (dash or hyphen-minus), input is read from stdin.
options:
-h, --help Print help (this message) and exit.
--include-path=PATHS Set include paths. Separated by `:'. Use `;' on Windows.
-M, --depends Output a makefile import dependency list to stdout
--no-color Disable colorized output.
--no-ie-compat Disable IE compatibility checks.
--no-js Disable JavaScript in less files
-l, --lint Syntax check only (lint).
-s, --silent Suppress output of error messages.
--strict-imports Force evaluation of imports.
--insecure Allow imports from insecure https hosts.
-v, --version Print version number and exit.
-x, --compress Compress output by removing some whitespaces.
--clean-css Compress output using clean-css
--clean-option=opt:val Pass an option to clean css, using CLI arguments from
https://github.com/GoalSmashers/clean-css e.g.
--clean-option=--selectors-merge-mode:ie8
and to switch on advanced use --clean-option=--advanced
--source-map[=FILENAME] Outputs a v3 sourcemap to the filename (or output filename.map)
--source-map-rootpath=X adds this path onto the sourcemap filename and less file paths
--source-map-basepath=X Sets sourcemap base path, defaults to current working directory.
--source-map-less-inline puts the less files into the map instead of referencing them
--source-map-map-inline puts the map (and any less files) into the output css file
--source-map-url=URL the complete url and filename put in the less file
-rp, --rootpath=URL Set rootpath for url rewriting in relative imports and urls.
Works with or without the relative-urls option.
-ru, --relative-urls re-write relative urls to the base less file.
-sm=on|off Turn on or off strict math, where in strict mode, math
--strict-math=on|off requires brackets. This option may default to on and then
be removed in the future.
-su=on|off Allow mixed units, e.g. 1px+1em or 1px*1px which have units
--strict-units=on|off that cannot be represented.
--global-var='VAR=VALUE' Defines a variable that can be referenced by the file.
--modify-var='VAR=VALUE' Modifies a variable already declared in the file.
-------------------------- Deprecated ----------------
-O0, -O1, -O2 Set the parser's optimization level. The lower
the number, the less nodes it will create in the
tree. This could matter for debugging, or if you
want to access the individual nodes in the tree.
--line-numbers=TYPE Outputs filename and line numbers.
TYPE can be either 'comments', which will output
the debug info within comments, 'mediaquery'
that will output the information within a fake
media query which is compatible with the SASS
format, and 'all' which will do both.
--verbose Be verbose.
If the command line doesn't suite you, Grunt is great at this type of thing. You can configure the grunt-contrib-less plugin to generate inline maps with a config like this:
less: {
options: {
sourceMap:true,
outputSourceFiles: true
},
lessFiles: {
expand: true,
flatten:false,
src: ['**/*.less'],
dest: ['dist/'],
ext: '.css',
}
},
https://github.com/gruntjs/grunt-contrib-less
Example to Create Map and CSS file from Less File
Install latest Node JS and go to command prompt and run npm install less, Now less installed successfully
Go to Command Prompt and move to less file folder that we are going to create
For e.g., I am going to change HelloWorld [Less File]
In Command prompt go to C:\Project\CSS or give the correct path in the below command.
Run following Command in Command Prompt
lessc HelloWorld.less HelloWorld.css --source-map=HelloWorld.css.map –verbose
Now CSS and Map file is generated in the respective folder.
For more reference check the link : royalarun.blogspot.com