I am not very good at write make files. But I have a need to write a GNUMAKE based test harness. I did some reserch, but I was not able to find anything useful. So I am not even sure where to begin.
TEST_SUITE_DIR:=testSuite
#Get all test names with path & extention
TEST_SCRIPTS_WITH_PATH:=$(wildcard $(TEST_SUITE_DIR)/*.txt)
#Test name with out path and extention
TEST_SCRIPT_NAME:=$(notdir $(patsubst %.txt,%,$(TEST_SCRIPTS_WITH_PATH)))
vpath %.txt $(TEST_SUITE_DIR)
TEST_LOG_FILE:=$(TEST_SCRIPT_NAME)OutPutFile.txt
#This is the program ./bin/programName being tested
PUT:=man
#Test requrements:
#1) Each test in /testSuite dir should have its own logFile
#2) Each testout will be checked against a goldenout file in /goldenOutput dir to see
# if the expected resuls match with the test output
# #3) If the test & golden output file hasnt been modified, we do not want to run that test so
# we can save time
# #4) STDERR should be redirected to a stderr.log
#5) During the regression, if a test failed, test name should be written into the regressionReport.log
.PHONY: clean test
test:
for i in $(TEST_SCRIPTS_WITH_PATH); do \
echo $$i; \
$(PUT) `head -n 1 $$i` > $$iOutPutFile.txt; \
done
#$(foreach i, $(TEST_SCRIPTS_WITH_PATH), $(PUT) `head -n 1 $($i)` > $($i)OutPutFile.txt )
#$(PUT) `head -n 1 $(TEST_SCRIPTS) ` > $(TEST_SCRIPTS)logFile.log
clean:
rm -f *.d $(OBJ_DIR)/*.o $(PROG)
-include *.d
Here is my dataFile.txt(at the moment, I am only trying to get 1 command working),
date
A makefile is a way of automating certain tasks, so you can't do anything with Make until you know how how to do it without Make.
There is more than one way to do what you want (a common situation with Make), and you should think about how you want the makefile to scale. The simplest way to construct that command is probably:
man `head -n 1 dataFile.txt` > logFile.log
So this makefile would suffice:
.PHONY: all
all:
man `head -n 1 dataFile.txt` > logFile.log
Many advances on this are possible, but not until we know what you intend to do beyond this.
I'm not sure I understand what you are trying to do, why do you need dependencies for the test rule (specially that you made it phony).
Also if you want to test content of file you need to use diff not test
Related
rule all:
input:
"../data/A_checkm/{genome}"
rule A_checkm:
input:
"../data/genomesFna/{genome}_genomic.fna.gz"
output:
directory("../data/A_checkm/{genome}")
threads:
16
resources:
mem_mb = 40000
shell:
"""
# setup a tmp working dir
tmp=$(mktemp -d)
mkdir $tmp/ref
cp {input} $tmp/ref/genome.fna.gz
cd $tmp/ref
gunzip -c genome.fna.gz > genome.fna
cd $tmp
# run checking
checkm lineage_wf -t {threads} -x fna ref out > stdout
# prepare output folder
cd {config[project_root]}
mkdir -p {output}
# copy results over
cp -r $tmp/out/* {output}/
cp $tmp/stdout {output}/checkm.txt
# cleanup
rm -rf $tmp
"""
Thank you in advance for your help!
I would like to run checkm on a list of ~600 downloaded genome files having the extension '.fna.gz'. Each downloaded file is saved in a separate folder having the same name as the genome. I would like also to have all the results in a separate folder for each genome and that's why my output is a directory.
When I run this code with 'snakemake -s Snakefile --cores 10 A_checkm', I get the following error:
WorkflowError: Target rules may not contain wildcards. Please specify concrete files or a rule without wildcards at the command line, or have a rule without wildcards at the very top of your workflow (e.g. the typical "rule all" which just collects all results you want to generate in the end).
Anyone could help me identifying the error, please?
You need to provide snakemake with concrete values for the {genome} wildcard. You cannot just leave it open and expect snakemake to work on all the files in some folder of your project just like that.
Determine the filenames/genome values of the files which you want to work on, using glob_wildcards(...). See the documentation for further details.
Now you can use these values to specify in rule all to create all the folders (using your other rule) with those {genome} values:
# Determine the {genome} for all downloaded files
(GENOMES,) = glob_wildcards("../data/genomesFna/{genome}_genomic.fna.gz")
rule all:
input:
expand("../data/A_checkm/{genome}", genome=GENOMES),
rule A_checkm:
input:
"../data/genomesFna/{genome}_genomic.fna.gz",
output:
directory("../data/A_checkm/{genome}"),
threads: 16
resources:
mem_mb=40000,
shell:
# Your magic goes here
If the download is supposed to happen inside snakemake, add a checkpoint for that. Have a look at this answer then.
I want to save some git information in a textfile, which I want to put in the src folder of my project.
In case branch Master is checked out, I only want the date of the latest commit. If any other branch is checked out I want the date and the name of the branch like so:
date-branchname
This is my code:
src/version2.txt:
DATE=$(shell git log -1 --date=format:"%Y.%m.%d" --format="%ad")
BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
ifeq ($(BRANCH), 'Master')
$(DATE) > $#
else
DATE+='-'
DATE+=$(BRANCH)
$(DATE) > $#
endif
I'm new to GNU make and quite confused with its syntax.
I assume my ifeq/else blocks are working fine, since I checked printing a dummy text to the version.txt file while having the Master or some other branch checked out.
I also saw my commands to get the date or the branch are ok, since I can put them to the textfile like so:
git log -1 --date=format:"%Y.%m.%d" --format="%ad" > $#
Only when I want to use variables, it seems the variable is empty, for example
$(DATE) > $#
seems to print an empty string to the textfile.
Also, I don't know if my way of creating the DATE-BRANCH output is correct at all.
I've spent ages trying and would really appreciate some help.
Thanks
You cannot create GNU make variables inside recipes and use them later in the GNU make file because GNU make first parses whole file and then runs recipes
You cannot use GNU make ifeqs inside recipes
% in command lines may cause problems in Windows, so I've added a workaround for it in the following code (skip it if your Makefile should work only for Unix-like systems)
' should not be used in ifeqs constants (GNU make reads it literaly)
src/version2.txt does not depend on anything and therefore will not be regenerated by GNU make (if it already exists), consider .PHONY for the target
Try this:
BRANCH:=$(shell git rev-parse --abbrev-ref HEAD)
ifeq ($(OS),Windows_NT)
P:=%%
else
P:=%
endif
.PHONY : src/version2.txt
ifeq ($(BRANCH),master)
src/version2.txt :
git log -1 --date=format:"$PY.$Pm.$Pd" --format="$Pad" >$#
else
src/version2.txt :
git log -1 --date=format:"$PY.$Pm.$Pd" --format="$Pad-$(BRANCH)" >$#
endif
Note that BRANCH will be computed each time you run make even if the src/version2.txt should not be regenerated.
I have two directories, I would like to do something based on the results of a comparison.
Below is my script
#!/bin/sh
# the script doesn't work below if a the above line says bash
for i in $(\ls -d /data_vis/upload/);
do
diff ${i} /data_vis/upload1/;
done
The output from the above script is
Common subdirectories: /data_vis/upload/2012_06 and /data_vis/upload1/2012_06
Common subdirectories: /data_vis/upload/2012_07 and /data_vis/upload1/2012_07
Only in data_vis/upload/: 2012_08
Only in /data_vis/upload/: 2012_09
Only in /data_vis/upload/: index.php
Only in /data_vis/upload/: index.php~
Question ?
How can I use this this output to do something e.g. see below
Pseudocode
if Only in data_vis/upload/: 2012_08 # e.g if directory only exists in upload directory
then do something
else
do something else
Finish
Any comments or better solutions/commands welcome!
I understood that You want to parse the output of the diff.
First, Your outermost for-loop is not necessary, since the "ls"-operation returns only one item. The task could be done as follows:
#!/bin/sh
diff data_vis/upload/ data_vis/upload1/ | while read line
do
if echo $line | grep "Only in">/dev/null;then
# parse the name of the directory where the not matched dir is located
dironlyin=$(echo $line|awk -F ":" '{split($1,f," ");print f[3];}');
# parse the name of the not matched dir
fileonlyin=$(echo $line|awk -F ":" '{l=length($2);print substr($2,2,l-2);}');
# prove that the parsing worked correctly
echo "do something with file \""$fileonlyin"\" in dir \""$dironlyin"\""
else
# do your own parsing here if needed
echo "do something else with "\"$line\"
fi
done
You need to do the parsing of the lines starting with "Common subdirectories" by yourself. I hope the awk mini-scripts can help You doing it!
Cheers
Jörg
I remember seeing a unix command that would take lines from standard input and execute another command multiple times, with each line of input as the arguments. For the life of me I can't remember what the command was, but the syntax was something like this:
ls | multirun -r% rm %
In this case rm % was the command to run multiple times, and -r% was an option than means replace % with the input line (I don't remember what the real option was either, I'm just using -r as an example). The complete command would remove all files in the current by passing the name of each file in turn to rm (assuming, of course, that there are no directories in the current directory). What is the real name of multirun?
The command is called 'xargs' :-) and you can run it as following
ls | xargs echo I would love to rm -f the files
I have a makefile which runs commands that can take a while. I'd like those commands to be chatty if the build is initiated from an interactive shell but quieter if not (specifically, by cron). Something along the lines of (pseudocode):
foo_opts = -a -b -c
if (make was invoked from an interactive shell):
foo_opts += --verbose
all: bar baz
foo $(foo_opts)
This is GNU make. If the specifics of what I'm doing matter, I can edit the question.
It isn't strictly determining whether it is invoked from an interactive shell or not, but for a cron job in which the output is redirected to a file, the answer to this question would be the same as for How to detect if my shell script is running through a pipe?:
if [ -t 0 ]
then
# input is from a terminal
fi
Edit: To use this to set a variable in a Makefile (in GNU make, that is):
INTERACTIVE:=$(shell [ -t 0 ] && echo 1)
ifdef INTERACTIVE
# is a terminal
else
# cron job
endif
http://www.faqs.org/faqs/unix-faq/faq/part5/section-5.html
5.5) How can I tell if I am running an interactive shell?
In the C shell category, look for the variable $prompt.
In the Bourne shell category, you can look for the variable $PS1,
however, it is better to check the variable $-. If $- contains
an 'i', the shell is interactive. Test like so:
case $- in
*i*) # do things for interactive shell
;;
*) # do things for non-interactive shell
;;
esac
I do not think you can easily find out. I suggest adopting an alternative strategy, probably by quelling the verbose output from the cron job. I would look to do that using a makefile like this:
VERBOSE = --verbose
foo_opts = -a -b -c ${VERBOSE}
all: bar baz
foo $(foo_opts)
Then, in the cron job, specify:
make VERBOSE=
This command-line specification of VERBOSE overrides the one in the makefile (and cannot be changed by the makefile). That way, the specialized task (cron job) that you set up once and use many times will be done without the verbose output; the general task of building will be done verbosely (unless you elect to override the verbose-ness on the command line).
One minor advantage of this technique is that it will work with any variant of make; it does not depend on any GNU Make facility.
I’m not really sure what "am interactive" means. Do you mean if you have a valid /dev/tty? If so, then you could check that. Most of us check isatty on stdin, though, because it answers the questions we want to know: is there someone there to type something.
Just a note: you can also see the related discussion that I had about detecting redirection of STDOUT from inside a Makefile.
I believe it will be helpful to readers of this question - executive summary:
-include piped.mk
all: piped.mk
ifeq ($(PIPED),1)
#echo Output of make is piped because PIPED is ${PIPED}
else
#echo Output of make is NOT piped because PIPED is ${PIPED}
endif
#rm -f piped.mk
piped.mk:
#[ -t 1 ] && PIPED=0 || PIPED=1 ; echo "PIPED=$${PIPED}" > piped.mk
$ make
Output of make is NOT piped because PIPED is 0
$ make | more
Output of make is piped because PIPED is 1
In my answer there I explain why the [-t 1] has to be done in an action and not in a variable assignment (as in the recommended answer here), as well as the various pitfalls regarding re-evaluation of a generated Makefile (i.e. the piped.mk above).
The term interactive in this question seems to imply redirection of STDIN... in which case replacing [ -t 1 ] with [ -t 0 ] in my code above should work as-is.
Hope this helps.