In snakemake, how do you use wildcards with scatter-gather processes? - wildcard

Question

I am trying to use snakemake's scatter-gather functionality to parallelize a slow step in my workflow. However, I cannot figure out how to apply it in situations where I am using wildcards. For example, I have defined the wildcard library in rule all, however, this does not seem to apply to the scatter function in ScatterIntervals:
import re
SCATTER_COUNT = 100
scattergather:
split=SCATTER_COUNT
rule all:
input:
expand("{library}_output.txt", library=["FC19271512", "FC19271513"])
rule ScatterIntervals:
input:
"{library}_baits.interval_list"
output:
temp(scatter.split("tmp/{library}_baits.scatter_{scatteritem}.interval_list"))
params:
output_prefix = (
lambda wildcards, output:
re.sub("\.scatter_\d+\.interval_list", "", output[0])
),
scatter_count = SCATTER_COUNT
shell:
"""
python ScatterIntervals.py \
-i {input} \
-o {params.output_prefix} \
-s {params.scatter_count}
"""
rule ProcessIntervals:
input:
bam = "{library}.bam",
baits = "tmp/{library}_baits.scatter_{scatteritem}.interval_list"
output:
temp("tmp/{library}_output.scatter_{scatteritem}.txt")
shell:
"""
python ProcessIntervals.py \
-b {input.bam} \
-l {input.baits} \
-o {output}
"""
rule GatherIntervals:
input:
gather.split("tmp/{library}_output.scatter_{scatteritem}.txt")
output:
"{library}_output.txt"
run:
inputs = "-i ".join(input)
command = f"python GatherOutputs.py {inputs} -o {output[0]}"
shell(command)
WildcardError in line 16 of Snakefile:
No values given for wildcard 'library'.

Answer 1

Evidently this works like expand, in that you can quote the wildcards that aren't scatteritem if you want DAG resolution to deal with them:
temp(scatter.split("tmp/{{library}}_baits.scatter_{scatteritem}.interval_list"))
The same logic applies for gather.split.