Adding files that rely on pipeline outputs - dvc

Question

In my workflow, I do the following:
Acquire raw data (e.g. a video containing people)
Transform it (e.g. automatically extract all crops with faces)
Manually label them (e.g. identify the person in each crop). The labels are stored in json files along with the crops.
Train a model on these data.
How should I track this pipeline with DVC?
My concerns:
If stage 2 is changed (e.g. crops are extracted with a different size), the manual data should be invalidated (and so should the final model).
The 3rd step is manual and therefore not precisely reproducible. But I do need its input to be reproducible.
Stage 4 has an element of randomness, so it's not precisely reproducible either.

Answer 1

Stage 3 is manual so you can't really codify it or automate it, nor guarantee its reproducibility (due to possible human error). But there's a way to get you as close as possible:
You could replace it with a helper script that just checks whether all the labels are annotated. If so, output a text file with content "green", otherwise "red" (for example) and error out.
Stage 4 should depend on both the inputs from stages 2 and 3, so it will only run if BOTH the face crops changed AND if they are thoroughly annotated.
Internally, it first checks the semaphore file (from 3) and dies on red. On green, it trains the model :)
The DAG looks like this:
+-----------+
| 1-acquire |
+-----------+
*
*
*
+---------+
| 2-xform |
+---------+
you ** **
--> ** **
* **
+---------+ *
| 3-check | **
+---------+ **
** **
** **
* *
+---------+
| 4-train |
+---------+
re randomness: while not ideal, non-determinism technically only affects intermediate stages of the pipeline, because it causes everything after that to always run. In this case, since it's in the last stage, it won't affect DVC's job.