sbt does not differenciate between flags given to test runners - sbt

Question

Currently:
Sbt has multiple test runners: scalaTest, junit-interface, etc..
Each test runner has it's own set of flags (scalaTest flags, junit-interface flags).
You can pass flags through sbt to the test runners, for example:
$ sbt '<project>/test-only * -- -f <out_file>' (-f is a scalaTest flag)
However, the flags seem to be passed to all test runners, even if a flag is not compatible with all test runners.
I'm also experiencing behavior contrary to what I found in the documentation. ScalaTest says the -v flag will "print the ScalaTest version" and junit-interface says it will "Log "test run started" / "test started" / "test run finished" events on log level "info" instead of "debug"." Instead ScalaTest throws an unrecognised flag exception.
$ sbt '<project>/test-only * -- -v'
java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: -v
at org.scalatest.tools.ArgsParser$.parseArgs(ArgsParser.scala:425)
at org.scalatest.tools.Framework.runner(Framework.scala:929)
...
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[error] (elasticSearchDriver/test:testOnly) java.lang.IllegalArgumentException: Argument unrecognized by ScalaTest's Runner: -v
[error] Total time: 1 s, completed Aug 15, 2017 11:12:56 AM
Question:
What is the actual underlying behavior of the flags passed to the test runners through sbt? Is there a bit of documentation that explains what's going on?

Answer 1

By looking at SBT (0.13.x) we eventually get to a part where:
def inputTests(key: InputKey[_]): Initialize[InputTask[Unit]] = inputTests0.mapReferenced(Def.mapScope(_ in key.key))
private[this] lazy val inputTests0: Initialize[InputTask[Unit]] =
{
val parser = loadForParser(definedTestNames)((s, i) => testOnlyParser(s, i getOrElse Nil))
Def.inputTaskDyn {
val (selected, frameworkOptions) = parser.parsed
val s = streams.value
val filter = testFilter.value
val config = testExecution.value
implicit val display = Project.showContextKey(state.value)
val modifiedOpts = Tests.Filters(filter(selected)) +: Tests.Argument(frameworkOptions: _*) +: config.options
val newConfig = config.copy(options = modifiedOpts)
val output = allTestGroupsTask(s, loadedTestFrameworks.value, testLoader.value, testGrouping.value, newConfig, fullClasspath.value, javaHome.value, testForkedParallel.value, javaOptions.value)
val taskName = display(resolvedScoped.value)
val trl = testResultLogger.value
val processed = output.map(out => trl.run(s.log, out, taskName))
Def.value(processed)
}
}
Notice this line: Tests.Filters(filter(selected)) +: Tests.Argument(frameworkOptions: _*) +: config.options
By reading this I deduce that sbt passes the arguments you pass to it to all the underlying testing frameworks.
Solution
Don't pass test framework flags in your commands. Configure them in your *.sbt files like:
testOptions in Test += Tests.Argument(TestFrameworks.ScalaCheck, "-f")
Documentation on test framework arguments