Parallel execution of tests on TFS - .net-core

Question

We use TFS on our project. I have set Parallelism -> Multi Agent in the phase settings. The command itself to run (.NET Core) is:
dotnet test --filter TestCategory="Mobile" --logger trx -m:1.
Do I understand correctly that these settings will not split the tests between the two agents, but run the command above on the two agents?

Answer 1

The Visual Studio Test (- task: VSTest#2) has built-in magic to distribute the test based on configurable criteria:
You could switch to using the vstest task instead; to run your tests to get this "magic".
The dotnet core task or invoking dotnet straight from the command line doesn't have this magic.
There is a github repo that shows how to take advantage of the default of the hidden variables that are set by the agent when running in parallel:
#!/bin/bash
filterProperty="Name"
tests=$1
testCount=${#tests[#]}
totalAgents=$SYSTEM_TOTALJOBSINPHASE
agentNumber=$SYSTEM_JOBPOSITIONINPHASE
if [ $totalAgents -eq 0 ]; then totalAgents=1; fi
if [ -z "$agentNumber" ]; then agentNumber=1; fi
echo "Total agents: $totalAgents"
echo "Agent number: $agentNumber"
echo "Total tests: $testCount"
echo "Target tests:"
for ((i=$agentNumber; i <= $testCount;i=$((i+$totalAgents)))); do
targetTestName=${tests[$i -1]}
echo "$targetTestName"
filter+="|${filterProperty}=${targetTestName}"
done
filter=${filter#"|"}
echo "##vso[task.setvariable variable=targetTestsFilter]$filter"
This way you can slice the tasks in your pipeline:
- bash: |
tests=($(dotnet test . --no-build --list-tests | grep Test_))
. 'create_slicing_filter_condition.sh' $tests
displayName: 'Create slicing filter condition'
- bash: |
echo "Slicing filter condition: $(targetTestsFilter)"
displayName: 'Echo slicing filter condition'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*Tests/*Tests.csproj'
arguments: '--no-build --filter "$(targetTestsFilter)"'
I'm not sure whether this will support 100.000's of tests. In that case you may have to break the list into batches and call dotnet test multiple times in a row. I couldn't find support for vstest playlists.