Can my unit test class keep track of test execution - integration-testing

Question

Our code defines some "rules" in a List<Rule> collection. Each rule contains some logic in the form of a string. The rules are all passed to a "rule engine" along with some data. The rule engine sequentially evaluates the data against rules until it finds a rule which evaluates as true, then it returns that Rule.
We want to make to automatically test every rule. The tests will actually be integration tests, rather than unit tests, because they'll test the combination of the rule engine and the rules.
How do I write a test that says "make sure each rule evaluates as true in at least one unit test"?
I've figured out a way to run some fixture teardown code after all the tests have run (see https://xunit.github.io/docs/shared-context.html#class-fixture), and by using a static variable to record evaluated rules I can check during the teardown whether all the rules have been returned during unit tests. But this approach has the undesirable effect that it causes individual test to report as failed (in teardown) which didn't actually fail.

Answer 1

TLDR: control the test sequence. Since these are integration tests, that's not the big no-no it would be if they were unit tests.
I found that by using a static variable to record the rules that evaluated as true, and ensuring the have-all-rules-evaluated-as-true test last, as per the advice here https://hamidmosalla.com/2018/08/16/xunit-control-the-test-execution-order/, it becomes trivially easy to achieve what I needed.
In case the link dies, here's how to implement this approach
First, create an AlphabeticalOrderer implementing xUnit's ITestCaseOrderer:
public class AlphabeticalOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases)
where TTestCase : ITestCase
{
var result = testCases.ToList();
result.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
return result;
}
}
Then use the newly-created attribute on your test class, and create a static variable to record progress:
[TestCaseOrderer("MyCompany.AlphabeticalOrderer", "MyTestsDLL")]
public class RulesTests
{
private static List<Rule> _untestedRules;
public RulesTests()
{
if (_untestedRules == null) // this will run once per [Fact]
{
_untestedRules = <some way of getting all the rules> // this will only run first time around
}
}
Each time a rule triggers, record which rule it was:
private void MarkRuleAsTested(LendingRule testedRule)
{
var rulesToRemove = _untestedRules.Where(r => r.Logic == testedRule.Logic).ToList();
foreach (var ruleToRemove in rulesToRemove)
{
_untestedRules.Remove(ruleToRemove);
}
}
Then the last test to run should check the collection:
[Fact]
public void ZZ_check_all_rules_have_been_triggered_by_other_tests()
{
// This test must run last because it validates that the
// OTHER tests taken together have resulted in each rule
// being evaluated as true at least once.
if (!_untestedRules.Any())
{
return;
}
var separator = "\r\n------------------------\r\n";
var untestedRules = string.Join(separator, _untestedRules.Select(ur => $"Logic: {ur.Logic}"));
throw new SomeRulesUntestedException($"The following rules have not been tested to resolve as True: {separator}{untestedRules}{separator}");
}
This results in a nicely-readable test failure.
You could equally start with an empty static collection of _testedRules and at the end compare that set to the full set of rules.