I've got a method that contains logic that varies between release mode and debug mode. It is not advanced at all, but i still want a unit test for it, since my application will be used in a bigger picture and i want to redirect the user to other sites if it is not used in release mode.
And now to my question, is there any way to force the unit test to run in Release Mode? I don't want to manually change build configuration for every time i want to run my unit tests.
Instead of running your unit tests in Release mode you can create a test seam so you can control which behavior you want to elicit. You might be able to something like this:
public class Foo {
public int Bar() {
if (IsDebugModeEnabled()) {
return 1;
} else {
return 0;
}
}
public boolean IsDebugModeEnabled() {
#if DEBUG
return true;
#else
return false;
#endif
}
}
This way you have a couple of options to test both paths of your logic. You can create a subclass Foo and override IsDebugModeEnabled or use a partial mock to directly set the return value.
Related
Our code defines some "rules" in a List<Rule> collection. Each rule contains some logic in the form of a string. The rules are all passed to a "rule engine" along with some data. The rule engine sequentially evaluates the data against rules until it finds a rule which evaluates as true, then it returns that Rule.
We want to make to automatically test every rule. The tests will actually be integration tests, rather than unit tests, because they'll test the combination of the rule engine and the rules.
How do I write a test that says "make sure each rule evaluates as true in at least one unit test"?
I've figured out a way to run some fixture teardown code after all the tests have run (see https://xunit.github.io/docs/shared-context.html#class-fixture), and by using a static variable to record evaluated rules I can check during the teardown whether all the rules have been returned during unit tests. But this approach has the undesirable effect that it causes individual test to report as failed (in teardown) which didn't actually fail.
TLDR: control the test sequence. Since these are integration tests, that's not the big no-no it would be if they were unit tests.
I found that by using a static variable to record the rules that evaluated as true, and ensuring the have-all-rules-evaluated-as-true test last, as per the advice here https://hamidmosalla.com/2018/08/16/xunit-control-the-test-execution-order/, it becomes trivially easy to achieve what I needed.
In case the link dies, here's how to implement this approach
First, create an AlphabeticalOrderer implementing xUnit's ITestCaseOrderer:
public class AlphabeticalOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases)
where TTestCase : ITestCase
{
var result = testCases.ToList();
result.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
return result;
}
}
Then use the newly-created attribute on your test class, and create a static variable to record progress:
[TestCaseOrderer("MyCompany.AlphabeticalOrderer", "MyTestsDLL")]
public class RulesTests
{
private static List<Rule> _untestedRules;
public RulesTests()
{
if (_untestedRules == null) // this will run once per [Fact]
{
_untestedRules = <some way of getting all the rules> // this will only run first time around
}
}
Each time a rule triggers, record which rule it was:
private void MarkRuleAsTested(LendingRule testedRule)
{
var rulesToRemove = _untestedRules.Where(r => r.Logic == testedRule.Logic).ToList();
foreach (var ruleToRemove in rulesToRemove)
{
_untestedRules.Remove(ruleToRemove);
}
}
Then the last test to run should check the collection:
[Fact]
public void ZZ_check_all_rules_have_been_triggered_by_other_tests()
{
// This test must run last because it validates that the
// OTHER tests taken together have resulted in each rule
// being evaluated as true at least once.
if (!_untestedRules.Any())
{
return;
}
var separator = "\r\n------------------------\r\n";
var untestedRules = string.Join(separator, _untestedRules.Select(ur => $"Logic: {ur.Logic}"));
throw new SomeRulesUntestedException($"The following rules have not been tested to resolve as True: {separator}{untestedRules}{separator}");
}
This results in a nicely-readable test failure.
You could equally start with an empty static collection of _testedRules and at the end compare that set to the full set of rules.
My reading of the TestNG docs suggests that if I have a test method marked like this:
#BeforeSuite(alwaysRun = true)
#Test
public void MyTestMethod { ... }
then MyTestMethod would run before any other test defined anywhere, regardless of class, suite or group. But that does not seem to be the case.
Is there a way to make a test method run unconditionally before everything else? (And that if it fails no other tests will be run.)
Edit:
The test class:
class Suite_Setup
extends BaseTestSuite
{
#BeforeSuite(alwaysRun = true)
def setup() {
System.out.println("Conducting test suite setup...")
// Verify that the internal API is configured properly and that the API host is available...
new Action(ApiHostname, new BasicCookieStore)
}
}
Edit:
The answer:
We generate our own TestNG.xml files (automatically) and the #BeforeSuite method was not being included in it. Once it was included, #BeforeSuite had the expected effect.
However, it does appear that both #BeforeSuite (and presumably other #Before... and #After... annotations) can be mixed with #Test and rather than inhibiting the execution of the annotated method, they cause it to run more than once!
Also, I remiss in not indicating which version of TestNG I'm using. It's 6.2.
Try using groups either on class level or on method level.
In my case, I have a set of smoke tests that need to run before everything and if any of those fail, no other tests should run.
#Test(groups="smoke")
public class SmokeTests {
public void test1 {
//
}
...
}
And all other classes with tests are:
#Test(dependsOnGroups = "smoke")
public class OtherTests {
public void test1 {
//
}
...
}
I think that the answer is
to separate the configuration-methods from the test-methods,
to use #BeforeSuite for the method to be executed befor all tests in the suite (for example to get an EntityManager)
to use dependsOnMethods to decide in which order the tests shall be run
In the following example the testRemove will run after testInsert, testReadAll and testUpdate have run.
testReadAll will run after testInsert has run.
testUpdate will run after testInsert has run.
Note that there is no dependency between testReadAll and testUpdate.
#Test
public void testInsert()
{..}
#Test(dependsOnMethods={"testInsert"})
public void testUpdate()
{..}
#Test(dependsOnMethods={"testInsert"})`
public void testReadAll()`
{..}
#Test(dependsOnMethods={"testInsert", "testUpdate", "testReadAll"})`
public void testRemove()`
{..}
Remove #Test, a method cannot be both a configuration and a test method.
#Test (alwaysRun=True)
makes the test always run irrespective of methods or groups it depends on and even it is failed
Is there a way to mark a test method so it will unconditionally be run before everything else?
Just try to assign to target test the lowest priority value.
#Test(priority = -2147483648)
public void myTest() {
...
}
You can read more about TestNG test priority over there.
And that if it fails no other tests will be run.
You need to make other tests depending on this test method by using one of the following options:
Assign to your first method some group and mark other tests by dependsOnGroup.
Mark other tests by dependsOnMethod.
If you will use one of the dependency options you do not need to provide priority for your first method.
I want to declaratively turn on and off logger for my Flex application and it seems that my usual way of determining if it's debug mode or not works only sometimes. The code for it:
isDebugSWF = new Error().getStackTrace().search(/:[0-9]+]$/m) > -1;
Do you know a better way for it?
Edit:
I posted answer below.
The static property Capabilities.isDebugger in the flash.system.Capabilities class specifies if the Flash player installed is a debug version or not. It requires Flash Player 9 or AIR 1.0 as the minimum version.
Keep in mind though that debug Flash Player installers are publicly available, and that there is nothing stopping users from installing the debug version of Flash. If have something sensitive that you want to hide, using conditional compilation would be a better option.
The previous approach was based on great article . One of the comments suggested that in Release the stacktrace is null, so I modified it properly:
protected function configureLogger() : void
{
if(!isDebugPlayer()|| !isDebugBuild())
{
// stop logging
Logger.hide = true;
}
else
{
// resume logging
Logger.hide = false;
}
}
private function isDebugPlayer() : Boolean
{
return Capabilities.isDebugger;
}
/**
* Returns true if the swf is built in debug mode
**/
private function isDebugBuild() : Boolean
{
var stackTrace:String = new Error().getStackTrace();
if(stackTrace == null || stackTrace.length == 0)
{
//in Release the stackTrace is null
return false;
}
//The code searches for line numbers of errors in StackTrace result.
//Only StackTrace result of a debug SWF file contains line numbers.
return stackTrace.search(/:[0-9]+]$/m) > -1;
}
This way I can finally configure ThunderBoltAS3 logger depending on current build type.
Looks like you are looking for the same, as answered in another post. Look here: How can you tell programmatically if a Flex App is running in debug mode?
We are using conditional compilation at our project :
Adobe Documentation
I want to pass tests which get the following: "This test did not perform any assertions"
I know I could add something like assertTrue(true) however is it possible to add something to the config to make these tests pass cleaner?
I'm pretty sure this only happens since version PHPUnit 3.5.0 with the introduction of
--strict
Edit: You've got a few choices depending on which version you're using, whether you want to ignore all risky tests or just a few, and if you want it to be permanent or temporary.
Prior to 5.6, if you didn't want to add bogus assertions to all your tests, you had to avoid passing --strict to PHPUnit or add strict="false" to your phpunit.xml. The point of this option is to "Mark a test as incomplete if no assertions are made."
At some point, PHPUnit added the related --dont-report-useless-tests command-line switch and beStrictAboutTestsThatDoNotTestAnything="false" config option. I haven't checked if they are replacements or additional fine-grained versions.
The above options affect all risky tests. Using them will leave you open to accidentally writing tests without assertions. The following new options are safer as you have to purposefully mark each risky test that you wish to allow.
PHPUnit 5.6 added the #doesNotPerformAssertions annotation to mark individual test cases as "not risky" even though they perform no assertions.
/**
* #doesNotPerformAssertions
*/
public function testWithoutAsserting() {
$x = 5;
}
PHPUnit 7.2 introduced TestCase::expectNotToPerformAssertions() which does the same thing.
public function testWithoutAsserting() {
$this->expectNotToPerformAssertions();
$x = 5;
}
Use the #doesNotPerformAssertions annotation:
/**
* #doesNotPerformAssertions
*/
public function testCodeWithoutUsingAssertions()
{
// do stuff...
}
With PHPUnit 7.2 you get another alternative:
public function testCodeWithoutUsingAssertions()
{
$this->expectNotToPerformAssertions();
// do stuff...
}
See also https://github.com/sebastianbergmann/phpunit/pull/3042.
Make use of $this->addToAssertionCount(1). See below.
class NoAssertTest extends PHPUnit_Framework_TestCase
{
function testWithoutAssertions() {
$x = 5;
// Increment the assertion count to signal this test passed.
// This is important if you use a #depends on this test
$this->addToAssertionCount(1);
}
}
If you have PHP SimpleTest, another approach would be to use:
$this->pass();
This will mark the test as completed and passed.
On the other hand, for a test that you want to fail, you can use:
$this->fail();
For example:
if (someComplicatedLogicValidation) {
// Do more stuff and asserts
} else{
$this->fail();
}
I tried this in PHP 5.5 and works:
function testThatWorks() {
$this->pass();
}
function testThatFails() {
$this->fail();
}
Output:
1) Fail at [...ExampleTest.unit.php line 23]
in testThatFails
in ExampleTest
in .../ExampleTest.unit.php
fail in 3.02s
Maybe the method pass can still be easily implemented in PHPUnit. Source from SimpleTest:
function pass($message = "Pass") {
if (! isset($this->reporter)) {
trigger_error('Can only make assertions within test methods');
}
$this->reporter->paintPass(
$message . $this->getAssertionLine());
return true;
}
In my case, the test has no assertion because I'm just checking everything using prophecy lib to mock and check that methods are being called as expected...
PHPUnit by default expects that test has at least one assertion, and warn you to put it.
Just adding this line in any part of your test is enough.
$this->expectNotToPerformAssertions();
But also you can design a test that in some cases have assertions or not
public function testCodeNoAssertions()
{
if ($something === true) {
$this->expectNotToPerformAssertions();
}
// do stuff...
if ($somethingElse === true) {
//do any assertion
}
}
In conclusion, It's not a bug... It's a feature.
PHPUnit 7.2+. $this->expectNotToPerformAssertions() can be used, if you have conditions, that might disable assertions. Seems it is not in the official docs, but can be found here: https://github.com/sebastianbergmann/phpunit/pull/3042/files
public function testCodeNoAssertions()
{
$this->expectNotToPerformAssertions();
}
At least PHPUnit 7.0+. #doesNotPerformAssertions can be used. Current docs do not state when this was added. https://phpunit.readthedocs.io/en/7.0/annotations.html#doesnotperformassertions
/**
* #doesNotPerformAssertions
*/
public function testCodeWithoutUsingAssertions()
{
}
If your test is simply incomplete, you should be using $this->markTestIncomplete() https://phpunit.readthedocs.io/en/7.0/incomplete-and-skipped-tests.html
public function testCodeThatIsIncomplete()
{
$this->markTestIncomplete('Implementation of this test is not complete.');
}
If you want to temporarily disable a test $this->markTestSkipped() should be used. https://phpunit.readthedocs.io/en/7.0/incomplete-and-skipped-tests.html#skipping-tests
public function testCodeThatIsDisabledTemporarily()
{
$this->markTestSkipped('This test is disabled to test, if it is interfering with other tests.');
}
I created a superclass and placed SimpleTest -like function there:
/**
* Pass a test
*/
public function pass()
{
//phpunit is missing pass()
$this->assertTrue(true);
}
Then you can call it
$this->pass();
Based on this article, I've written a custom class which implements the Watin.Core.interfaces.IFindByDefaultFactory, but I don't think I'm correctly assigning it to the watin settings, because it is never used.
Basically, Where/when should I assign to the Settings.FindByDefaultFactory? I've tried in my test Setup, and the text fixture's constructor, but neither seem to cause my custom class to be used. The tests still run and work, but I have to use the full asp.net ID's.
I'm using Watin 2.0.15.928 in VS2008 from nUnit 2.5.2.9222. I am running visual studio as administrator, and tests run sucessfully as long as I don't rely on my custom find logic.
Here's what the start of my text fixture looks like, where I set the FindByDefaultFactory
namespace Fundsmith.Web.Main.BrowserTests
{
[TestFixture]
class WatinHomepageTests
{
private IE _ie;
[SetUp]
public void Setup()
{
Settings.FindByDefaultFactory = new FindByAspIdFactory();
_ie = new IE("http://localhost/somepage.aspx");
}
//etc etc...
And this is what my custom Find By Default factory looks like (simplified), unfortunately, it's never called.
using System.Text.RegularExpressions;
using WatiN.Core;
using WatiN.Core.Constraints;
using WatiN.Core.Interfaces;
namespace Fundsmith.Web.Main.BrowserTests
{
public class FindByAspIdFactory : IFindByDefaultFactory
{
public Constraint ByDefault(string value)
{
// This code is never called :(
// My custom find by id code to cope with asp.net webforms ids...
return Find.ById(value);
}
public Constraint ByDefault(Regex value)
{
return Find.ById(value);
}
}
}
Edit: Extra information after the fact.
Based on me fuguring this out, (see answer below), It turns out that the way I was consuming Watin to find the elements was wrong. I was explicitly calling Find.ById, rather than letting the default action occur. So I'd reassigned the default but was then failing to use it!
[Test]
public void StepOneFromHomepageShouldRedirectToStepTwo()
{
_ie.TextField(Find.ById("textBoxId")).TypeText("100");
//Other test stuff...
}
Right, I've figured this one out, and it was me being an idiot and explicitly calling the Find.ById method, rather than letting the default action occur. It seems the test setup is a fine place to set the FindByDefaultFactory.
ie, I was doing this (wrong):
[Test]
public void StepOneFromHomepageShouldRedirectToStepTwo()
{
_ie.TextField(Find.ById("textBoxId")).TypeText("100");
//Other test stuff...
}
When I should have been simply doing this. (Without the explicit "Find.ById")
[Test]
public void StepOneFromHomepageShouldRedirectToStepTwo()
{
_ie.TextField("textBoxId").TypeText("100");
//Other test stuff...
}
Not only was this me being stupid, but I didn't include this in my original question, so it would have been impossible for anyone else to figure it out for certain. Double slaps for me.