I'm noticing when I use mock objects, PHPUnit will correctly report the number of tests executed but incorrectly report the number of assertions I'm making. In fact, every time I mock it counts as another assertion. A test file with 6 tests, 7 assert statements and each test mocking once reported 6 tests, 13 assertions.
Here's the test file with all but one test removed (for illustration here), plus I introduced another test which doesn't stub to track down this problem. PHPUnit reports 2 tests, 3 assertions. I remove the dummy: 1 test, 2 assertions.
require_once '..\src\AntProxy.php';
class AntProxyTest extends PHPUnit_Framework_TestCase {
const sample_client_id = '495d179b94879240799f69e9fc868234';
const timezone = 'Australia/Sydney';
const stubbed_ant = "stubbed ant";
const date_format = "Y";
public function testBlankCategoryIfNoCacheExists() {
$cat = '';
$cache_filename = $cat.'.xml';
if (file_exists($cache_filename))
unlink($cache_filename);
$stub = $this->stub_Freshant($cat);
$expected_output = self::stubbed_ant;
$actual_output = $stub->getant();
$this->assertEquals($expected_output, $actual_output);
}
public function testDummyWithoutStubbing() {
$nostub = new AntProxy(self::sample_client_id, '', self::timezone, self::date_format);
$this->assertTrue(true);
}
private function stub_FreshAnt($cat) {
$stub = $this->getMockBuilder('AntProxy')
->setMethods(array('getFreshAnt'))
->setConstructorArgs(array(self::sample_client_id, $cat, self::timezone, self::date_format))
->getMock();
$stub->expects($this->any())
->method('getFreshAnt')
->will($this->returnValue(self::stubbed_ant));
return $stub;
}
}
It's like an assert has been left in one of the framework's mocking methods. Is there a way to show every (passing) assertion being made?
After each test method completes, PHPUnit verifies the mock expectations setup during the test. PHPUnit_Framework_TestCase::verifyMockObjects() increments the number of assertions for each mock object created. You could override the method to undo this if you really want by storing the current number of assertions, calling the parent method, and subtracting the difference.
protected function verifyMockObjects()
{
$count = $this->getNumAssertions();
parent::verifyMockObjects();
$this->addToAssertionCount($count - $this->getNumAssertions());
}
Of course, verifyMockObjects() will throw an assertion failure exception if any expectation is unsatisfied, so you'll need to catch the exception and rethrow it after resetting the count. I'll leave that to you. :)
Related
Our code defines some "rules" in a List<Rule> collection. Each rule contains some logic in the form of a string. The rules are all passed to a "rule engine" along with some data. The rule engine sequentially evaluates the data against rules until it finds a rule which evaluates as true, then it returns that Rule.
We want to make to automatically test every rule. The tests will actually be integration tests, rather than unit tests, because they'll test the combination of the rule engine and the rules.
How do I write a test that says "make sure each rule evaluates as true in at least one unit test"?
I've figured out a way to run some fixture teardown code after all the tests have run (see https://xunit.github.io/docs/shared-context.html#class-fixture), and by using a static variable to record evaluated rules I can check during the teardown whether all the rules have been returned during unit tests. But this approach has the undesirable effect that it causes individual test to report as failed (in teardown) which didn't actually fail.
TLDR: control the test sequence. Since these are integration tests, that's not the big no-no it would be if they were unit tests.
I found that by using a static variable to record the rules that evaluated as true, and ensuring the have-all-rules-evaluated-as-true test last, as per the advice here https://hamidmosalla.com/2018/08/16/xunit-control-the-test-execution-order/, it becomes trivially easy to achieve what I needed.
In case the link dies, here's how to implement this approach
First, create an AlphabeticalOrderer implementing xUnit's ITestCaseOrderer:
public class AlphabeticalOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases)
where TTestCase : ITestCase
{
var result = testCases.ToList();
result.Sort((x, y) => StringComparer.OrdinalIgnoreCase.Compare(x.TestMethod.Method.Name, y.TestMethod.Method.Name));
return result;
}
}
Then use the newly-created attribute on your test class, and create a static variable to record progress:
[TestCaseOrderer("MyCompany.AlphabeticalOrderer", "MyTestsDLL")]
public class RulesTests
{
private static List<Rule> _untestedRules;
public RulesTests()
{
if (_untestedRules == null) // this will run once per [Fact]
{
_untestedRules = <some way of getting all the rules> // this will only run first time around
}
}
Each time a rule triggers, record which rule it was:
private void MarkRuleAsTested(LendingRule testedRule)
{
var rulesToRemove = _untestedRules.Where(r => r.Logic == testedRule.Logic).ToList();
foreach (var ruleToRemove in rulesToRemove)
{
_untestedRules.Remove(ruleToRemove);
}
}
Then the last test to run should check the collection:
[Fact]
public void ZZ_check_all_rules_have_been_triggered_by_other_tests()
{
// This test must run last because it validates that the
// OTHER tests taken together have resulted in each rule
// being evaluated as true at least once.
if (!_untestedRules.Any())
{
return;
}
var separator = "\r\n------------------------\r\n";
var untestedRules = string.Join(separator, _untestedRules.Select(ur => $"Logic: {ur.Logic}"));
throw new SomeRulesUntestedException($"The following rules have not been tested to resolve as True: {separator}{untestedRules}{separator}");
}
This results in a nicely-readable test failure.
You could equally start with an empty static collection of _testedRules and at the end compare that set to the full set of rules.
I'm trying to write an integration test for a service method. The test compiles and runs without error, but it says that the number of calls that match the predicate are 0.
Test setup:
[TestCase]
public void Save_Submission_Processing_And_ClientGroupMapping_Type()
{
Mock<ISubmissionRepository> submissionRepositoryMock = new Mock<ISubmissionRepository>();
submissionRepositoryMock.Setup(x => x.GetOne(It.IsAny<Guid>())).Returns(QueryResult<Submission>.Ok(new Submission()));
IServiceCollection services = new ServiceCollection();
services.AddSingleton(x => submissionRepositoryMock.Object);
ClientGroupMappingService clientGroupMappingService = new ClientGroupMappingService(services.BuildServiceProvider());
clientGroupMappingService.ProcessClientGroupMappingImport(Guid.NewGuid());
submissionRepositoryMock.Verify(c => c.Save(It.Is<Submission>(d => d.SubmissionStatus == SubmissionStatus.Processing)), Times.Once);
}
Unit under test:
public class ClientGroupMappingService : IClientGroupMappingService
{
private readonly ISubmissionRepository _submissionRepository;
public ClientGroupMappingService(IServiceProvider serviceProvider)
{
_submissionRepository = serviceProvider.GetRequiredService<ISubmissionRepository>();
}
public void ProcessClientGroupMappingImport(Guid submissionID)
{
Submission submission = _submissionRepository.GetOne(submissionID).Value;
submission.SubmissionStatus = SubmissionStatus.Processing;
_submissionRepository.Save(submission);
// ..other stuff
}
}
Moq.MockException :
Expected invocation on the mock once, but was 0 times: c => c.Save(It.Is<Submission>(d => (int)d.SubmissionStatus == 2))
So Verify should see that the call was made to Save, and the param passed to Save matches the condition in the supplied predicate. My knee-jerk reaction is that once I pull the object out of the mock using submissionRepositoryMock.Object, I am no longer tracking the Mock, so calls to the Object are not going to register on the Mock. But if this is the case, what is the correct way to verify that my method made the required call?
The issue was in the "// ...other stuff" that I took out for brevity.
Later on in the method, the SubmissionStatus is updated again, and Mock.Verify only seems to be evaluating the REFERENCE to the object that was passed into the Save call as opposed to the VALUE, as all of the Invocations of Save show that it was called with SubmissionStatus.Success (which isn't actually true).
Hope this makes sense and helps anyone with a similar problem.
I have a controller I'd like to create functional tests for. This controller makes HTTP requests to an external API via a MyApiClient class. I need to mock out this MyApiClient class, so I can test how my controller responds for given responses (e.g. what will it do if the MyApiClient class returns a 500 response).
I have no problems creating a mocked version of the MyApiClient class via the standard PHPUnit mockbuilder: The problem I'm having is getting my controller to use this object for more than one request.
I'm currently doing the following in my test:
class ApplicationControllerTest extends WebTestCase
{
public function testSomething()
{
$client = static::createClient();
$apiClient = $this->getMockMyApiClient();
$client->getContainer()->set('myapiclient', $apiClient);
$client->request('GET', '/my/url/here');
// Some assertions: Mocked API client returns 500 as expected.
$client->request('GET', '/my/url/here');
// Some assertions: Mocked API client is not used: Actual MyApiClient instance is being used instead.
}
protected function getMockMyApiClient()
{
$client = $this->getMockBuilder('Namespace\Of\MyApiClient')
->setMethods(array('doSomething'))
->getMock();
$client->expects($this->any())
->method('doSomething')
->will($this->returnValue(500));
return $apiClient;
}
}
It seems as though the container is being rebuilt when the second request is made, causing the MyApiClient to be instantiated again. The MyApiClient class is configured to be a service via an annotation (using the JMS DI Extra Bundle) and injected into a property of the controller via an annotation.
I'd split each request out into its own test to work around doing this if I could, but unfortunately I can't: I need to make a request to the controller via a GET action and then POST the form it brings back. I'd like to do this for two reasons:
1) The form uses CSRF protection, so if I just POST directly to the form without using the crawler to submit it, the form fails the CSRF check.
2) Testing that the form generates the correct POST request when it is submitted is a bonus.
Does anyone have any suggestions on how to do this?
EDIT:
This can be expressed in the following unit test that does not depend on any of my code, so may be clearer:
public function testAMockServiceCanBeAccessedByMultipleRequests()
{
$client = static::createClient();
// Set the container to contain an instance of stdClass at key 'testing123'.
$keyName = 'testing123';
$client->getContainer()->set($keyName, new \stdClass());
// Check our object is still set on the container.
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName))); // Passes.
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName))); // Passes.
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName))); // Fails.
}
This test fails, even if I call $client->getContainer()->set($keyName, new \stdClass()); immediately before the second call to request()
When you call self::createClient(), you get a booted instance of the Symfony2 kernel. That means, all config is parsed and loaded. When now sending a request, you let the system do it's job for the first time, right?
After the first request, you may want to check what went on, and therefore, the kernel is in a state, where the request is sent, but it's still running.
If you now run a second request, the web-architecture requires, that the kernel reboots, because it already ran a request. This reboot, in your code, is executed, when you execute a request for the second time.
If you want to boot the kernel and modify it before the request is sent to it (like you want), you have to shutdown the old kernel-instance and boot a fresh one.
You can do that by just rerunning self::createClient(). Now you again have to apply your mock, as you did the first time.
This is the modified code of your second example:
public function testAMockServiceCanBeAccessedByMultipleRequests()
{
$keyName = 'testing123';
$client = static::createClient();
$client->getContainer()->set($keyName, new \stdClass());
// Check our object is still set on the container.
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName)));
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName)));
# addded these two lines here:
$client = static::createClient();
$client->getContainer()->set($keyName, new \stdClass());
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName)));
}
Now you may want to create a separate method, that mocks the fresh instance for you, so you don't have to copy your code ...
I thought I'd jump in here. Chrisc, I think what you want is here:
https://github.com/PolishSymfonyCommunity/SymfonyMockerContainer
I agree with your general approach, configuring this in the service container as a parameter is really not a good approach. The whole idea is to be able to mock this dynamically during individual test runs.
The behaviour you are experiencing is actually what you would experience in any real scenario, as PHP is share nothing and rebuilds the whole stack on each request. The functional test suite imitates this behaviour to not generate wrong results. One example would be doctrine, which has a ObjectCache, so you could create objects, not save them to the database and your tests would all pass because it takes the objects out of the cache all the time.
You can solve this problem in different ways:
Create a real class which is a TestDouble and emulates the results you would expect from the real API. This is actually very easy: You create a new MyApiClientTestDouble with the same signature as your normal MyApiClient, and just change the method bodies where needed.
In your service.yml, you alright might have this:
parameters:
myApiClientClass: Namespace\Of\MyApiClient
service:
myApiClient:
class: %myApiClientClass%
If this is the case, you can easily overwrite which class is taken by adding the following to your config_test.yml:
parameters:
myApiClientClass: Namespace\Of\MyApiClientTestDouble
Now the service container will use your TestDouble when testing. If both classes have the same signature, nothing more is needed. I don't know if or how this works with the DI Extras Bundle. but I guess there is a way.
Or you could create a ApiDouble, implementing a "real" API which behaves in the same way your external API does but returns test data. You would then make the URI of your API handled by the service container (e.g. setter injection) and create a parameters variable which points to the right API (the test one in case of dev or test and the real one in case of the production environment).
The third way is a bit hacky, but you can always make a private method inside your tests request which first sets up the container in the right way and then calls the client to make the request.
I do not know if you ever found out how to fix your problem. But here is the solution i used. This is also good for other people finding this.
After a long search for the problem with mocking a service between multiple client requests i found this blog post:
http://blog.lyrixx.info/2013/04/12/symfony2-how-to-mock-services-during-functional-tests.html
lyrixx talk about how the kernel shutsdown after each request making the service overrid invalid when you try to make another request.
To fix this he creates a AppTestKernel used only for the function tests.
This AppTestKernel extends the AppKernel and only apply some handlers to modifie the Kernel:
Code examples from lyrixx blogpost.
<?php
// app/AppTestKernel.php
require_once __DIR__.'/AppKernel.php';
class AppTestKernel extends AppKernel
{
private $kernelModifier = null;
public function boot()
{
parent::boot();
if ($kernelModifier = $this->kernelModifier) {
$kernelModifier($this);
$this->kernelModifier = null;
};
}
public function setKernelModifier(\Closure $kernelModifier)
{
$this->kernelModifier = $kernelModifier;
// We force the kernel to shutdown to be sure the next request will boot it
$this->shutdown();
}
}
When you then need to override a service in your test you call the setter on the testAppKernel and applies the mock
class TwitterTest extends WebTestCase
{
public function testTwitter()
{
$twitter = $this->getMock('Twitter');
// Configure your mock here.
static::$kernel->setKernelModifier(function($kernel) use ($twitter) {
$kernel->getContainer()->set('my_bundle.twitter', $twitter);
});
$this->client->request('GET', '/fetch/twitter'));
$this->assertSame(200, $this->client->getResponse()->getStatusCode());
}
}
After following this guide i had some problems getting the phpunittest to startup with the new AppTestKernel.
I found out that the symfonys WebTestCase (https://github.com/symfony/symfony/blob/master/src/Symfony/Bundle/FrameworkBundle/Test/WebTestCase.php)
Takes the first AppKernel file it finds. So one way to get out of this is to change the name on the AppTestKernel to come before AppKernel or to override the method to take the TestKernel Instead
Here i overrride the getKernelClass in the WebTestCase to look for a *TestKernel.php
protected static function getKernelClass()
{
$dir = isset($_SERVER['KERNEL_DIR']) ? $_SERVER['KERNEL_DIR'] : static::getPhpUnitXmlDir();
$finder = new Finder();
$finder->name('*TestKernel.php')->depth(0)->in($dir);
$results = iterator_to_array($finder);
if (!count($results)) {
throw new \RuntimeException('Either set KERNEL_DIR in your phpunit.xml according to http://symfony.com/doc/current/book/testing.html#your-first-functional-test or override the WebTestCase::createKernel() method.');
}
$file = current($results);
$class = $file->getBasename('.php');
require_once $file;
return $class;
}
After this your tests will load with the new AppTestKernel and you will be able to mock services between multiple client requests.
Based on the answer by Mibsen you can also set this up in a similar way by extending the WebTestCase and overriding the createClient method. Something along these lines:
class MyTestCase extends WebTestCase
{
private static $kernelModifier = null;
/**
* Set a Closure to modify the Kernel
*/
public function setKernelModifier(\Closure $kernelModifier)
{
self::$kernelModifier = $kernelModifier;
$this->ensureKernelShutdown();
}
/**
* Override the createClient method in WebTestCase to invoke the kernelModifier
*/
protected static function createClient(array $options = [], array $server = [])
{
static::bootKernel($options);
if ($kernelModifier = self::$kernelModifier) {
$kernelModifier->__invoke();
self::$kernelModifier = null;
};
$client = static::$kernel->getContainer()->get('test.client');
$client->setServerParameters($server);
return $client;
}
}
Then in the test you would do something like:
class ApplicationControllerTest extends MyTestCase
{
public function testSomething()
{
$apiClient = $this->getMockMyApiClient();
$this->setKernelModifier(function () use ($apiClient) {
static::$kernel->getContainer()->set('myapiclient', $apiClient);
});
$client = static::createClient();
.....
Make a mock:
$mock = $this->getMockBuilder($className)
->disableOriginalConstructor()
->getMock();
$mock->method($method)->willReturn($return);
Replace service_name on mock-object:
$client = static::createClient()
$client->getContainer()->set('service_name', $mock);
My problem was to use:
self::$kernel->getContainer();
I faced with the same problem in Symfony 4.4.
After reading
Create mocks in api functional testing with Symfony
I found a solution - self::ensureKernelShutdown()
...
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName))); // Passes.
self::ensureKernelShutdown()
$client->request('GET', '/any/url/');
$this->assertEquals('stdClass', get_class($client->getContainer()->get($keyName))); // Passes.
...
I have a unit test that I am writing and have run into an annoying problem... Let's say I have the following function I am testing:
public function functionToTest(array &$data, parameter2)
{
// perform some action on the array that is being passed in by reference
}
Now, when I attempt to call this function in my unit test, I would do something like this:
public function testMyFunction()
{
$data = array('key1' => 'val1');
$mockOfClass = $this->getMockBuilder('ClassName')
->disableOriginalConstructor()
->setMethods(array('method1', 'method2')) // My function to test is NOT in this list
->getMock();
$this->mockOfClass->functionToTest($data, true);
// Perform assertions here
}
However, I receive the following error message:
Parameter 1 to ClassName::addNewFriendsToProfile() expected to be a reference, value given
This seemed very strange to me. First of all, I'm just passing an array by reference, so it it shouldn't have a problem with this. Secondly, Why parameter 1? Doesn't it mean parameter 0? Then, I tried changing the call to the following:
$this->mockOfClass->functionToTest(&$data, true);
After making this change, it works fine. Unfortunately, it also produces the following warning:
Call-time pass-by-reference has been deprecated in /PathToFile on line xxx
I do not encounter this error when running the actual code. It only throws this error in the unit test. Also, I need to use the mock as there are methods in the class I am mocking; So I can't simply create a new instance of the class and call the method that is being tested. Is there any way I can get around this?
It turns out that PHPUnit clones each of the parameters that are being passed in (Thanks Tim Lytle for pointing me to this source: Pass by reference in a callback when mocking in PHPUnit). This is what causes the error if the array is passed in without a reference at call-time in the unit test. Luckily, the solution is simple. Instead of passing the array by reference, I pass in the array by value and return the array.
before:
public function someFunction(array &$myArray)
{
$myArray[] = 'new val';
}
after:
public function someFunction(array $myArray)
{
$myArray[] = 'new val';
return $myArray;
}
PHPUnit contains an assertEquals() method, but it also has an assertSame() one. At first glance it looks like they do the same thing.
What is the difference between the two? Why are they both specified?
I use both sporadically, but according to the docs:
assertSame
Reports an error identified by $message if the two variables $expected and $actual do not have the same type and value."
And as you can see in the example below the above excerpt, they are passing '2204' and 2204, which will fail using assertSame because one is a string and one is an int, basically:
'2204' !== 2204
assertSame('2204', 2204) // this test fails
assertEquals
"Reports an error identified by $message if the two variables $expected and $actual are not equal."
assertEquals does not appear to take datatype into consideration so using the above example of 2204:
'2204' == 2204
assertEquals('2204', 2204) // this test passes
I just ran some unit tests against the above examples, and indeed they resulted in documented behavior.
When it comes to objects comparison:
assertSame
Can only assert if two objects are referencing the same object instance. So even if two separate objects have for all of their attributes exactly the same values, assertSame() will fail if they don't reference the same instance.
$expected = new \stdClass();
$expected->foo = 'foo';
$expected->bar = 'bar';
$actual = new \stdClass();
$actual->foo = 'foo';
$actual->bar = 'bar';
$this->assertSame($expected, $actual); // FAILS
assertEquals
Can assert if two separate objects match their attribute values in any case. So it's the method suitable for asserting object match.
$this->assertEquals($expected, $actual); // PASSES
Reference
$this->assertEquals(3, true);
$this->assertSame(3, true);
The first one will pass!
The second one will fail.
That is the difference.
I think you should always use assertSame.
As it's been said before, assertSame reports an error if the two elements do not share type and value but it's also important to note this from the documentation:
Reports an error identified by $message if the two variables $expected
and $actual do not reference the same object.
So this test would fail too even though they share type and value:
class SameTest extends TestCase
{
public function testFailure()
{
$this->assertSame(new stdClass, new stdClass);
}
}
Moreover,
// Passes
$this->assertSame("123.", "123.");
$this->assertEquals("123.", "123");
// Fails
$this->assertSame("123.", "123");
Never use assertEquals (and family)*
* unless arguments were first manually type-checked, or you're fully aware of the behavior and are prepared for tests to pass when it treats, say, true and 42, as equal--I can't imagine why one would tolerate the risk of extremely subtle false positives in critical testing code.
For primitive values (other than floats) and arrays
Use assertSame. The behavior is predictable. "42" and true are not equal.
For floats
Avoid if possible by using integers. If you must compare floats, assertEqualsWithDelta is as broken as assertEquals:
$this->assertEqualsWithDelta(0.99, "1", 0.1); // passes :\
$this->assertEqualsWithDelta(99999, true, 0.1); // passes D:
The implementation of assertEqualsWithDelta uses an inaccurate approach, as Comparing Floating Point Numbers describes: abs($this->value - $other) < PHP_FLOAT_EPSILON;, so you might want to write a custom function.
Or type-check before calling assertEqualsWithDelta to avoid the more immediate issues:
function is_number($v) {
return is_float($v) || is_int($v);
}
class FloatTest extends TestCase {
private function assertClose(
$expected,
$actual,
$delta = 1e-7
): void {
if (!is_number($expected)) {
$type = gettype($expected);
throw new Error("\$expected type $type was not a number");
}
else if (!is_number($actual)) {
$type = gettype($actual);
throw new Error("\$actual value $type was not a number");
}
else if (!is_number($delta)) {
$type = gettype($delta);
throw new Error("\$delta value $type was not a number");
}
$this->assertEqualsWithDelta($actual, $expected, $delta);
}
public function testFloats() {
$this->assertClose(2, "2"); // fails as it should
}
}
For objects
Use a custom equals(self $other): bool function and assertObjectEquals.
If all else fails
You can't go wrong with assertTrue, the downsides being a lack of semantic appropriateness and poor default error message.
assertSame() == Tests that if the actual output and the expected parameter are same.
that is :
$this->assertSame('$expected','$expected');
or
$this->assertSame('100','100');
assertEquals == If we see with respect to a website page, i have a page which has 2 'table' so when i run assertEquals i will check its count that the 'table' are 2 by using a count function.
Eg:
$this->assertEquals(2, $var->filter('table')->count());
Here we can see that assertEquals checks that there are 2 tables found on the web page. we can also use divisions found on the page using '#division name' inside the bracket.
Eg 2:
public function testAdd()
{
$calc = new Calculator();
$result = $calc->add(30, 12);
// assert that our calculator added the numbers correctly!
$this->assertEquals(42, $result);
}
As previously mentioned, assertEquals() is primarily about an interpreted value, be it by type juggling or an object with an __magic presentation method (__toString() for example).
A good use case for assertSame() is testing a singleton factory.
class CacheFactoryTest extends TestCase
{
public function testThatCacheFactoryReturnsSingletons()
{
$this->assertSame(CacheFactory::create(), CacheFactory::create());
}
}