Why phpunit is green when there are tons of exceptions? - phpunit

Question

Why phpunit is green when there are tons of exceptions?
Here's an example job in travis: https://travis-ci.org/ConnectCorp/nexmo-client/jobs/79275172
Strict standards: Declaration of Nexmo\Tests\Service\TestServiceTrait::exec() should be compatible with Nexmo\Service\Service::exec($params, $method = 'GET') in /home/travis/build/ConnectCorp/nexmo-client/tests/Service/VoiceTest.php on line 42
Call Stack:
0.0013 415584 1. {main}() /home/travis/.phpenv/versions/5.4.37/bin/phpunit:0
0.0209 1038200 2. PHPUnit_TextUI_Command::main() /home/travis/.phpenv/versions/5.4.37/bin/phpunit:722
0.0209 1044680 3. PHPUnit_TextUI_Command->run() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/TextUI/Command.php:104
0.0209 1045112 4. PHPUnit_TextUI_Command->handleArguments() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/TextUI/Command.php:114
0.0281 1812016 5. PHPUnit_Util_Configuration->getTestSuiteConfiguration() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/TextUI/Command.php:664
0.0281 1812928 6. PHPUnit_Util_Configuration->getTestSuite() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/Util/Configuration.php:814
0.0318 2063368 7. PHPUnit_Framework_TestSuite->addTestFiles() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/Util/Configuration.php:901
0.1145 6533504 8. PHPUnit_Framework_TestSuite->addTestFile() /home/travis/build/ConnectCorp/nexmo-client/vendor/phpunit/phpunit/src/Framework/TestSuite.php:403
0.1145 6571744 9. PHPUnit_Util_Fileloader::checkAndLoad() /home/travis/build/ConnectCorp/nexmo-client/vendor/phpunit/phpunit/src/Framework/TestSuite.php:331
0.1146 6571880 10. PHPUnit_Util_Fileloader::load() phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/Util/Fileloader.php:42
0.1148 6681536 11. include_once('/home/travis/build/ConnectCorp/nexmo-client/tests/Service/VoiceTest.php') phar:///home/travis/.phpenv/versions/5.4.37/bin/phpunit/phpunit/Util/Fileloader.php:58
PHPUnit 4.8.6 by Sebastian Bergmann and contributors.
............................................................... 63 / 117 ( 53%)
......................................................
Time: 511 ms, Memory: 13.00Mb
OK (117 tests, 210 assertions)
https://travis-ci.org/ConnectCorp/nexmo-client/jobs/79275172
I would expect it would fail and travis report that a build has failed, but it just silently passes. Is there any phpunit config option I am missing?
Current phpunit config is this:
<?xml version="1.0" encoding="utf-8"?>
<phpunit colors="true"
bootstrap="./vendor/autoload.php"
convertErrorsToExceptions="true"
convertNoticesToExceptions="true"
convertWarningsToExceptions="true"
stopOnFailure="false">
<testsuites>
<testsuite name="Nexmo Test Suite">
<directory>./tests/</directory>
</testsuite>
</testsuites>
<filter>
<blacklist>
<directory suffix=".php">vendor/*</directory>
</blacklist>
</filter>
</phpunit>

Answer 1

Even though there is an error, no tests have actually failed, so phpunit will return a success code.
Before running the tests, phpunit first loads and parses *Test.php files and this is where your errors are coming from, rather than from a test method itself.
Changing the config option stopOnFailure="false" to stopOnFailure="true" or stopOnError="true" should make this task fail.
From the documentation:
--stop-on-error Stop execution upon first error.
--stop-on-failure Stop execution upon first error or failure.
Edit
Actually, just been playing with this and as it is a notice and before the tests actually start those flags won't have any effect.

Answer 2

You seem to discovered a bug in phpunit and I suggest you report it to their repository.
It only seems to be a problem if a notice is triggered from the test class. Your mock classes are defined in the same file as a test class. If you move them out to its own file tests will start failing as expected. Methods need to have the same signature as their parent.
Might be also worth to configure error reporting in your phpunit.xml to report all errors (although this alone will not solve your current issue, error reporting will be enable no matter how environment is configured):
<php>
<ini name="error_reporting" value="-1" />
</php>