I'm working on upgrading our codebase.
In this process I upgraded PHPUnit from 7 to 9. This decreased our assertions quite drastically.
I downgraded to PHPUnit 8 to see what happened, they were still gone.
Downgrading back to PHPUnit 7 made them 'popup' again.
I've ran the checks with '--debug' and compared the output. This does not show tests being skipped. I also don't see much difference in other metrics... Code coverage is the similar as well.
PHPUnit 7:
Time: 14.08 minutes, Memory: 1.71 GB
OK, but incomplete, skipped, or risky tests!
Tests: 2053, Assertions: 143695, Skipped: 56.
PHPUnit 8:
Time: 16.14 minutes, Memory: 1.31 GB
OK, but incomplete, skipped, or risky tests!
Tests: 2053, Assertions: 22304, Skipped: 56, Risky: 56.
We have some tests which run 'optionally'. So if a setting isn't true, they're skipped. This is marked as 'risky' in PHPUnit 8 and I plan to address that later. However that should not impact the number of assertions as in both cases the tests isn't running.
I've looked at the changelog (https://github.com/sebastianbergmann/phpunit/blob/8.0.0/ChangeLog-8.0.md#800---2019-02-01) but that doesn't really help me.
It would be great if somebody could help me to figure out if this is a problem, or a change in PHPUnit which I could safely ignore.
Update:
I've added a tearDown function which echos '$this->getCount() and $this->getNumAssertions()'. This shows the following numbers. The getNumAssertions is mostly 0, so I guess this is not the right place to call it or the right way to use it.
PHPUnit 7:
ERRORS!
Tests: 2053, Assertions: 142339, Errors: 3, Skipped: 67.
Sum of 'getCount' results: 22341
PHPUnit 8:
ERRORS!
Tests: 2053, Assertions: 22641, Errors: 2, Skipped: 67.
Sum of 'getCount' results: 22342
PS: Numbers are a bit different compared to the first results, but I've worked on some tests in the meanwhile.
Interesting thing here is that the getCount results for PHPUnit7, PHPUnit8 and the total for PHPUnit 8 are comparable. Still no explanation for the missing assertions though.
Related
If I run my tests using PHP7.2 or PHP7.1 they are about 3x slower than if I run them using PHP7.0. Is there anyway to get to the bottom of why this is happening?
Even when I run the test suites (Feature & Unit) separately I still see the slow down. It's only when I run the tests individually does the speed difference become insignificant.
I'm using Laravel 5.5.20 and Laravel Homestead 7.0.1. I have 47 rather simple tests, some hitting the database, others just simple assertions; so there isn't anything that should take ages.
I installed johnkary/phpunit-speedtrap to see which tests take the longest so I could remove those but there isn't a specific test that takes a long time because if I remove the offending test, the next one will take ages (see below).
First Run Second Run
Test A 0.2 sec Test A 0.2 sec
Test B. 0.3 sec Test B. 0.3 sec
Test C 0.1 sec Test C 0.1 sec
Test D 0.1 sec Test D 0.1 sec
Test E 9.3 sec REMOVED Test E
Test F 0.3 sec Test F 9.3 sec <-- Test F now takes ages
Test G 0.2 sec Test G 0.2 sec
I am also using an in-memory SQLite3 database, with the Laravel CreatesApplication and RefreshDatabase trait as I want each test to run independently.
I do not have Xdebug installed or running. Is there something known that PHP7.1 and PHP7.2 take a long time to run PHPUnit tests? Is there something else I can install (or even run it with Xdebug) to track down what exactly it is that is causing the issue?
Setup
Laravel 5.5.20
Laravel Homestead 7.0.1 (Per-project installation)
PHPUnit 6.4.4
Vagrant 2.0.1
Virtualbox 5.2.4
Results
PHP 7.2 PHPUnit 6.4.4
Time: 12.4 seconds, Memory: 162.00MB
PHP 7.1 PHPUnit 6.4.4
Time: 12.19 seconds, Memory: 162.00MB
PHP 7.0 PHPUnit 6.4.4
Time: 4.88 seconds, Memory: 162.00MB
I had the same issue as you but with XDebug installed. I found a pretty good hint by a user named Roni on Laracasts (I can't find the link anymore, sorry) which says to run tests with the -n flag of the php command, like this: php -n vendor/bin/phpunit.
According to the documentation at php.net (command line options) this is for running the command without what php.ini is defining. Which means no includes of extensions.
-n No php.ini file will be used
So, for me it runs tests in a minute now, not in 15 minutes. The problem is kind of strange, because it began with php 7.2 on my machine but others in my team don't have the issue, despite the fact that xdebug is installed. I wonder what really is behind this issue.
I have an R package that was passing all Travis CI builds successfully and randomly started to fail. I've pondered where the problem is at but I can't seem to fix it.
Here is the travi.yaml from the package
Here is the failing log of Travis-CI
Here is the last log the passed (about 2-3 logs behind the last fail)
I think the fail might be related to line 1495 from the last log that passed:
The command "for name in $(find "${RCHECK_DIR}" -type f -name "*out");do echo ">>> Filename: ${name} <<<";cat ${name};done" exited with 0.
which does not appear in the failed log from above. After thoroughly checking my tests with devtools:
R CMD check results
0 errors | 0 warnings | 0 notes
I don't know what it might be. I thought it was related to some examples taking too much time (some examples pass the 5 second thresholds, so I comment them out but the failing persists. I tried posting this in the Rstudio community but no luck so far.
Any ideas?
I have installed the pabot lib but still struggling to trigger my testcases.
Can someone please suggest how to start the testcase run.
I ran it like below :
C:\Python27\SampleScripts\DataDrivenApproach\KITE>pabot Pabt.robot --processes 2
tests
And got this :
get_suite_names: output_file='.\pabot_results\suite_names.xml' does not exist
No tests to execute
Elapsed time: 0 minutes 0.6 seconds
I've got an interesting problem with Grunt-Concurrent. When a registered task such as Uglify or Karma runs without errors everything is fine. But if Uglification or Karma tests fail then Grunt-Concurrent will continuously loop until the error is fixed. This was annoying with Uglify but with Karma unit tests it's really hard to have it looping until the bug is fixed.
Any ideas of how to solve this?
I can't really provide examples of our exact set up.
It will just keep running with this message until the unit tests all pass:
Running "watch:karma" (watch) task
Waiting...
Running "karma:unit:run" (karma) task
Firefox 39.0.0 (Mac OS X 10.10.0) it should work should work FAILED
Expected true to be false.
...: Executed 2 of 2 (1 FAILED) (0.007 secs / 0.004 secs)
Warning: Task "karma:unit:run" failed.
It then runs it over again and again and again.
Looks like the issue was actually with grunt-contrib-watch and once I started going down that path I found the answer.
I found this great SO question: Prevent `grunt-watch` from looping when there is a syntax error in less files?
I found that by changing spawn: true that it no longer loops like it used to.
When running PHPUnit, if there are no errors it runs in aprox. 6 seconds, nevertheless if there are errors it can take up to 3 or 4 minutes....Only to report the error information.
I'm running PHP version 5.5.3 and PHPUnit version 3.7.28 in Windows.
The tests are painfully simple, even the simplest assertion fail makes the test report take forever.
class MyTest extends PHPUnit_Framework_TestCase
{
public function someTest() {
$this->assertEquals(true, false); // takes forever to report
}
}
Running a database test with --debug I get the following output
D:\xampp\htdocs\blueberry-cms
λ phpunit --debug tests\unit\DatabaseTest.php
PHPUnit 3.7.28 by Sebastian Bergmann.
Starting test 'DatabaseTest::testCRUD'.
F
Starting test 'DatabaseTest::testRetrieval'.
.
Time: 8.91 seconds, Memory: 2.25Mb
There was 1 failure:
1) DatabaseTest::testCRUD
Failed asserting that 19 matches expected 18.
D:\xampp\htdocs\blueberry-cms\tests\unit\DatabaseTest.php:37
FAILURES!
Tests: 2, Assertions: 22, Failures: 1.
The test runs fine until "There was 1 failure:" then it stops outputting text... stops for like 2 or 3 minutes and then finishes outputting text.
I'm having the same issue. I found out a workaround:
Download a .phar copy of PHPunit and put it in your document root. (link to phpunit downloads page)
Instead of running the phpunit command from the php "bin folder" (or wherever you have it in your $PATH), run php phpunit.phar from the root of my repository.
Ej: php phpunit.phar --bootstrap=./tests/bootstrap.php ./tests/
Probably this issue can be solved by updating phpunit.
(In my case (PHP 5.6) the phpunit version in my PHP "bin folder" is 3.7.21, while the .phar I downloaded is version 4.8.36)
I haven't tested this, but it should work as well:
Replace the phpunit version located in your php "bin folder" (or in wherever you have it in your $PATH)
Now every time you run phpunit you will be using the latest version, whithout having to copy the .phar to every new project.