I am converting my azure pipeline to YAML pipeline. When I trigger the build, it gets failed on the Unit test step and gives the error as below
[error]vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
I could not find a way to add the VSTEST_CONNECTION_TIMEOUT value anywhere. Could you please help me with this.
Here is the sample .yml I am using
- task: VSTest#2
displayName: 'Test'
inputs:
testAssemblyVer2: '**\bin\**\Tests.dll'
testFiltercriteria: 'TestCategory=Unit'
runSettingsFile: XYZ.Tests/codecoverage.runsettings
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
diagnosticsEnabled: true
I would recommend you to use the dotnetCli task instead. It is shorter, clearer and straight forward (It will have the "same" effect as executing dotnet test in your console)
- task: DotNetCoreCLI#2
displayName: 'Run tests'
inputs:
command: 'test'
Even in microsoft documentation page, they use the DotNetCoreCLI task.
If the vstest task can run successfully on your classic pipeline. It should be working in yaml pipeline too. You can check the agent pool selection and the task's settings to make sure they are same both in yaml and classic pipeline.
1,Your Unit tests seem like running on Vs2017 in the yaml pipeline. You can try running the pipeline on windows-latest agent to run the tests on Vs2019.
If your pipeline has to run on specific agent. You can use VisualStudioTestPlatformInstaller task to download the latest version. Then set the vsTestVersion: toolsInstaller for Vstest task. See below:
- task: VisualStudioTestPlatformInstaller#1
- task: VSTest#2
displayName: 'Test'
inputs:
testAssemblyVer2: '**\bin\**\Tests.dll'
...
...
vsTestVersion: toolsInstaller
2,You can also check out the solution in this thread. As it mentioned in the solution deleting the entire solution folder, re-cloning the project. If you were running your pipeline on your self-hosted agent. You can try using Checkout in the yaml pipeline to clean the source folder before cloning your repo. See below:
steps:
- checkout: self
clean: true
You can also try adding below to your codecoverage.runsettings file under element <CodeCoverage> to exclude the microsoft assemblies as mentioned in the thread.
<ModulePath>.*microsoft\.codeanalysis\.csharp\.dll$</ModulePath>
<ModulePath>.*microsoft\.codeanalysis\.csharp\.workspaces\.dll$</ModulePath>
<ModulePath>.*microsoft\.codeanalysis\.dll$</ModulePath>
<ModulePath>.*microsoft\.codeanalysis\.workspaces\.dll$</ModulePath>
3,You can also try updating 'Microsoft.NET.Test.Sdk' to latest version for test projects.
I am trying to get the results of a php unit tests folder , but not using the CLI, and instead using a php file.
I would like to get the answer ( OK, 4 tests passed ) in a variable or something so I can decide whether the script should execute or not, what's the best way to do this? i don't want to use batch files, I want to force the execution of the tests, inside the library itself.
I started with
require_once 'PHPUnit/Autoload.php';
When I included the tests, don't know how to start them thought.
Any advice?
You can check PHPUnits exit code, 0 means no test failed.
To run the tests from a php file, then to check the exit code, try something like this:
//Composer installs phpunit to /vendor/bin/phpunit
exec('/vendor/bin/phpunit', $result, $exitCode);
if ($exitCode == 0) {
// continue exiting the script
} else {
// there was a test failure, more info will be in $result
}
If you're looking for an enterprise quality solution, look into a Continuous Integration product like Jenkins or Bamboo.
I am trying to use Jenkins xUnit plug-in for my Qt unit test project, but I cannot make it work...
Here is What I've done so far:
First of all, I build my unit test project with qmakebuilder plug-in (providing the .pro to the qmakebuilder plug-in), then I add an Execute Shell build section
where I first rm -f the testResult xml file, following to that, I run the test binary with -xunitxml flag to have it generate the xml file for me And I name the xml file testResult.xml, in other word:
rm -f /home/guest/QT/unitTest/testResult.xml
cd /home/guest/QT/unitTest
./tst_unittesttest -xunitxml > testResult.xml
Finally in Post Build Action, I select Publish xUnit test result and specify the pattern as *.xml.
Building the job will fail though, Here is the output that I get when building the job with Jenkins:
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing QTestlib-Version N/A
[xUnit] [INFO] - [QTestlib-Version N/A] - 1 test report file(s) were found with
the pattern '*.xml' relative to '/home/guest/QT/unitTest' for the testing framework
'QTestlib-Version N/A'.
[xUnit] [ERROR] - The converted file for the result file '/home/guest/QT/unitTest
/testResult.xml' (during conversion process for the metric 'QTestlib') is not
valid.
The report file has been skipped.
[xUnit] [ERROR] - The plugin hasn't been performed correctly: hudson.util.IOException2:
Failed to read /home/guest/QT/unitTest/generatedJUnitFiles/QTestlib/TEST--735044756.xml
Build step 'Publish xUnit test result report' changed build result to FAILURE
Build step 'Publish xUnit test result report' marked build as failure
Finished: FAILURE
As you can see there is problem converting my test result xml file to what xUnit plug-in can read. My testResult.xml file looks like this:
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="2" failures="0" tests="4" name="UnitTestTest">
<properties>
<property value="4.7.4" name="QTestVersion"/>
<property value="4.7.4" name="QtVersion"/>
</properties>
<testcase result="pass" name="initTestCase">
<!-- message="called before everything else" type="qdebug" -->
</testcase>
<testcase result="pass" name="myFirstTest"/>
<testcase result="pass" name="mySecondTest"/>
<testcase result="pass" name="cleanupTestCase">
<!-- message="called after myFirstTest and mySecondTest" type="qdebug" -->
</testcase>
<system-err>
<![CDATA[called before everything else]]>
<![CDATA[called after myFirstTest and mySecondTest]]>
</system-err>
</testsuite>
but TEST--735044756.xml is an one line xml file: <?xml version="1.0" encoding="UTF-8"?>
Anyone has a clue where am I doing what wrong? There should be a problem with generating the output xml file.
On a different note, I have added the log thingi in the system log Jenkins menu, it doesn't seem to be the reason for this problem though... at least IMO
Executing the test binary with -xml flag will just work fine. Weirdly using -xunitxml creates corrupt xml file which its <testcase> doesn't have time attribute. Running the test binary only with -xml results in building the job successfully.
Sorry for another 'phpunit doesn't work' question. It used to work for years now. Today I reinstalled PEAR and phpunit for reasons not connected to this problem. Now when I run phpunit as I usually did. Nothing happens. The cli just shows me a new line, no output whatsoever.
Has anyone encountered this problem or has an idea what could have caused it.
PHPUnit Version: 3.5.15
PEAR Version: 1.9.4
PHP Version: 5.3.8
Windows 7
I'm on OSX and MAMP. To get error messages I had to adjust the following entries in php.ini:
display_errors = On
display_startup_errors = On
Please not that this has to go into /Applications/MAMP/bin/php/php5.3.6/conf/php.ini .
For future reference, for those who are facing any problem with PHPUnit, and PHPUnit is failing silently, just add this three lines inside phpunit.xml:
<phpunit ....... >
...
...
<php>
<ini name="display_errors" value="true"/>
</php>
</phpunit>
After that run the tests again, and now you can see why PHPUnit is failing,
AND ... ENJOY UNIT TESTING :)
I know the orignal poster's question is already answered, but just for any people searching in the future: one thing that can cause PHPUnit to fail silently (i.e. it just stops running tests without telling you why) is that it has an error handler that it set up before each test run, which is intended to capture errors and display them at the end of the test run. The problem is that certain errors halt execution of the whole test run.
What I generally do when this happens, as a first step, is reset the error handler to something that will immediately output the error message. In my base test class I have a method called setVerboseErrorHandler, which I'll call at the top of the test (or in setUp) when this happens. The below requires php 5.3 or higher (due to the closure), so if you're on 5.2 or lower you can make it a regular function.
protected function setVerboseErrorHandler()
{
$handler = function($errorNumber, $errorString, $errorFile, $errorLine) {
echo "
ERROR INFO
Message: $errorString
File: $errorFile
Line: $errorLine
";
};
set_error_handler($handler);
}
Create the simplest test class you can without a bootstrap.php or phpunit.xml to first verify that your new installation works. PHPUnit will stop without any message if it cannot instantiate all of the test cases--one for each test method and data provider--before running any tests.
You have already figured out how to get it to work, but my solution was a little different.
First thing you can do is check the exit status. If it's not 0, then PHP exited and because of the INI configuration settings set, none of the PHP error messages were outputted. What I did was to enable the "display_errors" INI setting, and set "error_reporting" to E_ALL. I was then able to identify errors such as PHP not being able to parse a certain script. Once I fixed that, PHPUnit ran properly.
I managed to spectacularly paint myself in a corner with a custom "fatal error handler" that in certain rare conditions turned out to output nothing. Those conditions, in accordance with Murphy's Law, materialized once I had forgotten the handler was in place.
Since it wasn't really a "PHPunit problem", none of the other answers helped [although #David's problem was at the bottom the same thing], even though the symptoms were the same - phpunit terminating with no output, no errors, no log and no clue.
In the end I had to resort to a step-by-step tracing of the whole test suite by adding this in the bootstrap code:
register_shutdown_function(function() {
foreach ($GLOBALS['lastStack'] as $i => $frame) {
print "{$i}. {$frame['file']} +{$frame['line']} in function {$frame['function']}\n";
}
});
register_tick_function(function() {
$GLOBALS['lastStack'] = debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS, 8);
});
declare(ticks=1);
If anyone ever manages to do worse than this, and somehow block stdout as well, this modification should work:
register_shutdown_function(function() {
$fp = fopen("/tmp/path-to-debugfile.txt", "w");
foreach ($GLOBALS['lastStack'] as $i => $frame) {
fwrite($fp, "{$i}. {$frame['file']} +{$frame['line']} in function {$frame['function']}\n");
}
fclose($fp);
});
an old thread this one, but one I stumbled across when having the same problem.
I had the same problem, nothing being returned to console including print, print_r, echo etc.
solved it by using --stderr as a test-runner option.
Check that you haven't written any logic into your code that just dies, with no output. For example,
<?php
if (!array_key_exists('SERVER_NAME', $_SERVER)) {
die();
}
This was exactly my case; I'd made some assumptions about the environment which were correct when running the code via Apache, but weren't fulfilled when running from CLI and the code did not echo any output.
PHPUnit tried to include the bootstrap file before giving the usual init output, but died during the bootstrapping proccess, hence exiting with status 0 and no output.
If when you run from command line a recent version of phpunit like this
> php phpunit
or
> ./phpunit
or
> php ./phpunit.phar
or
> ./phpunit.phar
And you immediatly return to the prompt with no messages, this is probably due to a "suhosin secutiry" setup.
phpunit is now a "phar" package including all libraries. To be able to run such file when php has the suhosin security module enabled, you must first set this
suhosin.executor.include.whitelist = phar
into you php.ini file (for example, with debian/ubuntu, you may have to edit file /etc/php5/conf.d/suhosin.ini
i tried everything here, but nothing worked until i tried phpunit --no-configuration simpletest.php. that finally gave me some output, which implies that my phpunit.xml.dist file is broken. (i'll come back and update this once i debug it.)
the contents of simpletest.php are below, but any test file should work.
<?php
use PHPUnit\Framework\TestCase;
final class FooTest extends TestCase
{
public function testFoo()
{
$this->assertEquals('x', 'y');
}
}
Check if the phpunit you're running and the one you installed are the same:
$ pear list phpunit/phpunit
...
script /path/to/phpunit
...
Try to execute exactly that phpunit with the full path.
Then check your PATH variable and see if the correct directory is in it. If not, fix that.
If that does not help, use write something into the phpunit executable, e.g. "echo 123;" and run phpunit. Check if you see that.
For me the conflict was with Xdebug's directive
xdebug.remote_enable=1
If you are using composer, check to make sure your included PHP files are not ending the code executions. The same goes for when you included certain PHP files explicitly.
In my case, I was working on a WordPress plugin and one of the PHP files I included directly in composer.json (which I don't want to load through PSR-4 because WordPress's coding standards don't support it yet) had this code on top;
if (!defined('ABSPATH')) {
exit(); // exit if accessed directly
}
And since ABSPATH will not be defined when I run the tests directly, the code was exiting.
This is because, since I told Composer to always load these files each time, this part of the code will execute, while the other files included though autoload PSR will load on demand.
So check to make sure any of the files you included are not stopping the code execution. If it happens, then even when you run phpunit --info the code will still exit and you won't see any output.
I was facing a seems problem. I could run phpunit from root directory but not from anywhere else. so I put the "--configuration" tag, and point it to my xml configuration.
$ ./<path_to_phpunit>phpunit --configuration <path_to_phpunitxml>/phpunit.xml
The path to phpunit is optinal, I used it because I installed locally by composer.
In /composer/vendor/phpunit/phpunit/src/TextUI/Command.php in main() function in catch (Throwable $t) {} block var_dump (echo / print_r) exception. If an exception exists, you, probably, will solve the current problem.