I have code coverage running from PhpUnit, and I can generate a diff file in Git that will show me what's about to be released.
How would I go about mapping one to the other, so that I can say the code we are about to release is covered by our tests?
Ideally, this would be something that could be run automatically as a build step in Jenkins.
Before you run your phpunit tests by your Ant / maven build, run a phpscript. In this php script copy every file specified in the git file which is about to be release along with the matching unit tests to a temporary directory. Of course this requires that you use proper naming conventions and structure for unit tests. Run phpunit on the temporary directory
-application
-app
-tests
-temp
-app
-tests
when you run your phpunit in your build file include a clover report
phpunit --coverage-clover="clover.xml" temp/tests
In Jenkins you can mark the build as unstable if the clover.xml doesn't meet a specified % coverage.
This should work. Let me know how it works out.
Related
On a Windows machine there is a GitLab-Runner run from a domain user with admin rights. When I log as this user and call dotnet build -c release to build an ASP.NET Core app, the dll has all the information from the AssemblyInfo.cs file. When I do the same as part of a CI job, the produced dll is missing all this information (for example the version number).
The AssemblyInfo.cs file is not part of the repository, instead, it is produced by a prebuild event (using gitWCRev.exe tool). However after running some tests I can see that the AssemblyInfo.cs is actually generated when the job is run by the runner.
Any help as to why the file is ignored and how to overcome this issue would be appreciated.
At first I thought that this might be related to Pre-build task of Visual Studio project fails in GitLab Runner issue, but I don't get any build errors.
On the same machine, I build a .Net Framework app which has the same AssemblyInfo setup, but is compiled using msbuild /property:Configuration=Release by the runner and the produced dll file has all the expected information.
It turns out the problem was partially related to the AssemblyInfo.cs file not being part of the repository.
SDK-style csproj normally don't list the files in the project, but figure them out based on the folder contents. When dotnet build was run, AssemblyInfo.cs wasn't present in the project directory (GitLab-Runner usually clears out files not present in the repository before starting a job/pipeline), so build tools had no idea they needed to load it. It made no difference that the file was being created by the build tools.
The solution proved to be creating an empty AssemblyInfo.cs file before running dotnet build. This way build tools knew they needed to compile it. Actual compilation still happened after prebuild events, so all the needed information was there.
I created the empty AssemblyInfo.cs file using PowerShell:
New-Item -Path "Properties/AssemblyInfo.cs" -ItemType File
Also, checking the build logs helped me finally figure it out. To get the build logs I've called build tools like this:
dotnet build -c release /flp:v=diag
The .Net Framework app didn't have this problem because it wasn't an SDK-style project. All the files needed for compilation were listed in the project file and build tools knew to load them, even if they were created during the prebuild event.
Getting different mutation scores each time I run Stryker from the root of the project which has multiple test projects.
Does stryker support running '.sln' files from the root of the project?
Command used from the root of the project:
dotnet stryker --solution-path "Project.sln"
Env: Linux, Windows, .NET core
Any insight is appreciated. Thank you.
Solution mode isn't working properly yet thus it's not really documented. You can use this script for now - a little raw but works.
I have faced the same issue previously and conceded defeat. The path of least resistance for me in terms of unit test code coverage and also stryker mutation testing is to have all unit tests in a single project and execute stryker from the folder where the test.csproj is location. My build pipeline script (running in GitLab on Linux) looks something like this:
- dotnet build
- cp ./resources/stryker/stryker-config.json ./*.Tests
- cd ./*.Tests
- dotnet stryker
It first builds the whole solution, then copies a known config file into the test project folder, then change directory into that folder and execute stryker.
I have a SonarQube installation and am running it against a .Net core application with multiple xUnit projects. The SonarQube picks up the bugs and smells fine but like so many people, I can't get the code coverage to complete.
At present, I am generating cobertura coverage XML files using the following command:
dotnet test --collect:"XPlat Code Coverage"
I then copy these to a centralized directory from each of the test projects and use the following command to run/import the files:
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.vscoveragexml.reportsPaths=".\TestResults\*.xml"
dotnet build
dotnet SonarScanner end
According to the logs, the files are found but are not ingested.
INFO: Parsing the Visual Studio coverage XML report
C:...\TestResults\5.coverage.cobertura.xml WARN: Could not import
coverage report '..\TestResults\5.coverage.cobertura.xml' because
'Missing root element in
C:...\TestResults\5.coverage.cobertura.xml at line 2'
I've confirmed that the files contain valid XML that looks to be correct, but I'm not even sure that SonarQube is supposed to accept cobertura reports.
Is my approach the way others have gone when trying to get xUnit coverage reports into SonarQube?
Is there a better way? I'm happy to use OpenCover or similar if that is easier.
In case anyone comes across this in the future, I gave up trying to use the built in coverage in VS and used DotCover. Bit of an easier setup and now seem to have it working to a degree.
Download/extract dotCover and add the folder to Path
dotCover.exe --output=AppCoverageReport.html --reportType=HTML dotnet -- test
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.dotcover.reportsPaths=AppCoverageReport.html
dotnet build
dotnet SonarScanner end
Still having an issue where the test coverage gathering seems to be pulling in too much and as a result is causing an error. Investigating but not related to this question.
I am running automation tests in my local workspace. The test project hasn't been added to a build integration process yet. So I'd like to generate allure reports without running a build.
The necessary dependencies were added to Ivy.xml according to instructions (see https://github.com/allure-framework/allure-core/wiki/TestNG -> Ant), but I don't have a build.xml file, so I have no idea of what the next step is about.
Could you please provide instructions for generating allure xml reports after running the tests, but without running a build?
Thanks in advance
Generally speaking you should AllureTestListener to your tests classpath so TestNG can automatically use this listener. This is done by adding the following dependency to ivy.xml:
<dependency org="ru.yandex.qatools.allure" name="allure-testng-adaptor" rev="1.4.11"/>
Then you need to run your tests in any way you want (e.g. by using your IDE runner). After running tests you should have a set of XML files generated by default in target/allure-results.
After getting these files just use a command line report generator:
$ allure generate -v 1.4.11 target/allure-results
Using PHPStorm on a project with Composer and PHPUnit (all of which have already work correctly) how do I run PHPUnit test on a composer plugin inside "vendor" in this case mikey179/vfsStream?
My project structure looks like this:
myscripts/
vendor/
- mikey179
- vfsStream
composer.json
I want to run PHPUnit to test vfsStream plugin.
I was able to run Testing on my own test code (outside vendor directory).
When I run PHPUnit test (That little green play button on top) I got the following:
usr/bin/php /tmp/ide-phpunit.php --bootstrap /path/to/project/vendor/autoload.php --configuration /path/to/project/vendor/mikey179/vfsStream/phpunit.xml.dist /path/to/project/vendor/mikey179/vfsStream
Testing started at 5:53 PM ...
PHPUnit 3.8-g8d770d8 by Sebastian Bergmann.
Configuration read from /path/to/project/vendor/mikey179/vfsStream/phpunit.xml.dist
Time: 2.22 seconds, Memory: 2.25Mb
No tests executed!
PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected the timezone 'UTC' for now, but please set date.timezone to select your timezone. in /path/to/project/vendor/phpunit/php-code-coverage/PHP/CodeCoverage/Report/HTML.php on line 127
PHP Stack trace:
PHP 1. {main}() /tmp/ide-phpunit.php:0
PHP 2. IDE_Base_PHPUnit_TextUI_Command::main() /tmp/ide-phpunit.php:506
PHP 3. PHPUnit_TextUI_Command->run() /tmp/ide-phpunit.php:268
PHP 4. PHPUnit_TextUI_TestRunner->doRun() /path/to/project/vendor/phpunit/phpunit/PHPUnit/TextUI/Command.php:173
PHP 5. PHP_CodeCoverage_Report_HTML->process() /path/to/project/vendor/phpunit/phpunit/PHPUnit/TextUI/TestRunner.php:465
PHP 6. date() /path/to/project/vendor/phpunit/php-code-coverage/PHP/CodeCoverage/Report/HTML.php:127
Generating code coverage report in HTML format ... done
Process finished with exit code 0
From what I see it's an issue with PhpStorm's helper/wrapper script (/tmp/ide-phpunit.php), which is used for integration purposes (track tests progress in real time -- messages must be formatted in specific way; etc).
Test files do not have standard and expected Test.php ending -- instead they have custom TestCase.php .. and helper script seems to be unable to apply that config option from phpunit.xml.dist file correctly (it will search for *Test.php files only).
If you rename all *TestCase.php file to be *Test.php it will start working in PhpStorm (I've renamed just a few .. and they got detected and executed just fine).
Right now I may only suggest to either rename test files as described above (which is not a good idea in long run if you plan to run those tests on regular basis and keep downloading latest sources of that library) or submit a bug ticket to PhpStorm Issue Tracker explaining the situation: http://youtrack.jetbrains.com/issues/WI and hopefully devs will fix it soon.
You don't run a test on a package in vendor/. You clone the repo and follow the instructions they've given in the testing documentation. In the case of mikey179/vfsStream, there's a github repo: https://github.com/bovigo/vfsStream and it even has a .travis.yml file you can read for inspiration: https://github.com/bovigo/vfsStream/blob/master/.travis.yml
If you're concerned that you've got a version of vfsstream that doesn't pass tests, then you can look at the version that's installed in your project with composer show mikey179/vfsStream and then check out that tag in the repo before running the tests.
The reason is that the package might very well have different dev dependencies than your project, so no matter what else you do you'll end up with broken test runs. Composer ignores dev dependencies for packages other than the main project.