I have a SonarQube installation and am running it against a .Net core application with multiple xUnit projects. The SonarQube picks up the bugs and smells fine but like so many people, I can't get the code coverage to complete.
At present, I am generating cobertura coverage XML files using the following command:
dotnet test --collect:"XPlat Code Coverage"
I then copy these to a centralized directory from each of the test projects and use the following command to run/import the files:
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.vscoveragexml.reportsPaths=".\TestResults\*.xml"
dotnet build
dotnet SonarScanner end
According to the logs, the files are found but are not ingested.
INFO: Parsing the Visual Studio coverage XML report
C:...\TestResults\5.coverage.cobertura.xml WARN: Could not import
coverage report '..\TestResults\5.coverage.cobertura.xml' because
'Missing root element in
C:...\TestResults\5.coverage.cobertura.xml at line 2'
I've confirmed that the files contain valid XML that looks to be correct, but I'm not even sure that SonarQube is supposed to accept cobertura reports.
Is my approach the way others have gone when trying to get xUnit coverage reports into SonarQube?
Is there a better way? I'm happy to use OpenCover or similar if that is easier.
In case anyone comes across this in the future, I gave up trying to use the built in coverage in VS and used DotCover. Bit of an easier setup and now seem to have it working to a degree.
Download/extract dotCover and add the folder to Path
dotCover.exe --output=AppCoverageReport.html --reportType=HTML dotnet -- test
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.dotcover.reportsPaths=AppCoverageReport.html
dotnet build
dotnet SonarScanner end
Still having an issue where the test coverage gathering seems to be pulling in too much and as a result is causing an error. Investigating but not related to this question.
Related
Iam trying to push my asp.net(mvc) code azure devops using Git bash, but I am getting VSBuild error saying:
Error: Preparing the test sources file failed.
Error: No test sources found matching the given filter '**\*test*.dll,!**\*TestAdapter.dll,!**\obj\**'
VsTest task failed.
That error should be separate from a git push.
Meaning:
you are pushing your sources
but your Azure pipeline might not be correctly configured
See this thread for instance:
From this error (No test sources found matching the given filter '**\*,!**\obj\*'), we can see that you release pipeline do not get the build Artifacts sources.
So you need add a "Copy Files" task before your publish task in the build pipeline:
Or add a "Copy Files" task before your "VsTest" task in the release pipeline.
If it still an issue for you, please share your Organization name and log in account.(please choose "viewable by Microsoft Only")
See also "Publishing test assemblies with artifacts in order to use them during functional tests on VSTS", and this thread, which illustrates a correct pipeline configuration:
From the error we know that Vstest task failed to find the test assemblies.
It may because the test files you defined in Test Files field is incorrect, or you specified a wrong search folder where the test assemblies are not exist. For below example. Vstest task will search for all the files that matching *test*.dll in folder $(System.DefaultWorkingDirectory)(eg. c:/agent/_work/1/s). Check here to find more predefined variables.
The fix is simple if you know where the test assemblies is copied by VsBuild task. You can check the log of Visual Studio Build task, to find out where the test.dll files are located.
Hope you find above helpful.
In DevOps you can create a dotnet build task where "Path to project(s)" can be set to **/*.csproj to build all the projects.
This works just fine, but it (obviously) also builds my *.Test.csproj projects.
I found some posts mentioning exclude patterns, so I tried doing the same as in those posts and tried the following combinations:
**/*.csproj;-**/*.Test.csproj
**/*.csproj;!**/*.Test.csproj
**/*.csproj;-:**/*.Test.csproj
**/*.csproj;!:**/*.Test.csproj
For all attempts I get the following error in the DevOps log:
Project file(s) matching the specified pattern were not found.
So, how can I create a dotnet build task to build all my projects except the *.Test.csproj projects?
here's what I've been using to run all tests except for some of them:
Test/**/*.csproj
!**/*.Billing.Test.csproj
!**/*.Queues.Test.csproj
!**/*.WidgetDataProvider.Test.csproj
so by the same token you need to use:
**/*.csproj
!**/*.Test.csproj
as far as I understand, order matter, so this will not work:
!**/*.Test.csproj
**/*.csproj
I have set up a pipeline for my .NET Core project in Azure Devops using the '.NET Core with SonarCloud' template. When I build the analysis gets run in SonarCloud but with 0% Code coverage (I have tests in my solution).
No matter what configuration tweaks I make to the build I cannot get the code coverage working.
What am I missing?
I came across this article and https://dejanstojanovic.net/aspnet/2019/may/publishing-code-analysis-to-sonarcloud-from-azure-build-pipeline/ implemented the powershell script described in it but still I get no code coverage in SonarCloud
I tried using coverlet as described here but still no joy
https://gunnarpeipman.com/aspnet/azure-devops-code-coverage/
My pipeline consists of the following tasks
.NET Core - Restore
Prepare Analysis Configuration
.NET Core - Build
.NET Core - Test
Run Code Analysis
Publish Quality Gate Result
My test task is configured:
Arguments: --configuration $(BuildConfiguration)
Publish test results and code coverage - checked
In the console of the Run Code Analysis task I get:
10:43:54.7 Fetching code coverage report information from TFS...
10:43:54.702 Attempting to locate a test results (.trx) file...
10:43:54.753 Looking for TRX files in: C:\\TFSBuilds\\TJPYHG04-GHJ01\\_work\\475\\TestResults
10:43:54.755 No test results files found
10:43:54.81 Did not find any binary coverage files in the expected location.
10:43:54.811 Falling back on locating coverage files in the agent temp directory.
10:43:54.812 Searching for coverage files in C:\\TFSBuilds\\TJPYHG04-GHJ01\\_work\\_temp
10:43:54.814 No coverage files found in the agent temp directory.
hope this answer still relevant to you.
Recently I have similar problem as you and I am also using Azure DevOps for my case.
This is how I solved it.
Step 1 - Change directory into your unit testing sub folder (same as the unit-testing .csproj file) and run the following dotnet command.
dotnet add package coverlet.msbuild
Step 2 - Add followings into SonarCloudPrepare Task Additional Properties or append directly into the yml file (if you're using yml instead of classic editor)
extraProperties: |
sonar.exclusions=**/obj/**,**/*.dll
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
The directory is up to your choice to configure.
Or you can also create a file at your repo titled sonar-project.propertiesand store all the relevant SonarCloud properties inside.
Step 3 - Add followings into your dotnet test task
- task: DotNetCoreCLI#2
inputs:
command: 'test'
arguments: '--configuration $(BuildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx'
testRunTitle: 'dotnet test'
You may notice there's a tickbox for "Publish test results and code coverage" but I still prefer using /p:CollectCoverage=true.
You may also test locally to run the dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx command and coverage.opencover.xml will be generated at your unit testing folder.
Refer to reference 2 & 3 for more parameters and their description.
Side Note: If you're doing any Sonar test inclusion properties, note that Sonar.Tests and Sonar.Test.Inclusions their Test(s) are different. lol
References:
Using SonarCloud in Azure pipelines
Analysis Parameters (official docs)
Test Coverage & Execution (official docs)
Hope this helps :)
Not getting code coverage in SonarCloud from an Azure Devops .NET core build
This issue may caused by vstest output path changed recently:
The output path of the vstest coverage file change from
D:\a\1\s\TestResults\... to D:\a\_temp\...
Which broke subsequent scripts in the pipeline (like, codecoverage.exe to convert to xml and later import to sonarqube).
Microsoft suggest that use the rest APIs to check for the test artefacts and re-download them to the build agent.
More investigation on this issue, you can check the thread Azure DevOps (VSTS) extension no longer import coverage and unit tests automatically for the issue tracking.
Fortunately, SonarSourcer team have just release new versions of the SonarQube (v4.6.3) and SonarCloud (v1.6.3) extensions to address the coverage issue and the regression.
Hope this helps.
I had two build steps in VSTS:
To run tests (VSTS cmd task): DOTNET test -xml TEST-results.xml
To publish test results step (VSTS test publish task): format=XUnit and the file name from previous step
But after I upgraded to VS2017 the -XML tag is not working anymore. I changed step one to use this:
test --logger "trx;LogFileName=TEST-results.xml"
but the second step throws an error "Invalid results file. Please make sure the Test Result Format field in the task matches the result format of the file"
Is there another way to run .NetCore tests on VSTS? or am I doing something wrong?
Thanks,
starain-MSFT's answer will work, unless you want/need the xunit tests to be logged using an xunit logger. In that case, you'll need to do two things.
Add https://www.nuget.org/packages/XunitXml.TestLogger/1.0.2-pre-rtm as package ref to your test project, either through 'Manage NuGet Packages' in VS, or by adding the ref in your csproj file manually, i.e.
<PackageReference Include="xunitxml.testlogger" Version="1.0.2-pre-rtm" />
Modify the VSTS dotnet test build step to use this logger:
dotnet test -a:. -l:xunit
The -a:. switch, which specifies the adapter path, is only necessary for CLI tools V15.0, in 15.1 that can be removed (as discussed here). As of today, the VS2017 Hosted Queue is using 15.0, so you'll need the -a:. on VSTS for now.
The -l:xunit uses the friendlyname, which I think isn't so friendly since you have to dig into the source code for the particular logger to find the attribute where it is specified (as seen here for xunit and here for trx)
The docs for the -l switch are spotty to say the least, but in the github for vstest, there is a document which talks about test loggers and links to their repositories and nuget packages, which after you look at the source for the friendlyname, gets you all the way there for whichever logger you need. If you need a custom logger, those are great examples to help understand how to implement.
Finally, the publish step that you used originally should be fine, since the output file is still called TestResults.xml
Change "Test Result Format" to "VSTest of Publish Test" result step/task, it reads the result file correctly.
Use dotnet xunit instead of dotnet test. See Getting Started with .NET Core.
I am running automation tests in my local workspace. The test project hasn't been added to a build integration process yet. So I'd like to generate allure reports without running a build.
The necessary dependencies were added to Ivy.xml according to instructions (see https://github.com/allure-framework/allure-core/wiki/TestNG -> Ant), but I don't have a build.xml file, so I have no idea of what the next step is about.
Could you please provide instructions for generating allure xml reports after running the tests, but without running a build?
Thanks in advance
Generally speaking you should AllureTestListener to your tests classpath so TestNG can automatically use this listener. This is done by adding the following dependency to ivy.xml:
<dependency org="ru.yandex.qatools.allure" name="allure-testng-adaptor" rev="1.4.11"/>
Then you need to run your tests in any way you want (e.g. by using your IDE runner). After running tests you should have a set of XML files generated by default in target/allure-results.
After getting these files just use a command line report generator:
$ allure generate -v 1.4.11 target/allure-results