I am running automation tests in my local workspace. The test project hasn't been added to a build integration process yet. So I'd like to generate allure reports without running a build.
The necessary dependencies were added to Ivy.xml according to instructions (see https://github.com/allure-framework/allure-core/wiki/TestNG -> Ant), but I don't have a build.xml file, so I have no idea of what the next step is about.
Could you please provide instructions for generating allure xml reports after running the tests, but without running a build?
Thanks in advance
Generally speaking you should AllureTestListener to your tests classpath so TestNG can automatically use this listener. This is done by adding the following dependency to ivy.xml:
<dependency org="ru.yandex.qatools.allure" name="allure-testng-adaptor" rev="1.4.11"/>
Then you need to run your tests in any way you want (e.g. by using your IDE runner). After running tests you should have a set of XML files generated by default in target/allure-results.
After getting these files just use a command line report generator:
$ allure generate -v 1.4.11 target/allure-results
Related
On a Windows machine there is a GitLab-Runner run from a domain user with admin rights. When I log as this user and call dotnet build -c release to build an ASP.NET Core app, the dll has all the information from the AssemblyInfo.cs file. When I do the same as part of a CI job, the produced dll is missing all this information (for example the version number).
The AssemblyInfo.cs file is not part of the repository, instead, it is produced by a prebuild event (using gitWCRev.exe tool). However after running some tests I can see that the AssemblyInfo.cs is actually generated when the job is run by the runner.
Any help as to why the file is ignored and how to overcome this issue would be appreciated.
At first I thought that this might be related to Pre-build task of Visual Studio project fails in GitLab Runner issue, but I don't get any build errors.
On the same machine, I build a .Net Framework app which has the same AssemblyInfo setup, but is compiled using msbuild /property:Configuration=Release by the runner and the produced dll file has all the expected information.
It turns out the problem was partially related to the AssemblyInfo.cs file not being part of the repository.
SDK-style csproj normally don't list the files in the project, but figure them out based on the folder contents. When dotnet build was run, AssemblyInfo.cs wasn't present in the project directory (GitLab-Runner usually clears out files not present in the repository before starting a job/pipeline), so build tools had no idea they needed to load it. It made no difference that the file was being created by the build tools.
The solution proved to be creating an empty AssemblyInfo.cs file before running dotnet build. This way build tools knew they needed to compile it. Actual compilation still happened after prebuild events, so all the needed information was there.
I created the empty AssemblyInfo.cs file using PowerShell:
New-Item -Path "Properties/AssemblyInfo.cs" -ItemType File
Also, checking the build logs helped me finally figure it out. To get the build logs I've called build tools like this:
dotnet build -c release /flp:v=diag
The .Net Framework app didn't have this problem because it wasn't an SDK-style project. All the files needed for compilation were listed in the project file and build tools knew to load them, even if they were created during the prebuild event.
I have a SonarQube installation and am running it against a .Net core application with multiple xUnit projects. The SonarQube picks up the bugs and smells fine but like so many people, I can't get the code coverage to complete.
At present, I am generating cobertura coverage XML files using the following command:
dotnet test --collect:"XPlat Code Coverage"
I then copy these to a centralized directory from each of the test projects and use the following command to run/import the files:
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.vscoveragexml.reportsPaths=".\TestResults\*.xml"
dotnet build
dotnet SonarScanner end
According to the logs, the files are found but are not ingested.
INFO: Parsing the Visual Studio coverage XML report
C:...\TestResults\5.coverage.cobertura.xml WARN: Could not import
coverage report '..\TestResults\5.coverage.cobertura.xml' because
'Missing root element in
C:...\TestResults\5.coverage.cobertura.xml at line 2'
I've confirmed that the files contain valid XML that looks to be correct, but I'm not even sure that SonarQube is supposed to accept cobertura reports.
Is my approach the way others have gone when trying to get xUnit coverage reports into SonarQube?
Is there a better way? I'm happy to use OpenCover or similar if that is easier.
In case anyone comes across this in the future, I gave up trying to use the built in coverage in VS and used DotCover. Bit of an easier setup and now seem to have it working to a degree.
Download/extract dotCover and add the folder to Path
dotCover.exe --output=AppCoverageReport.html --reportType=HTML dotnet -- test
dotnet SonarScanner begin /k:"my-project" /d:sonar.cs.dotcover.reportsPaths=AppCoverageReport.html
dotnet build
dotnet SonarScanner end
Still having an issue where the test coverage gathering seems to be pulling in too much and as a result is causing an error. Investigating but not related to this question.
I have set up a pipeline for my .NET Core project in Azure Devops using the '.NET Core with SonarCloud' template. When I build the analysis gets run in SonarCloud but with 0% Code coverage (I have tests in my solution).
No matter what configuration tweaks I make to the build I cannot get the code coverage working.
What am I missing?
I came across this article and https://dejanstojanovic.net/aspnet/2019/may/publishing-code-analysis-to-sonarcloud-from-azure-build-pipeline/ implemented the powershell script described in it but still I get no code coverage in SonarCloud
I tried using coverlet as described here but still no joy
https://gunnarpeipman.com/aspnet/azure-devops-code-coverage/
My pipeline consists of the following tasks
.NET Core - Restore
Prepare Analysis Configuration
.NET Core - Build
.NET Core - Test
Run Code Analysis
Publish Quality Gate Result
My test task is configured:
Arguments: --configuration $(BuildConfiguration)
Publish test results and code coverage - checked
In the console of the Run Code Analysis task I get:
10:43:54.7 Fetching code coverage report information from TFS...
10:43:54.702 Attempting to locate a test results (.trx) file...
10:43:54.753 Looking for TRX files in: C:\\TFSBuilds\\TJPYHG04-GHJ01\\_work\\475\\TestResults
10:43:54.755 No test results files found
10:43:54.81 Did not find any binary coverage files in the expected location.
10:43:54.811 Falling back on locating coverage files in the agent temp directory.
10:43:54.812 Searching for coverage files in C:\\TFSBuilds\\TJPYHG04-GHJ01\\_work\\_temp
10:43:54.814 No coverage files found in the agent temp directory.
hope this answer still relevant to you.
Recently I have similar problem as you and I am also using Azure DevOps for my case.
This is how I solved it.
Step 1 - Change directory into your unit testing sub folder (same as the unit-testing .csproj file) and run the following dotnet command.
dotnet add package coverlet.msbuild
Step 2 - Add followings into SonarCloudPrepare Task Additional Properties or append directly into the yml file (if you're using yml instead of classic editor)
extraProperties: |
sonar.exclusions=**/obj/**,**/*.dll
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
The directory is up to your choice to configure.
Or you can also create a file at your repo titled sonar-project.propertiesand store all the relevant SonarCloud properties inside.
Step 3 - Add followings into your dotnet test task
- task: DotNetCoreCLI#2
inputs:
command: 'test'
arguments: '--configuration $(BuildConfiguration) /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx'
testRunTitle: 'dotnet test'
You may notice there's a tickbox for "Publish test results and code coverage" but I still prefer using /p:CollectCoverage=true.
You may also test locally to run the dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover --logger trx command and coverage.opencover.xml will be generated at your unit testing folder.
Refer to reference 2 & 3 for more parameters and their description.
Side Note: If you're doing any Sonar test inclusion properties, note that Sonar.Tests and Sonar.Test.Inclusions their Test(s) are different. lol
References:
Using SonarCloud in Azure pipelines
Analysis Parameters (official docs)
Test Coverage & Execution (official docs)
Hope this helps :)
Not getting code coverage in SonarCloud from an Azure Devops .NET core build
This issue may caused by vstest output path changed recently:
The output path of the vstest coverage file change from
D:\a\1\s\TestResults\... to D:\a\_temp\...
Which broke subsequent scripts in the pipeline (like, codecoverage.exe to convert to xml and later import to sonarqube).
Microsoft suggest that use the rest APIs to check for the test artefacts and re-download them to the build agent.
More investigation on this issue, you can check the thread Azure DevOps (VSTS) extension no longer import coverage and unit tests automatically for the issue tracking.
Fortunately, SonarSourcer team have just release new versions of the SonarQube (v4.6.3) and SonarCloud (v1.6.3) extensions to address the coverage issue and the regression.
Hope this helps.
I had two build steps in VSTS:
To run tests (VSTS cmd task): DOTNET test -xml TEST-results.xml
To publish test results step (VSTS test publish task): format=XUnit and the file name from previous step
But after I upgraded to VS2017 the -XML tag is not working anymore. I changed step one to use this:
test --logger "trx;LogFileName=TEST-results.xml"
but the second step throws an error "Invalid results file. Please make sure the Test Result Format field in the task matches the result format of the file"
Is there another way to run .NetCore tests on VSTS? or am I doing something wrong?
Thanks,
starain-MSFT's answer will work, unless you want/need the xunit tests to be logged using an xunit logger. In that case, you'll need to do two things.
Add https://www.nuget.org/packages/XunitXml.TestLogger/1.0.2-pre-rtm as package ref to your test project, either through 'Manage NuGet Packages' in VS, or by adding the ref in your csproj file manually, i.e.
<PackageReference Include="xunitxml.testlogger" Version="1.0.2-pre-rtm" />
Modify the VSTS dotnet test build step to use this logger:
dotnet test -a:. -l:xunit
The -a:. switch, which specifies the adapter path, is only necessary for CLI tools V15.0, in 15.1 that can be removed (as discussed here). As of today, the VS2017 Hosted Queue is using 15.0, so you'll need the -a:. on VSTS for now.
The -l:xunit uses the friendlyname, which I think isn't so friendly since you have to dig into the source code for the particular logger to find the attribute where it is specified (as seen here for xunit and here for trx)
The docs for the -l switch are spotty to say the least, but in the github for vstest, there is a document which talks about test loggers and links to their repositories and nuget packages, which after you look at the source for the friendlyname, gets you all the way there for whichever logger you need. If you need a custom logger, those are great examples to help understand how to implement.
Finally, the publish step that you used originally should be fine, since the output file is still called TestResults.xml
Change "Test Result Format" to "VSTest of Publish Test" result step/task, it reads the result file correctly.
Use dotnet xunit instead of dotnet test. See Getting Started with .NET Core.
I have code coverage running from PhpUnit, and I can generate a diff file in Git that will show me what's about to be released.
How would I go about mapping one to the other, so that I can say the code we are about to release is covered by our tests?
Ideally, this would be something that could be run automatically as a build step in Jenkins.
Before you run your phpunit tests by your Ant / maven build, run a phpscript. In this php script copy every file specified in the git file which is about to be release along with the matching unit tests to a temporary directory. Of course this requires that you use proper naming conventions and structure for unit tests. Run phpunit on the temporary directory
-application
-app
-tests
-temp
-app
-tests
when you run your phpunit in your build file include a clover report
phpunit --coverage-clover="clover.xml" temp/tests
In Jenkins you can mark the build as unstable if the clover.xml doesn't meet a specified % coverage.
This should work. Let me know how it works out.