Showing the result of QTestlib with Jenkins xUnit plug-in - qt

I am trying to use Jenkins xUnit plug-in for my Qt unit test project, but I cannot make it work...
Here is What I've done so far:
First of all, I build my unit test project with qmakebuilder plug-in (providing the .pro to the qmakebuilder plug-in), then I add an Execute Shell build section
where I first rm -f the testResult xml file, following to that, I run the test binary with -xunitxml flag to have it generate the xml file for me And I name the xml file testResult.xml, in other word:
rm -f /home/guest/QT/unitTest/testResult.xml
cd /home/guest/QT/unitTest
./tst_unittesttest -xunitxml > testResult.xml
Finally in Post Build Action, I select Publish xUnit test result and specify the pattern as *.xml.
Building the job will fail though, Here is the output that I get when building the job with Jenkins:
[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing QTestlib-Version N/A
[xUnit] [INFO] - [QTestlib-Version N/A] - 1 test report file(s) were found with
the pattern '*.xml' relative to '/home/guest/QT/unitTest' for the testing framework
'QTestlib-Version N/A'.
[xUnit] [ERROR] - The converted file for the result file '/home/guest/QT/unitTest
/testResult.xml' (during conversion process for the metric 'QTestlib') is not
valid.
The report file has been skipped.
[xUnit] [ERROR] - The plugin hasn't been performed correctly: hudson.util.IOException2:
Failed to read /home/guest/QT/unitTest/generatedJUnitFiles/QTestlib/TEST--735044756.xml
Build step 'Publish xUnit test result report' changed build result to FAILURE
Build step 'Publish xUnit test result report' marked build as failure
Finished: FAILURE
As you can see there is problem converting my test result xml file to what xUnit plug-in can read. My testResult.xml file looks like this:
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="2" failures="0" tests="4" name="UnitTestTest">
<properties>
<property value="4.7.4" name="QTestVersion"/>
<property value="4.7.4" name="QtVersion"/>
</properties>
<testcase result="pass" name="initTestCase">
<!-- message="called before everything else" type="qdebug" -->
</testcase>
<testcase result="pass" name="myFirstTest"/>
<testcase result="pass" name="mySecondTest"/>
<testcase result="pass" name="cleanupTestCase">
<!-- message="called after myFirstTest and mySecondTest" type="qdebug" -->
</testcase>
<system-err>
<![CDATA[called before everything else]]>
<![CDATA[called after myFirstTest and mySecondTest]]>
</system-err>
</testsuite>
but TEST--735044756.xml is an one line xml file: <?xml version="1.0" encoding="UTF-8"?>
Anyone has a clue where am I doing what wrong? There should be a problem with generating the output xml file.
On a different note, I have added the log thingi in the system log Jenkins menu, it doesn't seem to be the reason for this problem though... at least IMO

Executing the test binary with -xml flag will just work fine. Weirdly using -xunitxml creates corrupt xml file which its <testcase> doesn't have time attribute. Running the test binary only with -xml results in building the job successfully.

Related

How to make azure devops build fail when R linting issues occur

I am using lintr library in R to find linting issues in the code. I put them into an xml format like this:
<lintsuites>
<lintissue filename="/home/.../blah.R" line_number="36" column_number="1" type="style" message="Trailing blank lines are superfluous."/>
<lintissue filename="/home/.../blahblah.R" line_number="1" column_number="8" type="style" message="Only use double-quotes."/>
</lintsuites>
Now, I would like to fail the Azure devops build when issues like this occur.
I was able to get my tests in a JUnit format like this:
<testsuite name="MB Unit Tests" timestamp="2020-01-22 22:34:07" hostname="0000" tests="29" skipped="0" failures="0" errors="0" time="0.05">
<testcase time="0.01" classname="1_Unit_Tests" name="1_calculates_correctly"/>
<testcase time="0.01" classname="1_Unit_Tests" name="2_absorbed_correctly"/>
...
</testsuite>
And when i do this step in the azure pipeline, my build fails if any tests in the test suite fail:
- task: PublishTestResults#2
displayName: 'Publish Test Results'
inputs:
testResultsFiles: '**/*.xml'
searchFolder: '$(System.DefaultWorkingDirectory)/fe'
mergeTestResults: true
failTaskOnFailedTests: true
I would like something similar for failing the build when there are linting issues. I would also like the users to see what those linting issues are in the build output.
Thanks
This is not possible to achieve a similar result for lintr xml with plishTestResults#2.
The workaround you can try is to use a powershell task to check for the content of your lintr xml file. If the content isnot empty, then fail the pipeline in the powershell task.
Below powershell task will check the content of lintr.xml(<car></car>) and will echo the content to the task logs and exit 1 to fail the task if the content is null.
- powershell: |
[xml]$XmlDocument = Get-Content -Path "$(system.defaultworkingdirectory)/lintr.xml"
if($XmlDocument.OuterXml){
echo $XmlDocument.OuterXml
}else{exit 1}
displayName: lintr result.
You can aslo use below statement in a powershell task to upload lintr xml file to the build summary page where you can download
echo "##vso[task.uploadsummary]$(system.defaultworkingdirectory)/lintr.xml"
You can check here for more logging commands.
Update:
A workaround to show the lintr results in a nice way is to create a custom extension to display html results in azure devops pipeline.
You can try creating a custom extension, and produce a html lint results. Please refer to the answer to this thread an example custom extension to display html
Other developers already submit requests to Microsoft for implementing this feature. Please vote it up here or create a new one.

Pact File upload to pact broker failing

Issue of "Failed to read pacts: NullPointerException" while uploading pact file into the pact broker. Below is my stack trace and POM file details.
Please help.
Below is my stack trace
[INFO] loading pacts from target/pacts
[INFO] found pact file: PRODUCT-CART.json
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.476 s
[INFO] Finished at: 2017-10-03T15:09:20+08:00
[INFO] Final Memory: 11M/243M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.warmuuh:pactbroker-maven-plugin:0.0.9:upload-pacts (default-cli) on project Auth_Api_Consumer: Failed to read pacts: NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.warmuuh:pactbroker-maven-plugin:0.0.9:upload-pacts (default-cli) on project Auth_Api_Consumer: Failed to read pacts
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at
Below is my POM.xml
<plugin>
<groupId>com.github.warmuuh</groupId>
<artifactId>pactbroker-maven-plugin</artifactId>
<version>0.0.9</version>
<executions>
<!--CONSUMER Upload Pact File Starts -->
<execution>
<id>upload-pacts</id>
<phase>test</phase>
<goals><goal>upload-pacts</goal></goals>
<configuration>
<brokerUrl>http://localhost:8080/</brokerUrl>
<!-- <brokerUrl>git#scm.hue.workslan:tools/hue-contract-testing.git</brokerUrl> -->
<pacts>${project.build.directory}/pacts</pacts>
</configuration>
</execution>
<!--CONSUMER Upload Pact File Ends -->
</executions>
</plugin>
This issue comes if one of the below occurs :
1) There is naming conflict in the Consumer/Provider Names in the json files you are uploading.
Solution : Make sure the names are unique, the matching rules are not only related to substring match but also catch "similar name".Please check the Broker matching code for more details. There is no documentation for this yet.
Example : Cart_service and order_service are NOT allowed, Car-order,order-processing are NOT allowed since they have common words.
2) Running the mvn upload-pacts will return NullPointerException if there is any issue. To see the real error make sure you run mvn test or mvn install instead. It will show you the exact server response status code.
See this documentation on publishing pacts for services with similar names:
409 when publishing a pact
When a pact is published normally (via a PUT to /pacts/provider/PROVIDER/consumer/CONSUMER/version/CONSUMER_APP_VERSION) the consumer, provider and consumer version resources are automatically created.
To prevent a pacticipant (consumer or provider) being created multiple times with slightly different name variants (eg. FooBar/foo-bar/foo bar/Foo Bar Service), if a new pacticipant name is deemed similar enough to an existing name, a 409 will be returned. The response body will contain instructions indicating that the pacticipant name should be corrected if it was intended to be an existing one, or that the pacticipant should be created manually if it was intended to be a new one.
Some Pact Broker clients unfortunately do not show the full error text when this happens. The full text is as follows:
This is the first time a pact has been published for "%{new_name}".
The name "%{new_name}" is very similar to the following existing consumers/providers:
%{existing_names}
If you meant to specify one of the above names, please correct the pact configuration, and re-publish the pact.
If the pact is intended to be for a new consumer or provider, please manually create "%{new_name}" using the following command, and then re-publish the pact:
$ curl -v -XPOST -H "Content-Type: application/json" -d '{"name": "%{new_name}"}' http://broker/pacticipants
If the pact broker requires authentication, include '-u <username>:<password>' in the command.
https://github.com/pact-foundation/pact_broker/wiki/Troubleshooting#409-when-publishing-a-pact

"Run Functional Tests" task and Specflow - App.config being ignored?

I've created a Release definition in TFS 2015 update 2 as seen below:
I want to use this as part of my CI process. The build and unit tests are executed on a dedicated BizTalk build server for each check-in. Executed nightly, this (currently problematic) release should deploy to an "DevTest" BizTalk server and execute integration tests.
All is good up to the last "Run Functional Tests" task. So, the BizTalk applications are removed, then deployed to the target server in the correct order. The "IntegrationTests" assembly contains a few Specflow tests which make use of the Transmock framework (http://transmock.codeplex.com/). Steps for these tests are contained within an assembly called "Common.Commponents". This is referenced from a test assembly called "Finance.IntegrationTests.dll". The tests pass when run on my Dev VM. However, when executed by the "Run Functional Tests" task, I get the following error:
Test method Finance.IntegrationTests.ASC_BacsFeature.BacsFileMoved threw exception: NUnit.Framework.InconclusiveException: No matching step definition found for one or more steps.using System;using TechTalk.SpecFlow;namespace MyNamespace{ [Binding] public class StepDefinitions { [Given(#"an output path specified by sso app '(.)' and setting '(.)'")]public void GivenAnOutputPathSpecifiedBySsoAppAndSetting(string p0, string p1){ ScenarioContext.Current.Pending();} [Given(#"the inpu
I have set the App.config of my test project to use MsTest as seen below:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<configSections>
<section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow"/>
</configSections>
<specFlow>
<!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
<unitTestProvider name="MsTest"/>
<stepAssemblies>
<stepAssembly assembly="LCC.Integration.Common.Components" />
</stepAssemblies>
</specFlow>
<startup><supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.1"/></startup></configuration>
It seems that when the tests are executed remotely, NUnit is used rather than MSTest.
The "Common.Components.dll" is successfully deployed to the GAC earlier in the release via GacUtil /I, so I'm not sure why it can't be found?
I've tried running the tests directly on the remote server. I RDP'd on, then opened a command window and ran "vstest.console Finance.IntegrationTests.dll". I then received warnings that the test input files could not be found, followed by errors because "Could not load file or assembly 'nunit.framework'".
I think the problem is, App.config of the Integration Test assembly is not being used when the tests are executed remotely. This explains why it tries to use NUnit rather than MSTest and also why it is unable to bind to the common steps - because their location is given in the node.
Does anyone know how I can make the App.config be used when testing remotely?
There were a couple of issues. I'd forgotten that the app.config would be compiled into finance.integrationtests.dll.config. References to dlls in the config needed to have the full strong name rather than just the name.

PHPUnit + Sonarqube exported coverage do not match with actual xml result

sonarqube: 5.1.2
sonar-runner: 2.2.1
php plugin: 2.6
PHPUnit 4.2.6
We're running phpunit over our application but I'm not able to get the correct coverage % on sonar as expected.
In my phpunit.xml we have filters defining folders that we only want to cover.
<whitelist addUncoveredFilesFromWhitelist="false">
<directory>./site-library-php/src/main/php/BabelCentral/Model/Content</directory>
</whitelist>
In my sonar properties
sonar.modules=php-module
php-module.sonar.phpUnit.coverage.analyzeOnly=true
php-module.sonar.phpUnit.analyzeOnly=true
php-module.sonar.php.tests.reportPath=site-main-php/src/test/target/reports/phpunit.xml
php-module.sonar.php.coverage.reportPath=site-main-php/src/test/target/reports/phpunit.coverage.xml
Viewing the log on jenkins, tests seem to run fine and ends with:
Tests: 1479, Assertions: 4165, Failures: 58, Incomplete: 14, Skipped: 55.
Generating code coverage report in Clover XML format ... done
Generating code coverage report in HTML format ... done
// further down
11:47:51.973 INFO - Analyzing PHPUnit test report: site-main-php/src/test/target/reports/phpunit.xml with org.sonar.plugins.php.phpunit.PhpUnitResultParser#35e7e715
11:47:52.604 INFO - Analyzing PHPUnit unit test coverage report: site-main-php/src/test/target/reports/phpunit.coverage.xml with PHPUnit Unit Test Coverage Result Parser
All seems to be working well. However, sonar would report a different result, covering the entire source folder. These are the number shown on the dashboard.
Unit Tests Coverage 1.8%
line coverage 1.8%
Unit test success 95.9%
Is there anyway to have sonar respect phpunit's configured filter?
Note:
I'm able to achieve the desired numbers on coverage if I explicitly set
php-module.sonar.sources
to the directories/files that I want. It's just that, comma-separated config is hard to manage than phpunit's xml config.
Defining the set of source files which should be analysed by SonarQube is crucial: it's on this set of files that coding rules are applied and that metrics are computed.
sonar.sources is therefore a mandatory property and is the main way to configure the set of source files.
Other properties may be used to refine the set of source files:
You may configure sonar.tests: test files are automatically excluded from sonar.sources.
You may have to exclude some other PHP files like dependencies or generated code with sonar.exclusions.
You should also be aware that several plugins may contribute to the global coverage of the project: if you installed the JavaScript plugin and if sonar.sources contains JavaScript files, they will also be taken into account when calculating the coverage metrics.
If you're only interested in adjusting coverage, you can exclude files from coverage metrics with sonar.coverage.exclusions.

reportng report overwrites previous one if run multiple test suites

for example, if I run the testng.xml and it contains multiple test suites
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Master Suite">
<suite-files>
<suite-file path="test_1.xml"></suite-file>
<suite-file path="test_2.xml"></suite-file>
</suite-files>
</suite>
after, I checked reportng report, it did not display both test suites report but only the last one. How to avoid overwriting previous test report?
please help
dont add listeners in suite files add those in ur testng.xml
this will solve ur issue.

Resources