One of our customers uses Katalon Studio and would like to see the reporting results in Xray and Jira again after the execution of the test cases.
Do I have to create a Json file at the end of the test run? That I have to import into Xray?
Is the Katalon export plugin sufficient here?
Or does the Xray plugin offer me exactly this way?
as of today, AFAIK, Katalon doesn't have yet a specific integration with Xray. You may ask the Katalon team to provide it by submiting them a feature request.
Meanwhile, you can follow these intructions to have visibility of the results in Xray using a JUnit XML report which will autoprovision Test issues.
https://confluence.xpand-it.com/display/XRAY/Testing+APIs+using+Katalon+Studio
If you would like to enforce a specific test in Katalon to an existing Test issue, you would have to use a different format such as Xray JSON; however, that would require you to build it using some logic based on the information in Katalon.
Related
I have a postman collection which in turn is integrated to Jenkins via newman.
I need to integrate my Jenkin results with Jira via X-ray plugin.
I tried using newman junitxray reporter but this report consider each request as test case.
In my collection i always need to run some series of request before running the actual request which contain pm.test.
But junitxray report is considering those series of request as also test cases.
I want only a specific request to be taken as test-case.
Can someone please help me on this.
You can use different reporters with newman; depending on that, what will be on the JUnit XML report will be different. Some time ago I've prepared a tutorial showing the differences.
If you're using Xray on Jira cloud, please check this tutorial and related code.
If you're using Xray on Jira server/datacenter, please check this tutorial and related code instead.
Im currently working on a project were i'm importing the output.xml from my test suite to JIRA via XRAY. This create the test (in case they don't exists already) and the execution test (with the results).
My problem is that right now, the test are created without description or documentation. Is there a way were I can set the [Documentation] field from the test cases in robot to the JIRA issue (test) via XRAY? I don't find anything in the XRAY API doc.
Maybe my only option is parse all the [Documentation] field of the tests cases and then import in the test on JIRA without XRAY?
Currently this is not yet possible but it make sense in my perspective.
You can vote and follow this suggestion so the Xray can be aware about your interest on it.
You can implement your own script to go over the Tests and update the Description field (using Jira's REST API) but you would need to know their key in advance; if that's the case, then fine.
Hello im a new junior test software and i've been asked to study about xray and robot framework and how to implement both.
I've made a few Test cases in xray and after i've started to learn about robot framework and till there was all good.
Now i've been trying to implement the results of this tests cases that i made on robot to the tests execution in xray but everytime that i try to import the output.xml from robot to xray instead of "syncronize" this tests the xray creats me new tests care with the result of the robot.
There is anyone around that has done it before that could help me? i've tryed to implement tags in robot or even use the same name of tests (in xray and robot) but it didnt work. Thanks in advance.
I recommend using Jenkins with the XRay - Jira plugin to sync the results of automated tests into xray test items.
You would use a Tag in robot to link a test case to an Xray Test item or if you don't specify an ID, the plugin would create a new Test item and keep it updated based on name
*** Test Cases ***
Add Multiple Records To Timesheet By Multi Add Generator
[Tags] PD-61083
Check this link for details on how to configure the integration
https://docs.getxray.app/display/XRAY/Integration+with+Jenkins
The plugin can keep track of the execution in a specific Test Execution item or create one per run but should keep referring to the same Test item.
When you upload the RF results, Xray will auto-provision Test issues, one per each Robot Framework's Test Case. This is the typical behavior, which you may override in case you want to report results against an existing Test issue. In that case, you would have a Test in Jira and then you would add a tag to the RF Test Case entry, with the issue key of the existing Test issue.
However, taking advantage of auto-provisioning of Tests is easier and is probably the most used case. Xray, will only provision/create Test issues if they don't exist; for this, Xray tries to figure out if a generic Test exists, having the same definition (i.e. name of RF Test suites plus the Test Case name). If it does find it, then it will just report results (i.e. create a Test Run) against the existing Test issue.
If Test issues are always being created each time you submit the test results, that's an unexpected behavior and needs to be analyzed in more detail.
There is another entity to have in mind: Test Execution.
Your results will be part of a Test Execution. Every time that you submit test results, a Test Execution... unless, you specify otherwise. In the REST API request (or in the Jenkins plugin) you may specify an existing Test Execution by its issue key. If you do so, then the results will be overwritten on that Test Execution and no new Test Execution issue will be created. Think on it as reusing a given Test Execution.
How the integration works and the available capabilities are described in some detail within the documentation.
As an additional reference, let me also share this RF tutorial as it may be useful to you.
I want to create a custom report. Response format for sonarqube web service API /api/issues/search is JSON or XML. How can I use that response to create a html or CSV file using "unix shell without using command line tools" so that I can use it as a Report. Or is there any other better way to achieve this?
you can generate a html file if you run an analysis in the preview mode http://docs.sonarqube.org/pages/viewpage.action?pageId=6947686
It looks as if the SonarQube team has been working hard to not allow people to do this. They appear to want people to purchase an Enterprise Subscription in order to export reports.
An old version of sonar-runner (now called sonar-scanner) had an option to allow local report output. But that feature is "no more supported".
ERROR: The preview mode, along with the 'sonar.analysis.mode' parameter, is no more supported. You should stop using this parameter.
Looks like version 2.4 of Sonar Runner does what you want. If you can find it. Of course they only have 2.5RC1 available on the site now.
Using the following command should work on version 2.4:
sonar-runner -Dsonar.analysis.mode=preview -Dsonar.issuesReport.html.enable=true
There at least two open-source projects that query the SQ API to generate reports in various formats.
https://github.com/cnescatlab/sonar-cnes-report/tree/dev (Java)
https://github.com/soprasteria/sonar-report (JavaScript/Node)
At the time of writing both are active.
My company uses Microsoft Test Management server to host its tests and results. For the manual tests this works fine a QA engineer runs the test and marks its status, I have been tasked with writing some automated tests and I need them to submit results to the server. I know there is a code api, but I want to do this from a non .Net test environment (I am going to use AutoIt) so I would like to submit results from an HTTP api, how can I do this? Where can I find some good examples? Or is there a better way, we are a very MS TFS shop so whatever I do needs to fit into that environment.
Thank you!
Microsoft has a UI automation framework and API called CodedUI that is fully integrated with Microsoft Test Manager.
You can:
Generate test automations from Action recordings of manual tests
Generate test automations in Visual Studio
Code test automations from scratch in VS
These automations can then be associated with a Test Case in MTM and pushed to test environments manually or via an API. I usually have new code built, deployed, and tested automatically using these techniques.
You can also plug other UI frameworks into this model.
If you want to submit Test Results from a non-windows environment then you should use the Cross-Platform API. As part of Team Explorer Everywhere you get both a command line and a Java based Object Model for manipulating TFS.
http://www.microsoft.com/en-us/download/details.aspx?id=47727
I should note that the API for constructing test result submissions is quite complex as your tests in MTM are part of a hierarchy of Suits and Plans, and each Test Case can exist in more than one location. You will need to create a Test Run and populate it with the appropriate data.