Is it possible to migrate/import test cases built on Katalon to Tosca? Or they need to be rewriten on Tosca to work?
Related
Im currently working on a project were i'm importing the output.xml from my test suite to JIRA via XRAY. This create the test (in case they don't exists already) and the execution test (with the results).
My problem is that right now, the test are created without description or documentation. Is there a way were I can set the [Documentation] field from the test cases in robot to the JIRA issue (test) via XRAY? I don't find anything in the XRAY API doc.
Maybe my only option is parse all the [Documentation] field of the tests cases and then import in the test on JIRA without XRAY?
Currently this is not yet possible but it make sense in my perspective.
You can vote and follow this suggestion so the Xray can be aware about your interest on it.
You can implement your own script to go over the Tests and update the Description field (using Jira's REST API) but you would need to know their key in advance; if that's the case, then fine.
Hello im a new junior test software and i've been asked to study about xray and robot framework and how to implement both.
I've made a few Test cases in xray and after i've started to learn about robot framework and till there was all good.
Now i've been trying to implement the results of this tests cases that i made on robot to the tests execution in xray but everytime that i try to import the output.xml from robot to xray instead of "syncronize" this tests the xray creats me new tests care with the result of the robot.
There is anyone around that has done it before that could help me? i've tryed to implement tags in robot or even use the same name of tests (in xray and robot) but it didnt work. Thanks in advance.
I recommend using Jenkins with the XRay - Jira plugin to sync the results of automated tests into xray test items.
You would use a Tag in robot to link a test case to an Xray Test item or if you don't specify an ID, the plugin would create a new Test item and keep it updated based on name
*** Test Cases ***
Add Multiple Records To Timesheet By Multi Add Generator
[Tags] PD-61083
Check this link for details on how to configure the integration
https://docs.getxray.app/display/XRAY/Integration+with+Jenkins
The plugin can keep track of the execution in a specific Test Execution item or create one per run but should keep referring to the same Test item.
When you upload the RF results, Xray will auto-provision Test issues, one per each Robot Framework's Test Case. This is the typical behavior, which you may override in case you want to report results against an existing Test issue. In that case, you would have a Test in Jira and then you would add a tag to the RF Test Case entry, with the issue key of the existing Test issue.
However, taking advantage of auto-provisioning of Tests is easier and is probably the most used case. Xray, will only provision/create Test issues if they don't exist; for this, Xray tries to figure out if a generic Test exists, having the same definition (i.e. name of RF Test suites plus the Test Case name). If it does find it, then it will just report results (i.e. create a Test Run) against the existing Test issue.
If Test issues are always being created each time you submit the test results, that's an unexpected behavior and needs to be analyzed in more detail.
There is another entity to have in mind: Test Execution.
Your results will be part of a Test Execution. Every time that you submit test results, a Test Execution... unless, you specify otherwise. In the REST API request (or in the Jenkins plugin) you may specify an existing Test Execution by its issue key. If you do so, then the results will be overwritten on that Test Execution and no new Test Execution issue will be created. Think on it as reusing a given Test Execution.
How the integration works and the available capabilities are described in some detail within the documentation.
As an additional reference, let me also share this RF tutorial as it may be useful to you.
One of our customers uses Katalon Studio and would like to see the reporting results in Xray and Jira again after the execution of the test cases.
Do I have to create a Json file at the end of the test run? That I have to import into Xray?
Is the Katalon export plugin sufficient here?
Or does the Xray plugin offer me exactly this way?
as of today, AFAIK, Katalon doesn't have yet a specific integration with Xray. You may ask the Katalon team to provide it by submiting them a feature request.
Meanwhile, you can follow these intructions to have visibility of the results in Xray using a JUnit XML report which will autoprovision Test issues.
https://confluence.xpand-it.com/display/XRAY/Testing+APIs+using+Katalon+Studio
If you would like to enforce a specific test in Katalon to an existing Test issue, you would have to use a different format such as Xray JSON; however, that would require you to build it using some logic based on the information in Katalon.
My company uses Microsoft Test Management server to host its tests and results. For the manual tests this works fine a QA engineer runs the test and marks its status, I have been tasked with writing some automated tests and I need them to submit results to the server. I know there is a code api, but I want to do this from a non .Net test environment (I am going to use AutoIt) so I would like to submit results from an HTTP api, how can I do this? Where can I find some good examples? Or is there a better way, we are a very MS TFS shop so whatever I do needs to fit into that environment.
Thank you!
Microsoft has a UI automation framework and API called CodedUI that is fully integrated with Microsoft Test Manager.
You can:
Generate test automations from Action recordings of manual tests
Generate test automations in Visual Studio
Code test automations from scratch in VS
These automations can then be associated with a Test Case in MTM and pushed to test environments manually or via an API. I usually have new code built, deployed, and tested automatically using these techniques.
You can also plug other UI frameworks into this model.
If you want to submit Test Results from a non-windows environment then you should use the Cross-Platform API. As part of Team Explorer Everywhere you get both a command line and a Java based Object Model for manipulating TFS.
http://www.microsoft.com/en-us/download/details.aspx?id=47727
I should note that the API for constructing test result submissions is quite complex as your tests in MTM are part of a hierarchy of Suits and Plans, and each Test Case can exist in more than one location. You will need to create a Test Run and populate it with the appropriate data.
We have a complete Lab Management environment running Coded UI tests in nightly builds. What we are trying to achieve is to run our integration tests (regular TestMethod() with SQL connections) just before all the Coded UI tests to verify that our db scripts are executed correctly and that there are no new changes causing any problems.
So far I have found a way to execute tests remotely through .testrunconfig. The problem we have with that approach is that it's not possible to choose a testcontroller connected to a team project so I guess that would be only useful for running tests on physical machines outside of Lab Management?
One option seem to be to create a Test Case for each integration test and that should run it together with the UI tests but it feels like it will be to much maintenance managing hundreds of test cases just to run the integration tests. Also it would be better to completely separate the test runs for the different kinds of tests.
Is there any easy way to achieve this that I have totally missed? Or do I have to modify the lab build template to deploy and run the tests?
I guess that would be only useful for running tests on physical machines outside of Lab Management?
If you run your tests remotely through .testrunconfig you have to connect the Test Agent to another Test Controller which is NOT connected to the team project.
Unfortunately it is impossible for the environment which are running under the Lab Management, to my knowledge.
What about this approach:
Create an Ordered Test containing all you integration tests.
Create a new Test Case "Integration Tests" and automate it by the ordered test
So you do not have to maintain hundreds of Test Cases.
You could also create several Ordered Tests if you want to group the integration tests and then
create a "main" Ordered Test containing them.
This way it will be easier to analyze test results especially if you have a lot of tests.
Let the integration tests run as a part of your existing nightly build.
Create a new Build Definition which does not start a build but uses the last successful nightly build and let your CodedUI tests run using Lab Build Template.
This way you will have different test runs for the different kinds of tests.
The only drawback is that you have to "synchronize" these two builds...
You could just schedule the second build later so you could be sure the fist build is done.
It's not really perfect, I know... but this way you could easily achieve your goal.
I am not sure if there is an alternative solution, but on the project I am currently working on we have both our Unit and Integration Test Assemblies set under the Process options (Process>Basic>AutomatedTest>TestAssembly) in our Nightly Build. This was achieved through altering the Default Build Process Template (not the Lab Default) a bit, as you suggested (I thought this was standard, but it's been a while).