We are using NightwatchJS automation tool for testing. We have 2 testcases in Jira-xray and 1 testcase in automation. When we run auomation, JUnit xml test results contains only 1 testcase. If JUnit xml test results are uploaded, will it mark 2 testcsaes mapped in Jira as pass/fail?
It should only map to one test case, following the rules described in the integration specifics page.
Xray will try to find an existing "Generic" Test issue having a Generic Definition field composed by the classname and name attributes of the element; these are used as a unique identifier for the test.
Related
I am attempting to write a tool that will automate the generation of a visual studio test playlist based on failed tests from the spec flow report, we recently increased our testThreadCount to 4 and when using the LivingDocumentation plugin to generate the TestExecution.json file it is only generating a result for 1 in 4 tests and I think this is due to the threadCount so 4 tests are being seen as a single execution.
My aim is to generate a fully qualified test name for each of the failed tests using the TestExecution file but this will not work if I am only generating 25% of the results. Could I ask if anyone has an idea of a workaround for this?
<Execution stopAfterFailures="0" testThreadCount="4" testSchedulingMode="Sequential" retryFor="Failing" retryCount="0" />
This is our current execution settings in the .srprofile
We made this possible with the latest version of SpecFlow and the SpecFlow+ LivingDoc Plugin.
You can configure the filename for the TestExecution.json via specflow.json.
Here is an example:
{
"livingDocGenerator": {
"enabled": true,
"filePath": "TestExecution_{ProcessId}_{ThreadId}.json"
}
}
ProcessId and ThreadId will be replaced with values and you get for every thread a separate TestExecution.json.
You can then give the livingdoc CLI tool or the Azure DevOps task a list of TestExecution.jsons.
Example:
livingdoc test-assembly BookShop.AcceptanceTests.dll -t TestExecution*.json
This generates you one LivingDoc with all the test execution results combines.
Documentation links:
https://docs.specflow.org/projects/specflow-livingdoc/en/latest/LivingDocGenerator/Setup-the-LivingDocPlugin.html
https://docs.specflow.org/projects/specflow-livingdoc/en/latest/Guides/Merging-Multiple-test-results.html
Through Katalon studio UI tool we are able to perform parallel execution of test suites in test suite collection for different browsers.
Problem: Same approach is not working when we tried through cmd. Below is the command for the same:
katalon -noSplash -runMode=console -projectPath="<projectPath>" -retry=0 -testSuitePath="<testSuitePath>" -executionProfile="default" -browserType="Chrome,IE"
Note: Works fine for single browser as parameter
Please let us know if above command is correct for multiple browser execution
Expected :
Single report folder containing parallel execution results of both the browsers
You can do that, by using Test Suite Collections.
You put your TC1 (test case) in TS1 (test suite). My test case is called "proba" in this example. Then you create a TSC1 (test suite collection) and you add the same test suite twice to the collection. See screenshot. And change "Run with" parameter to Chrome and IE, respectively.
If you now create a command line argument, you will get something like
katalon -noSplash -runMode=console -consoleLog -projectPath="C:\\Katalon Studio\PROJECT NAME\PROJECT NAME.prj" -retry=0 -testSuiteCollectionPath="Test Suites/TEST SUITE COLLECTION 1"
I'm currently developing a couple of test cases with robotframework to compare some excel value with value inside our database.
I have to do it inside a specific test case as it is deploy on zephyr.
I am checking each value inside this test case by calling a homemade Keyword that does :
Run Keyword Should Contain ${valeurExcel1} ${valeurBDD1}
Run Keyword Should Contain ${valeurExcel2} ${valeurBDD2}
etc...
I need every single one of those "Should Contain" to be display in a separated row in the report.html
It currently only appear as one row as it is one test case.
Is there anyway to specify to robot framework that i want him to consider every "Should Contain" as a unique test case and to display it in a row on the report.html ?
(Maybe by tagging ?)
No you can't. If you want a row for each "should contain" then each of those call should be made in its own test case.
But I think the problem lies in your "I have to do it inside a specific test case as it is deploy on zephyr". Whatever you need to do before/after a test case, can be done in a "suite setup" (and "suite teardown"). So you could have this kind of architecture:
*** Settings ***
Suite Setup deploy SUT / Zephyr
Suite Teardown shutdown SUT / Zephyr
*** Test Cases ***
tc1
Run Keyword Should Contain ${valeurExcel1} ${valeurBDD1}
tc2
Run Keyword Should Contain ${valeurExcel2} ${valeurBDD2}
I can't find a way to having something like setUp and tearDown when running a Plone robot test.
What I need is to create a whole site structure before each test. The only way I found is to create my own new robot keyword that create the structure from the browser itself, but I don't want to test the folder creation. This will slow down my tests for nothing.
I found something promising in the robotsuite library: https://github.com/collective/robotsuite/blob/b52000c65ea96b2be697bf899829c397985fc5aa/src/robotsuite/init.py#L217
However I can't make it works: I can define an empty setUp function, with no Plone context.
How can I do this?
At first, you can always define your test site structure in the setup methods of your plone.app.testing based test layer. (See docs for plone.app.testing for details.)
In addition to that, there are at least a few ways to build test site structure for plone.app.robotframework based Selenium tests:
The first is passing your test setUp function as setUp-keyword argument for RobotTestSuite-class:
import unittest
import robotsuite
from plone.testing import layered
from my.package.testing import MY_PACKAGE_ROBOT_TESTING
from zope.component.hooks import getSite
from transaction import commit
def mySetUp(testCase=None):
site = getSite()
# do stuff
commit()
def test_suite():
suite = unittest.TestSuite()
suite.addTests([
layered(
robotsuite.RobotTestSuite('test_something.robot',
setUp=mySetUp),
layer=MY_PACKAGE_ROBOT_TESTING
),
])
return suite
There should be no need for explicit tearDown, because plone.app.testing's functional test teardown already reverts all changes made to the test sandbox.
This feature is based on RobotTestSuite built to mimic DocTestSuite from doctest-module. The setUp-method takes a single argument, the test case object, but because the test layer is lost somewhere in the test framework, getSite is the only sane way to reach the portal object.
Finally, it's good to know that this approach works only as long as the Robot Framework test is executed within the same process where Plone test layer is built (which is the case when executing robosuite-wrapped test suites with e.g. zope.testrunner).
The second way would be to split the setUp function into a custom local python keyword library. For example, in my/package/testing.py you could have:
from plone import api
from transaction import commit
class MyPackageKeywords(object):
def create_stuff(self, stuff_id):
portal = api.portal.get()
obj = api.content.create(
type='Document',
title='My Stuff',
container=portal
)
commit()
return obj.absolute_url()
And you should be able to use it like:
*** Settings ***
...
Library my.package.testing.MyPackageKeywords
*** Test Cases ***
...
${URL} = Create stuff stuff_id=my-stuff
Go to ${URL}
Element should contain css=h1.documentFirstHeading My Stuff
This has the same limitation as the first approach.
The third possible way is to create the content in the actual Robot Framework test suites by using the content related remote keywords provided in the recent plone.app.robotframework releases.
In short, this is done by including REMOTE_LIBRARY_BUNDLE_FIXTURE in your test layer and using its Create content and Fire transition keywords for creating the required mock content.
import unittest
import robotsuite
from plone.app.robotframework.testing import REMOTE_LIBRARY_BUNDLE_FIXTURE
from plone.app.testing import PLONE_FIXTURE
from plone.testing import layered
from plone.testing import z2
PLONE_ACCEPTANCE_TESTING = z2.FunctionalTesting(
bases=(REMOTE_LIBRARY_BUNDLE_FIXTURE,
PLONE_FIXTURE,
z2.ZSERVER_FIXTURE),
name="Acceptance")
def test_suite():
suite = unittest.TestSuite()
suite.addTests([
layered(
robotsuite.RobotTestSuite('test_something.robot'),
layer=PLONE_ACCEPTANCE_TESTING
),
])
return suite
*** Settings ***
Resource plone/app/robotframework/keywords.robot
Resource Selenium2Screenshots/keywords.robot
Library Remote ${PLONE_URL}/RobotRemote
Test Setup Open test browser
Test Teardown Close all browsers
*** Test Cases ***
Scenario: View folder contents
Given a contributor
and a folder
and a document inside the folder
when at the folder listing
then can see the document title
*** Variables ***
${FOLDER_ID} a-folder
${DOCUMENT_ID} a-document
*** Keywords ***
A contributor
Enable autologin as Contributor
A folder
Create content type=Folder
... id=${FOLDER_ID}
... title=A folder
... description=This is the folder
A document inside the folder
Create content type=Document
... container=/${PLONE_SITE_ID}/${FOLDER_ID}
... id=${DOCUMENT_ID}
... title=A document
... description=This is the document
... text=<p><strong>Hello world!</strong></p>
At the folder listing
Go to ${PLONE_URL}/${FOLDER_ID}/folder_contents
Element should be visible name=folderContentsForm
Can see the document title
Page should contain A document
Capture and crop page screenshot folder-contents.png css=#content
Plese note, that Create content -keyword may also require use of Enable autologin as -keyword to authenticate the content creation calls. Also note, that Capture and crop page screenshot is nowadays in a separate package robotframework-selenium2screenshots.
Content creation keywords are not yet perfect and issues may be reported at GitHub. You can also implement your own remote python keywords. They are just methods in simple python classes, which is wrapped into REMOTE_LIBRARY_BUNDLE_FIXTURE.
Finally, see also the good known versions for plone.app.robotframework and friends.
I'm using Robot Framework (RF) to search a keyword relating to my website to know/find the ranking position of my web site in Google search page (ranking : 1st, 2nd,... and which page? 1st page, 2nd page ?).
Here are my code:
*** Test Cases ***
Rank
Open Browser http://www.google.com.vn gc
Input Text name=q atdd framework
Submit Form xpath=//form[#action='/search']
Wait Until Element Is Visible xpath=//div[#class='srg']/li[#class='g']
${xpa-count}= Get Matching Xpath Count xpath=//div[#class='srg']/li[#class='g']
${lis}= Create List
: FOR ${i} IN RANGE 1 ${xpa-count} + 1 # XPath indexes are 1-based, but Python is 0-based
\ ${li}= Get Text xpath=//div[#class='srg']/li[#class='g'][${i}]
\ Append To List ${lis} ${li}
\ Log ${li}
Log List ${lis}
List Should Contain Value ${lis} robotframework.org/
${rank}= Get Index From List ${lis} robotframework.org/
${rank}= Evaluate unicode(${rank} + 1) # increment to get back to 1-based index
Log ${rank}
Log robotframework.org has rank ${rank}
[Teardown] Close All Browsers
But on the ranking position on RF's log doesn't match to Google screen:
Documentation:
Logs the length and contents of the `list` using given `level`.
Start / End / Elapsed: 20140509 10:25:51.025 / 20140509 10:25:51.026 / 00:00:00.001
10:25:51.026 INFO List length is 10 and it contains following items:
0: atdd-with-robot-framework - Google Code
code.google.com/p/atdd-with-robot-framework/ - Dịch trang này
This project is a demonstration on how to use Acceptance Test Driven Development (ATDD, a.k.a. Specification by Example) process with Robot Framework.
1: ATDDWithRobotFrameworkArticle - Google Code
code.google.com/.../robotframework/.../ATDDWithRobot...
Dịch trang này
21-11-2010 - Acceptance Test-Driven Development with Robot Framework article by Craig Larman ... See also ATDD With Robot Framework demo project.
2: tdd - ATDD versus BDD and the proper use of a framework ...
stackoverflow.com/.../atdd-versus-bdd-and-the-proper-us...
Dịch trang này
29-07-2010 - The Quick Answer. One very important point to bring up is that there are two flavors of Behavior Driven Development. The two flavors are xBehave ...
3: ATDD Using Robot Framework - SlideShare
www.slideshare.net/.../atdd-using-robot-framework
Dịch trang này
23-11-2011 - A brief introduction to Acceptance Test Driven Development and Robot Framework.
4: [PDF]
acceptance test-driven development with robot framework
wiki.robotframework.googlecode.com/.../ATDD_with_Ro...
Dịch trang này
WITH ROBOT FRAMEWORK by Craig Larman and Bas Vodde. Version 1.1. Acceptance test-driven development is an essential practice applied by suc-.
5: Robot Framework
robotframework.org/
Dịch trang này
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test ...
6: Robot Framework - Google Code
https://code.google.com/p/robotframework/
Dịch trang này
Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test ...
7: Selenium 2 and Thucydides for ATDD | JavaWorld
www.javaworld.com/.../111018-thucydides-for-atdd.html
Dịch trang này
18-10-2011 - Find out how Thucydides extends and rethinks ATDD. ... In this article I introduce Thucydides, an open source ATDD framework built on top of ...
8: ATDD | Assert Selenium
assertselenium.com/category/atdd/
Dịch trang này
24-01-2013 - Thucydides In this article I introduce Thucydides, an open source ATDD framework built on top of Selenium 2. Introducing Thucydides ...
9: (ATDD) with Robot Framework - XP2011
xp2011.org/content.apthisId=180&contentId=179
Dịch trang này
Acceptance Test Driven Development (ATDD) with Robot Framework. Executable requirements neatly combine two important XP practices: user stories and ...
Please take a look at the 6th position is Stackoverflow, but on Google is 5th.
And one more question, how to extend my test case if my website doesn't exist in 1st page, and then i'll search on next page and then get the page ID ?
Thanks.
The results on the page could be different between when you test and when you visually inspect. The first step would be to use the seleneium Get Source keyword to dump the HTML of the page. You can then use that to verify whether your code is working properly or not.
To extend your test case, I would start by writing a keyword named "get result" that returns the result, and the ranking. I would write another keyword named "next page" which goes to the next page of results. Then I would write another keyword called "find page with result" that calls "get result" in a loop. In the body of the loop I would call "get result". If the result isn't found, call "Next page" and loop again. Eventually this keyword will find the page with the results.