We are using Robot framework and RIDE tool for test case execution. we have 100+ testcases and test execution takes more than 6 hours to complete.
RF result and log html is great for viewing results. But these 2 files are viewable only after completion of test case execution.
Is there any plugin / tool or mechanism to view the testcase result status during execution. in RIDE tool -"Run" tab - only shows pass:<> fail:<> and not very user useful.
Need real time testcase status report instead of waiting for completion
You can use the listener interface. With it, you can have robot framework call a python function each time a keyword, testcase or suite starts and finishes. For the case where they finish, the data that is passed in will include the pass or fail status.
Using the listener interface (as Bryan Oakley suggested) is surely the most flexible way to intercept test progession status. If you are looking for tools, Jenkins (with Robot Framework plugin) gives you the opportunity to follow a test run in real time at test case granularity. Just start a job and switch to (Jenkins) console to see the output dropping in.
Related
I got robot framework tests in below schema:
Suite Setup
Test Case 1
Test Case 2
Test Case 3
...
Suite Teardown
In tear down step I have got a loop that goes through all tests cases and do some additional checks for all test cases (i can do this when test cases are execute because it need to wait some time for some operations in external system). If any of this checks will fail, the teardown step is fail and it also fail every test case. I can set tear down keyword to don't fail tear down step but than I will have all pass in test suite.
Is there any option/feature (or walkaround) that will give me possibility to set status and error message of selected test case in tear down step (something like tc[23].status=fail, tc[23].message='something'.
This is not possible, at least not out-of-the-box. In any event I also think this is not a desirable test approach. Every test should be self contained and all the logic to assess PASS or FAIL should be in that test. Revisiting the result is in my view an anti-pattern.
It is understandable that when there is a large pause of inactivity that you would like to progress with your tests. However, I think that parallelising your tests is a better and more stable approach. For Robot Framework there is Pabot to help you with that, but creating your own test runner is possible.
Is there anyway that a Jenkins job can be paused until a notification is received. Ideally with a payload as well?
I have a "test" job which does a whole bunch of remote tests and I'd like it to wait until the test are done where I send a HTTP notification via curl with a payload including a test success code.
Is this possible with any default Jenkins plugins?
If Jenkins 2.x is an option for you, I'd consider taking a look at writing a pipeline job.
See https://jenkins.io/doc/book/pipeline/
Perhaps you could create a pipeline with multiple stages, where:
The first batch of work (your test job) is launched by the first pipeline stage.
That stage is configured (via Groovy code) to wait until your tests are complete before continuing. This is of course easy if the command to run your tests blocks, but if your tests launch and then detach without providing an easy way to determine when they exit, you can probably add extra Groovy code to your stage to make it poll the machine where the tests are running, to discover whether the work is complete.
Subsequent stages can be run once the first stage exits.
As for passing a payload from one stage to another, that's possible too - for exit codes and strings, you can use Groovy variables, and for files, I believe you can have a stage archive a file as an artifact; subsequent stages can then access the artifact.
Or, as Hani mentioned in a comment, you could create two Jenkins jobs, and have your tests (launched by the first job) use the Jenkins API to launch the second job when they complete successfully.
As you suggested, curl can be used to trigger jobs via the API, or you can use a Jenkins API wrapper package for to your preferred language (I've had success using the Python jenkinsapi package for this sort of work: http://pythonhosted.org/jenkinsapi/)
If you need to pass parameters from your API client code to the second Jenkins job, that's possible by adding parameters to the second job using the the Parameterized Build features built into Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build
As shown in below robot file, I have three testcases. I want to stop the test execution, if TESTCASE1 fails. TESTCASE2 should be executed only if TESTCASE1 passes.
*** Settings ***
Library pythonLib
*** Test cases ***
TESTCASE1
boot device
TESTCASE2
configure device
TESTCASE3
restart device
Is there any keyword or custom way to do this ?
There is a command line option for this, if you want the behavior that robot should stop running as soon as any test fails. This option is --exitonfailure. From the robot framework user guide, in a section titled Stopping when the first test fails:
If option --exitonfailure is used, test execution stops immediately if
any critical test fails. Also the remaining tests are marked as
failed.
You might also want to take a look at this answer to the question Automatic failing/non-execution of interdependent tests in Robot Framework, which shows how to write a keyword to implement dependencies between test cases.
There are multiple ways to get the job done, each appropriate for different situations.
--exitonfailure
The command line option --exitonfailure aborts the test run after any testcase fails, unless it is marked noncritical.
Fatal Error
You might only want to abort if exactly TESTCASE1 fails. The Fatal Error keyword exist just for this purpose:
TESTCASE1
${passed}= Run Keyword And Return Status boot device
Run Keyword If not ${passed} Fatal Error
If this seems clunky to you, you can throw fatal errors directly from Python/Java.
Suite Setup
These tools will get the job done, and are appropriate in some cases. Although in the asker's case, I observe:
The first testcase must pass for any other tests to run.
It runs a keyword called boot device.
To me that is not a testcase. That is a setup. If you need a setup to run once before a suite of testcases, you should specify it as a Suite Setup.
***Settings***
Suite Setup boot device
--pause_on_failure, this will stop the execution, whenever script hit the error. Script execution won't be resumed unless you explicitly start.
How to pause the script execution on failures and continue the executions, after performing some operations in Robot framework?
please suggest me.
You can use the Pause Execution keyword from the Dialogs library, which will open a dialog and pause robot until the dialog is dismissed.
Unfortunately, robot has no way of automatically running a keyword on any failure. You'll have to manage that yourself. For example, you could use Run Keyword If Test Failed in a test teardown. Of course, you won't be able to continue that particular test, but it will let you pause before the next test.
If you need to allow the current test to continue after a failed keyword there are many keywords that can help you, such as Run keyword and continue on failure and all of the other Run keyword if... and Run keyword unless... keywords.
Sometimes a test fails because of infrastructure failures, for example a network outage that does not indicate a regression. Is there any alternative in robotframework to PASS / FAIL? Something like ERROR?
Robot Framework offers only PASS or FAIL, nothing like ERROR.
I can see 2 strategies though to handle those intermittent problems (that most of us are facing).
1) use the "wait until keywords succeeds" keyword. For example, if you have to do a GET via REST on a remote server that could be unreachable for some network reason, then instead of
Get MyURL
you could do
wait until keywords succeeds Get http://example.com
and even better option would be to create a custom keyword for that
*** keywords ***
Get_until_succeeds
[Arguments] ${url}
wait until keywords succeeds Get ${url}
So then you just have to call:
Get_until_succeeds http://example.com
2) use the "--rerunfailed" option or Robot Framework that allows you to re-run the tests that failed. The way to use it is to first launch your suite the usual way:
pybot tests
And then give the output.xml of the previous execution as input of another round:
pybot --rerunfailed output.xml tests
(you can then merge the 2 reports and get a single nice report)