I know there is an option in PHPUnit to stop-on-failure but I don't want it to stop when any test fails, just when this specific test fails.
For example, in my setUp I connect to a DB, and in the first test I check if it connected to the correct DB. If that fails, I can't run the rest of the tests.
Use the #depends feature of PHP.
If a test depends on another, it will only be executed if that other test succeeded. Otherwise it will be skipped. This allows you to pinpoint problems better.
Usage: Add a PHPDOC block at the top of the test function that should only be executed when the other test is successful, and add a line #depends testConnectToDb.
See http://phpunit.de/manual/current/en/appendixes.annotations.html#appendixes.annotations.depends and http://phpunit.de/manual/current/en/writing-tests-for-phpunit.html#writing-tests-for-phpunit.test-dependencies for some more details.
Related
I got robot framework tests in below schema:
Suite Setup
Test Case 1
Test Case 2
Test Case 3
...
Suite Teardown
In tear down step I have got a loop that goes through all tests cases and do some additional checks for all test cases (i can do this when test cases are execute because it need to wait some time for some operations in external system). If any of this checks will fail, the teardown step is fail and it also fail every test case. I can set tear down keyword to don't fail tear down step but than I will have all pass in test suite.
Is there any option/feature (or walkaround) that will give me possibility to set status and error message of selected test case in tear down step (something like tc[23].status=fail, tc[23].message='something'.
This is not possible, at least not out-of-the-box. In any event I also think this is not a desirable test approach. Every test should be self contained and all the logic to assess PASS or FAIL should be in that test. Revisiting the result is in my view an anti-pattern.
It is understandable that when there is a large pause of inactivity that you would like to progress with your tests. However, I think that parallelising your tests is a better and more stable approach. For Robot Framework there is Pabot to help you with that, but creating your own test runner is possible.
Is there anyway that a Jenkins job can be paused until a notification is received. Ideally with a payload as well?
I have a "test" job which does a whole bunch of remote tests and I'd like it to wait until the test are done where I send a HTTP notification via curl with a payload including a test success code.
Is this possible with any default Jenkins plugins?
If Jenkins 2.x is an option for you, I'd consider taking a look at writing a pipeline job.
See https://jenkins.io/doc/book/pipeline/
Perhaps you could create a pipeline with multiple stages, where:
The first batch of work (your test job) is launched by the first pipeline stage.
That stage is configured (via Groovy code) to wait until your tests are complete before continuing. This is of course easy if the command to run your tests blocks, but if your tests launch and then detach without providing an easy way to determine when they exit, you can probably add extra Groovy code to your stage to make it poll the machine where the tests are running, to discover whether the work is complete.
Subsequent stages can be run once the first stage exits.
As for passing a payload from one stage to another, that's possible too - for exit codes and strings, you can use Groovy variables, and for files, I believe you can have a stage archive a file as an artifact; subsequent stages can then access the artifact.
Or, as Hani mentioned in a comment, you could create two Jenkins jobs, and have your tests (launched by the first job) use the Jenkins API to launch the second job when they complete successfully.
As you suggested, curl can be used to trigger jobs via the API, or you can use a Jenkins API wrapper package for to your preferred language (I've had success using the Python jenkinsapi package for this sort of work: http://pythonhosted.org/jenkinsapi/)
If you need to pass parameters from your API client code to the second Jenkins job, that's possible by adding parameters to the second job using the the Parameterized Build features built into Jenkins: https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build
I have PHPUnit test that directly interacts with database and when the assert fails, it throws an error and then stops the test and doesn't execute the next assert.
What I need is that when an assert fails, it should pass to the next assert and throw an error at the end of the test.
My biggest issue is because I am interacting with database, when I make Insert, Update, GetAll, GetbyID at the end of test I always make Delete.
I know there other ways of testing without interacting with the database but I was asked to make a test that interacts with the database, so what I need something like (SOFT ASSERT-TRY CATCH) in PHPUnit.
So please anyone that has an answer help me.
If you have a configuration file:
<phpunit
stopOnError="false"
stopOnFailure="false">
Set those two attributes to false, or if running off the command line use
--stop-on-error
--stop-on-failure
As shown in below robot file, I have three testcases. I want to stop the test execution, if TESTCASE1 fails. TESTCASE2 should be executed only if TESTCASE1 passes.
*** Settings ***
Library pythonLib
*** Test cases ***
TESTCASE1
boot device
TESTCASE2
configure device
TESTCASE3
restart device
Is there any keyword or custom way to do this ?
There is a command line option for this, if you want the behavior that robot should stop running as soon as any test fails. This option is --exitonfailure. From the robot framework user guide, in a section titled Stopping when the first test fails:
If option --exitonfailure is used, test execution stops immediately if
any critical test fails. Also the remaining tests are marked as
failed.
You might also want to take a look at this answer to the question Automatic failing/non-execution of interdependent tests in Robot Framework, which shows how to write a keyword to implement dependencies between test cases.
There are multiple ways to get the job done, each appropriate for different situations.
--exitonfailure
The command line option --exitonfailure aborts the test run after any testcase fails, unless it is marked noncritical.
Fatal Error
You might only want to abort if exactly TESTCASE1 fails. The Fatal Error keyword exist just for this purpose:
TESTCASE1
${passed}= Run Keyword And Return Status boot device
Run Keyword If not ${passed} Fatal Error
If this seems clunky to you, you can throw fatal errors directly from Python/Java.
Suite Setup
These tools will get the job done, and are appropriate in some cases. Although in the asker's case, I observe:
The first testcase must pass for any other tests to run.
It runs a keyword called boot device.
To me that is not a testcase. That is a setup. If you need a setup to run once before a suite of testcases, you should specify it as a Suite Setup.
***Settings***
Suite Setup boot device
--pause_on_failure, this will stop the execution, whenever script hit the error. Script execution won't be resumed unless you explicitly start.
We are using Robot framework and RIDE tool for test case execution. we have 100+ testcases and test execution takes more than 6 hours to complete.
RF result and log html is great for viewing results. But these 2 files are viewable only after completion of test case execution.
Is there any plugin / tool or mechanism to view the testcase result status during execution. in RIDE tool -"Run" tab - only shows pass:<> fail:<> and not very user useful.
Need real time testcase status report instead of waiting for completion
You can use the listener interface. With it, you can have robot framework call a python function each time a keyword, testcase or suite starts and finishes. For the case where they finish, the data that is passed in will include the pass or fail status.
Using the listener interface (as Bryan Oakley suggested) is surely the most flexible way to intercept test progession status. If you are looking for tools, Jenkins (with Robot Framework plugin) gives you the opportunity to follow a test run in real time at test case granularity. Just start a job and switch to (Jenkins) console to see the output dropping in.