I want to skip some phpunit tests in scrutinizer.
How can I achive the same?
Where do I need to do configuration changes for the same?
Many CI systems incl. Scrutinizer CI set environment variables in their build environment.
For example the environment variable SCRUTINIZER is set to TRUE. That is only one of many, learn more about Pre-defined Environment Variables on Scrutinizer CI.
Inside the test method (or inside the setUp() method for the whole class) you can check the environment variable (e.g. via $_ENV) and mark the test as skipped.
if (isset($_ENV['SCRUTINIZER'])) {
$this->markTestSkipped(
'Scrutinizer CI build'
);
}
See as well the more general question How to skip tests in PHPunit? and the Phpunit documentation Incomplete and Skipped Tests.
In my case, I have added
./vendor/bin/phpunit --exclude-group Group1, Group2 command as follows in .scrutinizer.yml file at application level to skip phpunits representing these groups as follows :
build:
nodes:
acsi:
tests:
override:
- './vendor/bin/phpunit --exclude-group Group1, Group2'
- phpcs-run --standard=phpcs.xml
- php-scrutinizer-run
Related
Basically 2 issues:
1. I plan to execute multiple test cases from argument file. The structure would look like that:
SOME_PATH/
-test_cases/
-some_keywords/
-argumentfile.txt
How should i define a suite setup and teardown for all those test cases executed from file (-A file)?
From what i know:
a) I could execute it in file with 1st and last test case, but the order of test cases may change so it is not desired.
b) provide it in init.robot and put it somewhere without test cases only to get the setup and teardown. This is because if I execute:
robot -i SOME_TAG -A argumentfile /path/to/init
and the init is in test_case folder it will execute the test_cases with a specific tag + those in a folder twice.
Is there any better way? Provide it, for example, in argumentfile?
2 How to provide PATH variable in argumentfiles in robotframework?
I know there is possibility to do:
--variable PATH:some/path/to/files
but is it not for test suite env?
How to get that variable to be visible in the file itself: ${PATH}/test_case_1.robot
For your 2nd question, you could create a temporary environment variable that you'd then use. Depending on the OS you're using, the way you'll do this will be different:
Windows:
set TESTS_PATH=some/path/here
robot -t %TESTS_PATH%/test_case_1.robot
Unix:
export TESTS_PATH="some/path/here"
robot -t $TESTS_PATH/test_case_1.robot
PS: you might want to avoid asking multiple, different questions in the same thread
I am trying to run Robotframework testcases from eclipse with Robotframework-maven plugin. Can anyone tell me the configuration of POM.xml to run the testcases according to my given order instead of alphabetical order? For example, I have the following tags in the corresponding test suites:
TestSuit1--->
Testcase1.robot -- >MyTestcase1 [Tags] a
Testcase2.robot --- >MyTestcase2 [Tags] b
Testcase3.robot -- - > MyTestcase3 [Tags] c
I want to executes the above test cases random order. If I write in pom.xml
<includes_cli>b,a,c</includes_cli>
It executes the tests according to alphabetical order instead of my given order. Can anyone have a solution for that?
Br,
Dew
You can use --randomize option to execute the test cases in random order as below:
Case 1:
robot --randomize tests <Testcase1.robot>
tests: Test cases inside each test suite will be executed in random order
Case 2:
robot --randomize suites <path/to/Testsuite>
suites: All test suites will be executed in a random order, but test cases inside suites will run in the order they are defined
It looks like the latest version of the maven plugin has a randomize option:
http://robotframework.org/MavenPlugin/run-mojo.html#randomize
options are:
<randomize>all</randomize>
<randomize>suite</randomize>
<randomize>test</randomize>
with the default being no randomization.
Looks like the same options as for the --randomize command line argument for the robot command:
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#randomizing-execution-order
I'm writing some tests which rely on an external binary being present. Having a test case for ensuring binary is present and functioning seems useful. I want multiple tests, expressing:
Is binary found by subprocess.Popen?
Can it do basic X?
Can it do basic Y?
I want a pattern to skip the basic tests X and Y if the first one fails. It would also be good to test several binaries in the same TestCase, with different kinds of X and Y.
Some approaches come to mind:
Decorate test_X() and test_Y() with #unittest.skipIf(basic failed), either through checking TestResults or setting a flag
Have test_basic() call TestResults.addSkip(test_binary_X) upon failure (but this doesn't appear to be exposed)
Put test_basic() into one TestCase and the others into another
Make test_basic() part of setUp(), so it's run for each of X and Y
Unittest doesn't run tests in the order written, so it may be hard to ensure sequence.
# Something like:
class TestBasics(unittest.TestCase):
test_basic(self):
try:
subprocess.Popen(["binary"], stdout=None, stderr=None)
catch OsError as e:
# fail depending on the error
#unittest.skipIf( test binary in path failed )
test_X(self):
# assert
#unittest.skipIf( test binary in path failed )
test_Y(self):
# assert
What do the numbers before test testcase in Test lab indicate? For example:
[1]Login with ur……
What does the [1] mean?
the [1] before the test case in the test lab indicates the number of occurrences of that test case. If you add the same test case twice it will look something like this.
[1] Test Case Name
[2] Test Case Name
it's the number of instances of that test case in the Execution Grid.
each time you add this test case to the grid you get a new instance added with a new number beside it.
It took me a while to understand instances of test 'cases'. I was thinking of it all wrong (because the test lab is where you run the tests, right?).
When you want a specific set of parameters for a test, use test configurations in the Test Plan.
Once the values are set, you create your instance in the Test Lab from the configuration so that you can execute it.
If, on the other hand, you have an instance in Test Lab with some specific values that you like, you can use right-click -> generate configuration.
This creates a configuration for you and you can give it a name of your choosing (something that I really wanted to do in the test lab, until I discovered how Configurations work).
There is also a button in the test plan configurations tab to 'push' updated values to the Test Lab. So while there is a disconnect between configurations and test lab (by design), but it is not such a disconnect that you can't connect them when you want.
In a larger project, I have set up ./tests/Makefile.am to run a number of tests when I call make check. The file global_wrapper.c contains the setup / breakdown code, and it calls test functions implemented in several subdirectories.
TESTS = global_test
check_PROGRAMS = global_test
global_test_SOURCES = global_wrapper.c foo/foo_test.c bar/bar_test.c
Works great. But the tests take a long time, so I would like to be able to optionally execute only tests from a single subdir. This is how I did it at first.
I added the subdirectories:
SUBDIRS = foo bar
In the subdirectories, I added local wrappers and Makefile.am's:
TESTS = foo_test
check_PROGRAMS = foo_test
# the foo_test.c here is of course the same as in the global Makefile.am
foo_test_SOURCES = foo_wrapper.c foo_test.c
This, too, works great - when I call make check in the subdirectory foo, only the foo tests are executed.
However, when I now call make check in ./tests, all tests are executed twice. Once through global_test, and once through the local test programs.
If I omit the SUBDIRS statement in the global Makefile.am, the subdirectory makefiles don't get build. If I omit TESTS from the local Makefile.am's, make check doesn't do anything for the local directories.
I'm not that familiar with automake, but I am pretty sure there is some way to solve this dilemma. Can anybody here give me a hint?
Break your tests up. In your tests/Makefile.am do:
TESTS = foo_test bar_test
and build foo_test bar_test appropriately with something like
foo_test_SOURCES = foo/foo_wrapper.c foo/foo_test.c
bar_test_SOURCES = bar/bar_wrapper.c bar/bar_test.c
Now, if you do a raw 'make check', both tests will be run. If you only want to run one test, you can do that with 'make check TESTS=foo_test' or 'make check TESTS=bar_test' and only the appropriate test will run. Typically, the Makefile.am lists all the tests that will be run by default in TESTS and the user selects alternate tests at make-time. Naturally, if you are running the tests a lot, you can 'export TESTS=foo_test' in your shell session and then only type 'make check'.
Can't you remove from "global_test" any test that is already executed in a subdirectory? (Just so they simply don't get executed twice.)
I think you could maybe overwrite the check rule at the top-level to define an environment variable:
check:
DISABLE_SUBTESTS=1 make check-recursive
and then test DISABLE_SUBTESTS in your sub-directories to decide whether to actually run the tests or not.
(Personally, I'd rather arrange to work in the existing make check framework by concealing the output of my tests, rather than overwriting the produced rules like this.)