I have tests of various types (unit, acceptance, etc) which I have assigned multiple labels for granularity
/**
* #test
* #group unit
* #group controllers
*/
/**
* #test
* #group unit
*/
/**
* #test
* #group controllers
*/
Is it possible to run phpunit tests that only match two or more groups? Something like
--group unit|controllers
In this case the only test that should run would be the first test as it has both the unit and controllers group while the other tests would not run.
Using the notation
--group unit,controllers
Runs all tests from unit and then all (or remaining - I can't quite remember) tests from controllers - In large projects this can cause long run times.
--group unit,controllers should work
--exclude also exists to run all tests except those in the group(s) specified
#group unit|controllers is not an allowed syntax
You need to rethink usage of #group annotations, start with splitting to test suites. You can try to follow phpunit rules about files structure or define test suites using xml.
For ex:
<phpunit bootstrap="src/autoload.php">
<testsuites>
<testsuite name="unit">
<directory>webroot/*/Tests/Unit</directory>
</testsuite>
<testsuite name="integration">
<directory>webroot/*/Tests/Integration</directory>
</testsuite>
<testsuite name="controllers">
<directory>webroot/*/Tests/Integration/Controller</directory>
</testsuite>
</testsuites>
</phpunit>
#groups usually used to combine by business entity, for ex. to run all tests related to search functionality of your application.
more information here https://phpunit.de/manual/current/en/organizing-tests.html
Related
Basically 2 issues:
1. I plan to execute multiple test cases from argument file. The structure would look like that:
SOME_PATH/
-test_cases/
-some_keywords/
-argumentfile.txt
How should i define a suite setup and teardown for all those test cases executed from file (-A file)?
From what i know:
a) I could execute it in file with 1st and last test case, but the order of test cases may change so it is not desired.
b) provide it in init.robot and put it somewhere without test cases only to get the setup and teardown. This is because if I execute:
robot -i SOME_TAG -A argumentfile /path/to/init
and the init is in test_case folder it will execute the test_cases with a specific tag + those in a folder twice.
Is there any better way? Provide it, for example, in argumentfile?
2 How to provide PATH variable in argumentfiles in robotframework?
I know there is possibility to do:
--variable PATH:some/path/to/files
but is it not for test suite env?
How to get that variable to be visible in the file itself: ${PATH}/test_case_1.robot
For your 2nd question, you could create a temporary environment variable that you'd then use. Depending on the OS you're using, the way you'll do this will be different:
Windows:
set TESTS_PATH=some/path/here
robot -t %TESTS_PATH%/test_case_1.robot
Unix:
export TESTS_PATH="some/path/here"
robot -t $TESTS_PATH/test_case_1.robot
PS: you might want to avoid asking multiple, different questions in the same thread
I want to skip some phpunit tests in scrutinizer.
How can I achive the same?
Where do I need to do configuration changes for the same?
Many CI systems incl. Scrutinizer CI set environment variables in their build environment.
For example the environment variable SCRUTINIZER is set to TRUE. That is only one of many, learn more about Pre-defined Environment Variables on Scrutinizer CI.
Inside the test method (or inside the setUp() method for the whole class) you can check the environment variable (e.g. via $_ENV) and mark the test as skipped.
if (isset($_ENV['SCRUTINIZER'])) {
$this->markTestSkipped(
'Scrutinizer CI build'
);
}
See as well the more general question How to skip tests in PHPunit? and the Phpunit documentation Incomplete and Skipped Tests.
In my case, I have added
./vendor/bin/phpunit --exclude-group Group1, Group2 command as follows in .scrutinizer.yml file at application level to skip phpunits representing these groups as follows :
build:
nodes:
acsi:
tests:
override:
- './vendor/bin/phpunit --exclude-group Group1, Group2'
- phpcs-run --standard=phpcs.xml
- php-scrutinizer-run
I have robot files in a folder (tests) as shown below:
tests
1_robotfile1.robot
2_robotfile2.robot
3_robotfile3.robot
4_robotfile4.robot
5_robotfile5.robot
6_robotfile6.robot
7_robotfile7.robot
8_robotfile8.robot
9_robotfile9.robot
10_robotfile10.robot
11_robotfile11.robot
Now if I execute '/root/users1/power$ pybot root/user1/tests' command, robot files are running in following order:
tests
1_robotfile1.robot
10_robotfile10.robot
11_robotfile11.robot
2_robotfile2.robot
3_robotfile3.robot
4_robotfile4.robot
5_robotfile5.robot
6_robotfile6.robot
7_robotfile7.robot
8_robotfile8.robot
9_robotfile9.robot
I want to force robot_framework to pick robot files in sequential order, like 1,2,3,4,5....
Do we have any option for this?
If you have the option of renaming your files, you just need to make sure that the prefix is sortable. For numbers, that means they should all have the same number of digits.
I recommend renaming your test cases to have three or four digits for the prefix:
001_robotfile1.robot
002_robotfile2.robot
003_robotfile3.robot
004_robotfile4.robot
005_robotfile5.robot
006_robotfile6.robot
007_robotfile7.robot
008_robotfile8.robot
009_robotfile9.robot
010_robotfile10.robot
011_robotfile11.robot
...
With that, they will sort in the order that you expect.
Following #Emna answer, RF docs ( http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#execution-order ) provides some solution.
So what could you do:
rename all the files to have consecutive and computer numbering (001-test.robot instead of 1-test.robot). This may break any internal references to other files (resources), hard to add test in-between,error prone when execution order needs to be changed
you can tag it as Emna
idea from RF docs - write a script to create argument file which will keep ordering in proper way and use it as argument to robot execution. For 1000+ files it should not take longer than few seconds.
try to design tests to not be dependent from execution order, use suite setup instead.
good luck ;)
Tag the tests as foo and bar so you can run each test separately:
pybot -i foo tests
or
pybot -i bar tests
and decide the order you want
pybot -i bar tests || pybot -i foo tests
I am trying to run Robotframework testcases from eclipse with Robotframework-maven plugin. Can anyone tell me the configuration of POM.xml to run the testcases according to my given order instead of alphabetical order? For example, I have the following tags in the corresponding test suites:
TestSuit1--->
Testcase1.robot -- >MyTestcase1 [Tags] a
Testcase2.robot --- >MyTestcase2 [Tags] b
Testcase3.robot -- - > MyTestcase3 [Tags] c
I want to executes the above test cases random order. If I write in pom.xml
<includes_cli>b,a,c</includes_cli>
It executes the tests according to alphabetical order instead of my given order. Can anyone have a solution for that?
Br,
Dew
You can use --randomize option to execute the test cases in random order as below:
Case 1:
robot --randomize tests <Testcase1.robot>
tests: Test cases inside each test suite will be executed in random order
Case 2:
robot --randomize suites <path/to/Testsuite>
suites: All test suites will be executed in a random order, but test cases inside suites will run in the order they are defined
It looks like the latest version of the maven plugin has a randomize option:
http://robotframework.org/MavenPlugin/run-mojo.html#randomize
options are:
<randomize>all</randomize>
<randomize>suite</randomize>
<randomize>test</randomize>
with the default being no randomization.
Looks like the same options as for the --randomize command line argument for the robot command:
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#randomizing-execution-order
If I want to run a test with just one data from the data provider how would I do it?
I tried the solution on this thread and did not work
Cannot run single test with data provider in PHPUnit
I found out that just adding #n would just work for numerically indexed data providers.
phpunit --filter ClassName::testName#datasetNumber
Add #name for associatively indexed named data providers.
phpunit --filter ClassName::testName#datasetName
Or you can use regular expressions. See the documentation for more examples.
If data sets have names containing spaces, not only quoting will work, you can also use "\ " (at least in a shell like sh or bash)
phpunit --filter ClassName::testCase#data\ set\ name
If data sets have names containing spaces, this works (note the "):
phpunit --filter "ClassName::testCase#data set name"