In our CI build, we run our unit tests (as expected) & they show up in the build summary, great! How can we control the display name of the parent of the tests?
It appears to be grouping them by the assembly and there's no reference to which assembly in the parent name.
-- VSTest_TestResults_NNN_X
---- test 1
---- test 2
-- VSTest_TestResults_NNN_Y
---- test 1
---- test 2
Build definition:
I assume we would have to customize the above build definition task?
Related
I am trying to learn robot & facing two confusions(m a total noob) .Pretty sure I am making a rookie mistake & would appreciate some help .
1)Every time I execute the tests using the "pabot --testlevelsplit Tests/demo.robot" command, the log.html doesn't show the failed test case (in this case: Fill the login form and select "Teacher" from the User option) even though the terminal output shows it as failed.
The same test case however passes when I run them individually and log.html shows 3 passed test cases.
code[Log showing only the 2 passed cases
terminal showing only 2 passed test cases
all the test cases passed and showing up in logs when executed using the ".robot /Test/demo.robot" command
Basically 2 issues:
1. I plan to execute multiple test cases from argument file. The structure would look like that:
SOME_PATH/
-test_cases/
-some_keywords/
-argumentfile.txt
How should i define a suite setup and teardown for all those test cases executed from file (-A file)?
From what i know:
a) I could execute it in file with 1st and last test case, but the order of test cases may change so it is not desired.
b) provide it in init.robot and put it somewhere without test cases only to get the setup and teardown. This is because if I execute:
robot -i SOME_TAG -A argumentfile /path/to/init
and the init is in test_case folder it will execute the test_cases with a specific tag + those in a folder twice.
Is there any better way? Provide it, for example, in argumentfile?
2 How to provide PATH variable in argumentfiles in robotframework?
I know there is possibility to do:
--variable PATH:some/path/to/files
but is it not for test suite env?
How to get that variable to be visible in the file itself: ${PATH}/test_case_1.robot
For your 2nd question, you could create a temporary environment variable that you'd then use. Depending on the OS you're using, the way you'll do this will be different:
Windows:
set TESTS_PATH=some/path/here
robot -t %TESTS_PATH%/test_case_1.robot
Unix:
export TESTS_PATH="some/path/here"
robot -t $TESTS_PATH/test_case_1.robot
PS: you might want to avoid asking multiple, different questions in the same thread
I want to skip some phpunit tests in scrutinizer.
How can I achive the same?
Where do I need to do configuration changes for the same?
Many CI systems incl. Scrutinizer CI set environment variables in their build environment.
For example the environment variable SCRUTINIZER is set to TRUE. That is only one of many, learn more about Pre-defined Environment Variables on Scrutinizer CI.
Inside the test method (or inside the setUp() method for the whole class) you can check the environment variable (e.g. via $_ENV) and mark the test as skipped.
if (isset($_ENV['SCRUTINIZER'])) {
$this->markTestSkipped(
'Scrutinizer CI build'
);
}
See as well the more general question How to skip tests in PHPunit? and the Phpunit documentation Incomplete and Skipped Tests.
In my case, I have added
./vendor/bin/phpunit --exclude-group Group1, Group2 command as follows in .scrutinizer.yml file at application level to skip phpunits representing these groups as follows :
build:
nodes:
acsi:
tests:
override:
- './vendor/bin/phpunit --exclude-group Group1, Group2'
- phpcs-run --standard=phpcs.xml
- php-scrutinizer-run
I am pretty new to robot framework. I would like to create test cases dynamically without having a input key-value driven approach.
Found some material that suggested the following:
suite = TestSuite('Example suite', doc='...')
tc = TestCase('Example test')
tc.add_step(TestStep('Log', args=['Hello, world!'])
suite.add_test(tc)
I dont see add_step in test case class, Will continue to look around and see if there are any solutions.
The TestSuite object has a keywords attribute which itself has a create method which can be used to create new keywords.
The robot framework api documentation gives this example:
from robot.api import TestSuite
suite = TestSuite('Activate Skynet')
suite.resource.imports.library('OperatingSystem')
test = suite.tests.create('Should Activate Skynet', tags=['smoke'])
test.keywords.create('Set Environment Variable', args=['SKYNET', 'activated'], type='setup')
test.keywords.create('Environment Variable Should Be Set', args=['SKYNET'])
The above gives you the same test as if you had written it like this:
*** Settings ***
Library OperatingSystem
*** Test Cases ***
Should Activate Skynet
[Tags] smoke
[Setup] Set Environment Variable SKYNET activated
Environment Variable Should Be Set SKYNET
I am trying to run Robotframework testcases from eclipse with Robotframework-maven plugin. Can anyone tell me the configuration of POM.xml to run the testcases according to my given order instead of alphabetical order? For example, I have the following tags in the corresponding test suites:
TestSuit1--->
Testcase1.robot -- >MyTestcase1 [Tags] a
Testcase2.robot --- >MyTestcase2 [Tags] b
Testcase3.robot -- - > MyTestcase3 [Tags] c
I want to executes the above test cases random order. If I write in pom.xml
<includes_cli>b,a,c</includes_cli>
It executes the tests according to alphabetical order instead of my given order. Can anyone have a solution for that?
Br,
Dew
You can use --randomize option to execute the test cases in random order as below:
Case 1:
robot --randomize tests <Testcase1.robot>
tests: Test cases inside each test suite will be executed in random order
Case 2:
robot --randomize suites <path/to/Testsuite>
suites: All test suites will be executed in a random order, but test cases inside suites will run in the order they are defined
It looks like the latest version of the maven plugin has a randomize option:
http://robotframework.org/MavenPlugin/run-mojo.html#randomize
options are:
<randomize>all</randomize>
<randomize>suite</randomize>
<randomize>test</randomize>
with the default being no randomization.
Looks like the same options as for the --randomize command line argument for the robot command:
http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#randomizing-execution-order