I have installed the pabot lib but still struggling to trigger my testcases.
Can someone please suggest how to start the testcase run.
I ran it like below :
C:\Python27\SampleScripts\DataDrivenApproach\KITE>pabot Pabt.robot --processes 2
tests
And got this :
get_suite_names: output_file='.\pabot_results\suite_names.xml' does not exist
No tests to execute
Elapsed time: 0 minutes 0.6 seconds
Related
If I run my tests using PHP7.2 or PHP7.1 they are about 3x slower than if I run them using PHP7.0. Is there anyway to get to the bottom of why this is happening?
Even when I run the test suites (Feature & Unit) separately I still see the slow down. It's only when I run the tests individually does the speed difference become insignificant.
I'm using Laravel 5.5.20 and Laravel Homestead 7.0.1. I have 47 rather simple tests, some hitting the database, others just simple assertions; so there isn't anything that should take ages.
I installed johnkary/phpunit-speedtrap to see which tests take the longest so I could remove those but there isn't a specific test that takes a long time because if I remove the offending test, the next one will take ages (see below).
First Run Second Run
Test A 0.2 sec Test A 0.2 sec
Test B. 0.3 sec Test B. 0.3 sec
Test C 0.1 sec Test C 0.1 sec
Test D 0.1 sec Test D 0.1 sec
Test E 9.3 sec REMOVED Test E
Test F 0.3 sec Test F 9.3 sec <-- Test F now takes ages
Test G 0.2 sec Test G 0.2 sec
I am also using an in-memory SQLite3 database, with the Laravel CreatesApplication and RefreshDatabase trait as I want each test to run independently.
I do not have Xdebug installed or running. Is there something known that PHP7.1 and PHP7.2 take a long time to run PHPUnit tests? Is there something else I can install (or even run it with Xdebug) to track down what exactly it is that is causing the issue?
Setup
Laravel 5.5.20
Laravel Homestead 7.0.1 (Per-project installation)
PHPUnit 6.4.4
Vagrant 2.0.1
Virtualbox 5.2.4
Results
PHP 7.2 PHPUnit 6.4.4
Time: 12.4 seconds, Memory: 162.00MB
PHP 7.1 PHPUnit 6.4.4
Time: 12.19 seconds, Memory: 162.00MB
PHP 7.0 PHPUnit 6.4.4
Time: 4.88 seconds, Memory: 162.00MB
I had the same issue as you but with XDebug installed. I found a pretty good hint by a user named Roni on Laracasts (I can't find the link anymore, sorry) which says to run tests with the -n flag of the php command, like this: php -n vendor/bin/phpunit.
According to the documentation at php.net (command line options) this is for running the command without what php.ini is defining. Which means no includes of extensions.
-n No php.ini file will be used
So, for me it runs tests in a minute now, not in 15 minutes. The problem is kind of strange, because it began with php 7.2 on my machine but others in my team don't have the issue, despite the fact that xdebug is installed. I wonder what really is behind this issue.
I've got an interesting problem with Grunt-Concurrent. When a registered task such as Uglify or Karma runs without errors everything is fine. But if Uglification or Karma tests fail then Grunt-Concurrent will continuously loop until the error is fixed. This was annoying with Uglify but with Karma unit tests it's really hard to have it looping until the bug is fixed.
Any ideas of how to solve this?
I can't really provide examples of our exact set up.
It will just keep running with this message until the unit tests all pass:
Running "watch:karma" (watch) task
Waiting...
Running "karma:unit:run" (karma) task
Firefox 39.0.0 (Mac OS X 10.10.0) it should work should work FAILED
Expected true to be false.
...: Executed 2 of 2 (1 FAILED) (0.007 secs / 0.004 secs)
Warning: Task "karma:unit:run" failed.
It then runs it over again and again and again.
Looks like the issue was actually with grunt-contrib-watch and once I started going down that path I found the answer.
I found this great SO question: Prevent `grunt-watch` from looping when there is a syntax error in less files?
I found that by changing spawn: true that it no longer loops like it used to.
When running PHPUnit, if there are no errors it runs in aprox. 6 seconds, nevertheless if there are errors it can take up to 3 or 4 minutes....Only to report the error information.
I'm running PHP version 5.5.3 and PHPUnit version 3.7.28 in Windows.
The tests are painfully simple, even the simplest assertion fail makes the test report take forever.
class MyTest extends PHPUnit_Framework_TestCase
{
public function someTest() {
$this->assertEquals(true, false); // takes forever to report
}
}
Running a database test with --debug I get the following output
D:\xampp\htdocs\blueberry-cms
λ phpunit --debug tests\unit\DatabaseTest.php
PHPUnit 3.7.28 by Sebastian Bergmann.
Starting test 'DatabaseTest::testCRUD'.
F
Starting test 'DatabaseTest::testRetrieval'.
.
Time: 8.91 seconds, Memory: 2.25Mb
There was 1 failure:
1) DatabaseTest::testCRUD
Failed asserting that 19 matches expected 18.
D:\xampp\htdocs\blueberry-cms\tests\unit\DatabaseTest.php:37
FAILURES!
Tests: 2, Assertions: 22, Failures: 1.
The test runs fine until "There was 1 failure:" then it stops outputting text... stops for like 2 or 3 minutes and then finishes outputting text.
I'm having the same issue. I found out a workaround:
Download a .phar copy of PHPunit and put it in your document root. (link to phpunit downloads page)
Instead of running the phpunit command from the php "bin folder" (or wherever you have it in your $PATH), run php phpunit.phar from the root of my repository.
Ej: php phpunit.phar --bootstrap=./tests/bootstrap.php ./tests/
Probably this issue can be solved by updating phpunit.
(In my case (PHP 5.6) the phpunit version in my PHP "bin folder" is 3.7.21, while the .phar I downloaded is version 4.8.36)
I haven't tested this, but it should work as well:
Replace the phpunit version located in your php "bin folder" (or in wherever you have it in your $PATH)
Now every time you run phpunit you will be using the latest version, whithout having to copy the .phar to every new project.
I am building a Qt GUI application via Jenkins. I added 3 build steps:
Building the test executable
Running the test executable
compiling a coverage report with gcovr
For some reason, the shell task for running the test executable stops after execution. Even a simple echo does not run after. The tests are written with Google Test and output xUnit XML files, which are analyzed after the build.
Some tests start the applications user interface, so I installed the jenkins xvnc plugin to get them to run.
The build tasks are as follows:
Build
cd $WORKSPACE/projectfiles/QMake
sh createbin.sh
Test
cd $WORKSPACE/bin
./Application --gtest_output=xml
Coverage Report
cd $WORKSPACE/projectfiles/QMake/out
gcovr -x -o coverage.xml
Now, an echo at the end of the first build task is correctly printed, but an echo at the end of the second is not. The third build task is therefore not even run, although the Google Test output is visible. I thought that maybe the problem is that some of the Google Tests fail, but why whould the script stop executing just because the tests fail?
Maybe someone can give me a hint on why the second task stops.
Edit
The console output looks like this:
Updating svn://repo/ to revision '2012-11-15T06:43:15.228 -0800'
At revision 2053
no change for svn://repo/ since the previous build
Starting xvnc
[VG5] $ vncserver :10
New 'ubuntu:10 (jenkins)' desktop is ubuntu:10
Starting applications specified in /var/lib/jenkins/.vnc/xstartup
Log file is /var/lib/jenkins/.vnc/ubuntu:10.log
[VG5] $ /bin/sh -xe /tmp/hudson7777833632767565513.sh
+ cd /var/lib/jenkins/workspace/projectfiles/QMake
+ sh createbin.sh
... Compiler output ...
+ echo Build Done
Build Done
[VG5] $ /bin/sh -xe /tmp/hudson4729703161621217344.sh
+ cd /var/lib/jenkins/workspace/VG5/bin
+ ./Application --gtest_output=xml
Xlib: extension "XInputExtension" missing on display ":10".
[==========] Running 29 tests from 8 test cases.
... Test output ...
3 FAILED TESTS
Build step 'Execute shell' marked build as failure
Terminating xvnc.
$ vncserver -kill :10
Killing Xvnc4 process ID 1953
Recording test results
Skipping Cobertura coverage report as build was not UNSTABLE or better ...
Finished: FAILURE
Generally, if one Build Step fails, the rest will not be executed.
Pay attention to this line from your log:
[VG5] $ /bin/sh -xe
The -x makes the shell print each command in console before execution.
The -e makes the shell exit with error if any of the commands failed.
A "fail" in this case, would be a return code of not 0 from any of the individual commands.
You can verify this by running this directly on the machine:
./Application --gtest_output=xml
echo $?
If the echo $? displays 0, it indicates successful completion of the previous command. If it displays anything else, it indicates an error code from the previous command (from ./Application), and Jenkins treats it as such.
Now, there are several things at play here. First is that your second Build Step (essentially a temporary shell script /tmp/hudson4729703161621217344.sh) is set to fail if one command fails (the default behaviour). When the Build Step fails, Jenkins will stop and fail the whole job.
You can fix this particular behaviour by adding set +e to the top of your second Build Step. This will not cause the script (Build Step) to fail due to individual command failure (it will display an error for the command, and continue).
However, the overall result of the script (Build Step) is the exit code of the last command. Since in your OP, you only have 2 commands in the script, and the last is failing, it will cause the whole script (Build Step) to be considered a failure, despite the +x that you've added. Note that if you add an echo as the 3rd command, this would actually work, since the last script command (echo) was successful, however this "workaround" is not what you need.
What you need is proper error handling added to your script. Consider this:
set +e
cd $WORKSPACE/bin && ./Application --gtest_output=xml
if ! [ $? -eq 0 ]; then
echo "Tests failed, however we are continuing"
else
echo "All tests passed"
fi
Three things are happening in the script:
First, we are telling shell not to exit on failure of individual commands
Then i've added basic error handling in the second line. The && means "execute ./Application if-and-only-if the previous cd was successful. You never know, maybe the bin folder is missing, or whatever else can happen. BTW, the && internally works on the same error code equals 0 principle
Lastly, there is now proper error handling for the result of ./Application. If the result is not 0, then we show that it had failed, else we show that it had passed. Note, this since the last command is not a (potentially) failing ./Application, but an echo from either of if-else possibilities, the overall result of the script (Built Step) will be a success (i.e 0), and the next Build Step will be executed.
BTW, you can as well put all 3 of your build steps into a single build step with proper error handling.
Yes... this answer may be a little longer than what's required, but i wanted you to understand how Jenkins and shell treat exit codes.
I'm supposed to run some jbehave(automated) tests in bamboo. Once the tests run I'll generate some junit compatible xml files so that bamboo could understand the same. All the jbehave tests are ran as part of a script, because I need to run the jbehave tests in a separate display screen(remember these are automated browser tests). Example script is as follows.
Ex:
export DISPLAY=:0 && xvfb-run --server-args="-screen 0, 1024x768x24"
mvn clean integration-test -DskipTests -P integration-test -Dtest=*
I have one more junit parser task which points to the generated junit compatible xml files. So, once the bamboo build runs and even if all the tests pass, I get red build with the message "No failed tests found, a possible compilation error occurred."
Can somone please help me on this regard.
Your build script might be producing successful test reports, but one (or both, possibly) of your tasks is failing. That means that the failure is probably* occurring after your tests complete. Check your build logs for errors. You might also try logging in to your Bamboo server (as the bamboo user) and running the commands by hand.
I've seen this message in the past when our test task was crashing halfway through the test run, resulting in one malformed report that Bamboo ignored and a bunch of successful reports.
*Check the build log to make sure that your tests are indeed running. If mvn clean doesn't clean out the test report directory, Bamboo might just be parsing stale test reports.
EDIT: (in response to Kishore's links)
It looks like your task to kill Xvfb is what is causing the build to fail.
18-Jul-2012 09:50:18 Starting task 'Kill Xvfb' of type 'com.atlassian.bamboo.plugins.scripttask:task.builder.script'
18-Jul-2012 09:50:18
Beginning to execute external process for build 'Functional Tests - Application Release Test - Default Job'
... running command line:
/bin/sh
/tmp/FUNC-APPTEST-JOB1-91-ScriptBuildTask-4153769009554485085.sh
... in: /opt/bamboo-home/xml-data/build-dir/FUNC-APPTEST-JOB1
... using extra environment variables:
<..snip (no meaningful output)..>
18-Jul-2012 09:50:18 Failing task since return code was 1 while expected 0
18-Jul-2012 09:50:18 Finished task 'Kill Xvfb'
What does your "Kill Xvfb" script do? Are you trying something like pkill -f "[x]vfb"? pkill -f silently returns non-zero if it can't match the expression to any processes.
My solution was to make a 'script' task:
#!/bin/bash
/usr/local/bin/phpcs --report=checkstyle --report-file=build/logs/checkstyle.xml --standard=PSR2 ./lib | exit 0
Which always exits with status 0.
This is because PHP code sniffer return exit status 1 when only 1 coding violation (warning / error) is found which causes the built to fail.
Turns out to be a simple fix.
General bamboo behavior is to scan the entire log and see for any failure codes(1). For this specific configuration i had some 6 scripts out of which one of them was to kill the xvfb(frame buffer). For some reason server is not able to kill xvfb and that task was returning a failure code. Because of this, though all the tests passed, bamboo got one of this error codes from previous tasks and build was failing.
Current fix is to remove the task which kills xvfb and the build went green! \o/.