We are in the process of implementing automated regression testing for our applications, and are looking for a solid batch-testing utility. We have QuickTest Professional 10.0, and it comes bundled with 'Test Batch Runner' which appears to be deprecated. It appears in previous versions there was 'Multi-Test Manager', which has been discontinued as well.
What alternatives exist, if any?
The canonical way to do this is via Quality Center, if you don't have QC you can use QTP's automation model from a vbs file. The documentation for this is available in Start -> Programs -> QuickTest Professional -> Documentation -> Automation Object Model Reference
QTP 10 works excellent with Multi Test Manager V8.2.4.
We use it for our project (previously used it with QTP 9.2).
Try google for an installation (of you don't have one), it should be free but just not supported by HP anymore.
From WinRunner times I very extensively used Test Driver scripts with great success due to the following benefits:
non-programming testers can easily create/maintain batches as their stored in XML format
test input files are externally configurable through mapping
a variety of customization parameters supported, from login credentials to prefixes and switches
test dependencies could be established so that if critical test cases is failed the whole branch of dependant test cases is skipped.
Now I continue using Test Drivers and introducing them to clients.
And Test Driver approach was integrated not only by the client companies that do not use Quality Center. Some others followed it because it gives much more flexibility and robustness in automated test plan execution.
Thank you,
Albert Gareev
http://automationbeyond.wordpress.com
I echo Motti...if i get your question right ....You can see the below written link as well..
Work with Test Batch runner
Related
I have reportportal installation running on Windows box. I am planning to use it as dashboard to look at unit test and other automated test results. I understand reportportal integration with unit test frameworks is done at the logger level so that the test app itself can send results back to dashboard.
I have a scenario where the test application is an exe that I want to launch by sending a command from dashboard to system under test.
Are there any provisions for doing it?
Do I have to build an agent that talks to reportportal using its api for this?
Thanks
No, nothing similar at the moment.
It is pretty popular request, so we have it in backlog, but still focus on test reports aggregations first. And the other types of functionality will come later.
We have set of automated test cases around 2000 and we need to run them daily on every new build that come up. It's currently taking 4 hours to finish the test on one machine. In order reduce this we planning to run tests in batches (500 per batch) on same machine by initiating multiple browsers of same type. Say 4 firefox browser sessions per test suite. So it can finish in 1 hour time.Is it possible to achieve this using selenium webdriver and testng? Please suggest.
It is possible using Selenium Grid and TestNG. Grid can help distribute your tests on various machines or browser instances as you require. To get started, refer : Grid2
I think you might need to change your driver instantiation code a bit to include RemoteWebDriver instead of concrete drivers, but would be ok if your driver instantiation code in your framework is isolated. TestNG and Grid can help provided your tests are well written to support parallel execution.
For TestNG, you can refer, parallel running in TestNG.
If you are using Python - the best way to go is to use py.test to drive the tests. To distribute the tests the pytest-xdist plugin can be used.
Anyway, for both Java and Python you can use Jenkins to run/distribute your tests (using Selenium plugin)
We are a MSFT shop with a far-reaching MSDN license.
After many years of doing things wrong, we finally have to start doing automated testing.
My group is the Guinea pigs at this. We need to create what was not there before. We looked at the multitude of options out there. Some people get by just fine with open-source alternatives such as CC.Net, Bamboo, MbUnit, etc. We want to give MsTest, CodedUI, Team Build a good try ... might as well because of MSDN licensing and MSFT focus.
The plus and minus of doing things the MSFT way is that MSFT makes monolithic things. You have got to install various tools that play with each other nicely, but with outsiders - not necessarily. The plus is that when things are done correctly, it should all function rather smoothly. There is the option of gated check-ins, of using TFS to store the reports, etc.
Frankly, I am confused by all of the options. Our traditional build system was hacked together with a bunch of perl, batch scripts, executables, but now the build team switched to Team Build, which ought to be cleaner, but for the most part it is just a wrapper to the same old perl crap.
I am inclined to hack things together for testing too, because I can at least see what the pieces are. So, I envision the poor man's version as:
* A dedicated fast computer to run tests
* Some script to copy build files (test code as well as product code) over to that computer.
* A batch/perl script which would run mstest.exe from command line and execute a few test batches on some by-category filter within some test dlls (the product is so huge, that we do want to organize tests by various categories).
* Some script which will invoke the latter script remotely from the build server using psexec.exe (http://technet.microsoft.com/en-us/sysinternals/bb897553), as well as grabbing the xml output from a shared drive, and then sending out an email with results to those who are interested.
This can probably work, but then I have to worry about how well error handling can work with so many potential points of failure. It would be nice to configure things the "right way", taking advantage of whatever MSFT has cooked up. I am just not sure where to turn for a good guide. Have you done something like this?
Eventually we will want to have a farm of test computers, if we are to run out of the allotted time. Something else of concern is - for coded ui tests to succeed, I think a user has to be logged in, so I am not sure if psexec will be of much help here.
Can you share your positive/negative experience, point me to a good guide perhaps? Thanks!
Here are some tips off the top of my head if you want to get started with testing using the MS tools:
If you have an MSDN subscription, install a Test Rig by installing the Test Controller on your network and the the Test Agent service on each of the machines that will be collecting diagnostic data. See the following link for reference: http://msdn.microsoft.com/en-us/library/dd293551.aspx.
Add a Test Project to your solution. See the first part of the following blog post: http://blogs.microsoft.co.il/blogs/eranruso/archive/2010/03/27/visual-studio-2010-coded-ui-test-user-guide-create-a-simple-coded-ui-test.aspx.
Automated test options can be configured through the .testsettings file(s) that are added automatically when you add a Test project (you can also manually add these files to your solution).
Install Team Foundation Server (2010 recommended) in order to take advantage automating your tests with a daily build. You will also need TFS 2010 if you want to use the VS2010 Test Manager tool to define test environments and plan manual tests (these can be fully automated with CodedUI). Customize your new automated build to setup / deploy your application after build and set the build to run tests. Deployment will likely not be necessary for unit tests, but they will be for Web Performance and CodedUI test types.
If you have VS Ultimate or Test Professional licenses, you can also go further and set up virtual test labs using "Lab Management" features.
Is it possible to run Google NaCl outside of the browser as a process sandbox?
Yes. The standalone build of Native Client contains a program called "sel_ldr" which runs the NaCl sandbox outside of the web browser. A lot of NaCl's test suite tests NaCl by running programs under sel_ldr rather than under the browser.
The Native Client documentation is probably a good starting point for trying to figure out how everything works.
As Mark Seaborn and Bennet Yee indicate, you are probably best off looking into using sel_ldr. More details about how sel_ldr is included in "The life of sel_ldr".
I haven't personally implemented anything that uses NaCl outside of a browser, so I can't unfortunately comment on the ease of doing so.
To add to Mark's answer, take a look at the sel_universal target. There are obviously some interfaces that cannot be supported, e.g., Pepper interfaces, since such a stand-alone run doesn't involve a browser. Furthermore, the stable ABI that we support is that which is exposed by the "integrated runtime" or irt, and the irt code thunk assumes that the browser is present. a standalone use of NaCl via sel_ldr would probably have to use the syscall interface initially, until an alternate irt is written. (NB: we make no guarantees about the stability of the syscall interface.)
You can try run.py which included in Native Client's source.
Here is an example that runs a hello world program.
make test_hello_world_nexe
python native_client/run.py out/Debug/hello_world_newlib_x64.nexe
This script can build, search and invoke sel_ldr, and pass proper arguments to it automatically.
Can I use Selenium IDE on a Drupal system?
I found http://drupal.org/project/selenium but that involves downloading Core and not using my current machine.
Does anybody know of a way to use the IDE with Drupal, or if not what do you suggest I do?
You can, in the same way that you can use selenium on just about any web page.
From what I can see (though I haven’t' used it) the selenium module seems to be more about using Drupal to store and run selenium tests, which is useful, but will not explicitly test your Drupal site. (edit: this module dosn't look like it is currently maintained)
Drupal comes with it's own testing framework which uses simpletest. This can do a lot of the testing that you will need. It can handle form submissions and check for text in a page. There are two cases that I personally have found that it is not very useful for:
Data integrity, where you would like to check that the content of a particular page is correct where you know what to expect on your system
Two is checking the site layout, where you wish to check your live site that certain elements appear as they should do.
Selenium can be used for both of these, in the simplest case you can record tests against your site and play them back to check it. You can check for an elements existence and style. You can check that elements have the text you would expect. Even a few of these can be useful as a tool for checking the configuration of your site.
It is also possible to integrate selenium with simplest so that while running your simpletests through Drupal it will run selenium tests, however this can get very complicated and confusing.
There's a nice new module that takes Selenium exports (in PHP) and converts them to Drupal SimpleTest suites. http://drupal.org/project/simpletest_selenium
This is nice because you can then do continuous integration testing by using Drush to script the running of your tests.
The question is what do you want to achieve?
testing a Drupal site?
saving / playing back configuration settings (automation through the browser)?
The Firefox & Selenium IDE (FF extension) combination works great with Drupal for functional testing.
It also works great if you want to capture some Drupal configuration settings (that you'd like to play back e.g. in another environment) but there are better ways - Patterns and Strongarm modules - to achieve that.
A couple of tips & tricks:
while capturing clicks be careful with timings (between clicks), sometimes you need to add "pause" lines to your test case,
for HTML form checkboxes Selenium often records the "click" action instead of the much safer "check" or "uncheck" action (this is important if you run your test case several times against the same page / form),
if your page reloads when you click a certain link, use "clickAndWait" instead of the simple "click" action,
use "verifyTextPresent" often, especially if you're running your tests "blindly", using Selenium RC.
I use selenium IDE all the time. It's an awesome tool for productivity and communication.
But Drupal's Simpletest framework has a built in browser for user testing. So you probably won't that module.
See:
http://www.lastcraft.com/web_tester_documentation.php
http://drupal.org/node/265762
http://www.lullabot.com/articles/drupal-module-developer-guide-simpletest
You can test Drupal 7 with SeLite, which extends Selenium IDE. It allows your tests to access (read and write to) a test DB (isolated from the DB of the tested application).
It's especially good for Drupal 7, which can have its data in (separate) SQLite DB. So if your test environment can use SQLite, your test data lifecycle will be very easy. However, you can benefit from SeLite even if your test Drupal instance uses other type of DB.
See https://code.google.com/p/selite/wiki/ProjectHome and https://code.google.com/p/selite/wiki/DrupalTutorial.