Some years ago I´ve tried to work with BDD and some tools of Acceptance tests, like Selenium/Web-driver, Fitnesse and JBehave.
I´d like to back to work with that in my current project, so I would like to know what the community is using of tools to perform that!
My project is based on Java.
Acceptance tests tools ?
BDD tools?
Should I consider a scripting language ?
The most successful teams are using conversations and not worrying too much about the tools!
Having said that, here are a few Java BDD tools that are in use in the community:
JBehave* (still)
Cucumber for the JVM (I don't think it's been as maven-ized as JBehave)
Fitnesse (though I recommend putting "Slim" behind it instead of "Fit")
Custom DSLs (it's not that hard).
Selenium is still the automation tool of choice for Java and the web.
*I helped write JBehave. One reason we got into it in the first place was because the acceptance tests we saw using scripts were such an astonishing mess. Meh. Also, you can't collaborate with the business or have conversations about scripts. I strongly recommend having conversations first**, then worrying about the tools!
** If you're working on your own, buy a rubber duck.
I strongly recommend spockframework + Geb. You need groovy support though. We have lot of tests running as part of CI every night. The reports are in junit format (being enhanced to be used by business users soon) and hence can be published to servers like Hudson or Sonar.
Related
We are evaluating and looking for a workflow engine which support .NET-Core and I'd really appreciate the community input. I would like to hear on the suggestion based on your guys implementation stories.
My main evaluation criteria, so far, are below;
open source and OEM friendly license
production installations (success stories are a great help)
technical support available
open standards support - BPMN
dynamic creation/assembly of the workflow based on input
embeddable
Currently I am evaluating Elsa, Workflow Core, Argo, and Airflow. Elsa seems like a good candidate as well but never used it.
Do you guys have any successful deployments on Elsa workflow engine?
Full disclosure: I am the project lead of Elsa, but I will try and be as objective as I can.
Elsa does not currently support BPMN, so if this is a hard requirement then Elsa might not be suitable for your project. At least not until it implements BPMN in the future.
As for technical support, there is no official paid support available as of yet, but the community is very friendly & helpful, though still relatively small.
Dynamic creation based on input is possible since you can programmatically define workflows. But you cannot update workflows while it executes (which would be more or less similar to being able to update your C# program statements as your program runs). Not sure if this is what you are looking for or not?
Other than that, Elsa is OEM friendly, runs in production successfully at several companies that I know of and is embeddable.
I am currently working on replacing an outdated test system that we're using at the company I work at with Robot Framework. The current system is used for function testing embedded systems that we produce. We are using an in-house software for executing the test scripts, which is good consider that the operators that are running the tests doesn't have a broad knowledge in programming. As it is right now they just need to know the name of the product and then press "Start test" in the GUI, which is exactly what we want.
What I've learned and seen now after spending some days playing around in Robot Framework is that there's no built-in GUI in some sort for executing test cases. I haven't even found a third-party solution for this, except using software like Jenkins, but that is not a good solution when the test needs to run locally on the computer where the operator is working. Running the tests from the terminal directly is not a good solution either, just because it's not that "user-friendly".
I've read about the Robot Framework Public API, which is so well documented that I could pretty easy build a new "Test Executor", but if there exists software that does this already I'm glad to hear about it!
I know that RIDE has the functionality to run selected tests via a GUI, but what I've experienced is that it's pretty easy to make horrible mistakes in that software. Don't get me wrong, RIDE is a awesome tool for creating .robot scripts, but it is not great as a "Test Executor", especially not when we really need to keep a constant production flow to meet our deadlines.
How should I handle this? Do you build your own tools for executing tests, or is it that I've missed something?
We are in search of an automated testing tool for our project. As we are in testing department we prefer a tool which would have less programming in it. Please suggest some tools for us .Till now we are testing our application manually.
Our project is being developed in Java.
Is there any freeware tool that I could use or is it better to go for a paid tool?
Thanks in Advance.
Less programming? You'll need something like JUnit to write unit tests if you want to do serious regression testing, but unit tests require you to write some code
Here's a big list of open-source testing tools, some of them may offer what you want: http://java-source.net/open-source/testing-tools/junit
For example, T2 claims to be a random testing tool. As one, it is fully automatic, but one must keep in mind that the code coverage of random testing is in general very limited. It should be used as a complement to other testing methods. T2 checks for internal errors, run time exceptions, method specifications, and class invariant.
Not sure if you mean a CI tool or not, but we use Hudson at Zappos and it works pretty well.
http://hudson-ci.org/
..and there's also CruiseControl: http://cruisecontrol.sourceforge.net/
If you're not talking about CI, maybe you mean QA testing - in which case you should take a look at something like Selenium (for web apps):
http://seleniumhq.org/
If you're doing GUI testing? I'm not really familiar with that area, but I've heard about WinRunner and Rational:
http://en.wikipedia.org/wiki/HP_WinRunner
http://www-01.ibm.com/software/rational/offerings/quality/
..though neither are really free tools. Something like AutoIT might help you move widgets around, but it lacks the reporting parts:
http://www.autoitscript.com/autoit3/index.shtml
There could be two answer to you question:
Besides Selenium, though it has ample of advantages, I am reading about another tool which uses same API which Selenium use. The only changes in API I have seen so far is it reduces the complexity of functions thus making it more easier and simpler for user who is learning.
The tool is called 'Helium' and it has 50% (and more) less complex functions and code as Selenium has.
The only problem with this tool is it is paid tool for learning purpose and for implementing not-so-big scale project you can use it. But yeah after some time its gonna cost you.
I have implemented some code on Helium. Please let me know , if you face any issue initially or you are thinking to implement it.
Other being, you can use Selenium Builder(http://khyatisehgal.wordpress.com/2014/05/26/selenium-builder-exporting-and-execution/) which is an advanced form of Selenium IDE. It imports your command in different languages and does work more effectively and efficiently as Selenium IDE does(http://khyatisehgal.wordpress.com/2014/05/25/selenium-builder/) . So you can import scripts in Eclipse IDE and just execute them as is.
Please let me know , if you have any doubt in any of the tool.
We're in the initial stages of a large project, and have decided that some form of automated UI testing is likely going to be useful for us, but have not yet sorted out exactly how this is going to work...
The primary goal is to automate a basic install and run-through of the app, so if a developer causes a major breakage (eg: app won't install, network won't connect, window won't display, etc) the testers don't have to waste their time (and get annoyed by) installing and configuring a broken build
A secondary goal is to help testers when dealing with repetitive tasks.
My question is: Who should create these kinds of tests? The implicit assumption in our team has been that the testers will do it, but everything I've read on the net always seems to imply that the developers will create them, as a kind of 'extended unit test'.
Some thoughts:
The developers seem to be in a much better position to do this, given that they know control ID's, classes, etc, and have a much better picture of how the app is working
The testers have the advantage of NOT knowing how the app is working, and hence can produce tests which may be much more useful
I've written some initial scripts using IronRuby and White. This has worked really well, and is powerful enough to do literally anything, but then you need to be able to write code to write the UI tests
All of the automated UI test tools we've tried (TestComplete, etc) seem to be incredibly complex and fragile, and while the testers can use them, it takes them about 100 times longer and they're constantly running into "accidental complexity" caused by the UI test tools.
Our testers can't code, and while they're plenty smart, all I got were funny looks when I suggested that testers could potentially write simple ruby scripts (even though said scripts are about 100x easier to read and write than the mangled mess of buttons and datagrids that seems to be the standard for automated UI test tools).
I'd really appreciate any feedback from others who have tried UI automation in a team of both developers and testers. Who did what, and did it work well? Thanks in advance!
Edit: The application in question is a C# WPF "rich client" application which connects to a server using WCF
Ideally it should really be QA who end up writing the tests. The problem with using a programmatic solution is the learning curve involved in getting the QA people up to speed with using the tool. Developers can certainly help with this learning curve and help the process by mentoring, but it still takes time and is a drag on development.
The alternative is to use a simple GUI tool which backs a language (and data scripts) and enables QA to build scripts visually, delving into the finer details of the language only when really necessary - development can also get involved here also.
The most successful attempts I've seen have definitely been with the latter, but setting this up is the hard part. Selenium has worked well for simple web applications and simple threads through the application. JMeter also (for scripted web conversations for web services) has worked well... Another option which is that of in house built test harness - a simple tool over the top of a scripting language (Groovy, Python, Ruby) that allows QA to put test data into the application either via a GUI or via data files. The data files can be simple properties files, or in more complex cases structured (something like YAML or even Excel) data files. That way they can build the basic smoke tests to start, and later expand that into various scenario driven tests.
Finally... I think rich client apps are way more difficult to test in this way, but it depends on the nature of the language and the tools available to you...
In my experience, testers who can code will switch jobs for a pay raise as developers.
I agree with you on the automated UI testing tools. Every place I've worked that was rich enough to afford WinRunner or LoadRunner couldn't afford the staff to actually use it. The prices may have changed, but back then, these were in the high 5-digit to low 6-digit price tags (think of the price of a starter home). The products were hard to use, and were usually kept uninstalled in a locked cabinet because everyone was afraid of getting in trouble for breaking them.
I worked over 7 years as an application developer before I finally switched to testing and test automation. Testing is much more challenging than coding, and any automation developer who wants to succeed should master testing skills.
Some time ago I put my thoughts on skill matrices in a couple of blog posts.
If interested to discuss:
http://automation-beyond.com/2009/05/28/qa-automation-skill-matrices/
Thanks.
I think having the developers write the tests will be of the most use. That way, you can get "breakage checking" throughout your dev cycle, not just at the end. If you do nightly automated builds, you can catch and fix bugs when they're small, before they grow into huge, mean, man-eating bugs.
What about the testers proposing the tests, and the developers actually writing it ?
I believe at first it largely depends on the tools you use.
Our company currently uses Selenium (We're a Java shop).
The Selenium IDE (which records actions in Firefox) works OK, but developers need to manually correct mistakes it makes against our webapps, so it's not really appropriate for QA to write tests with.
One thing I tried in the past (with some success), was to write library functions as wrappers for Selenium functions. They read as plain english:
selenium.clickButton("Button Text")
...but behind the scenes check for proper layout and tags on the button, has an id etc.
Unfortunately this required a lot of set up to allow easy writing of tests.
I recently became aware of a tool called Twist (from Thoughtworks, built on the Eclipse engine), which is a wrapper for Selenium, allowing plain English style tests to be written. I am hoping to be able to supply this to the testers, who can write simple assertions in plain English!
It automatically creates stubs for new assertions too, so the testers could write the tests, and pass them to developers if they need new code.
I've found the most reasonable option is to have enough specs such that the QA folks can stub out the test, basically figure out what they want to test at each 'screen' or on each component, and stub those out. The stubs should be named such that they're very descriptive as to what they're testing. This also offers a way to crystalize functional requirements. In fact, doing the requirements in this fashion are particularly easy, and help non-technical people really work through the muddy waters of their own though process.
The stubs can be filled in via a combination of QA/dev people. This allows you to CHEAPLY train QA people as to how to write tests, and they typically slurp it up as it furthers their job security.
I think it depends mostly on the skill level of your test team, the tools available, and the team culture with respect to how developers and testers interact with each other. My current situation is that we have a relatively technical test team. All testers are expected to have development skills. In our case, testers write UI Automation. If your test team doesn't have those skills they will not be set up for success. In that case, it may be best for developers to write you UI automation.
Other factors to consider:
What other testing tasks are on the testers' plate?
Who are your customers and what are their expectations related to quality?
What is the skill level of the development team and what is their willingness to take on test automation work?
-Ron
My team works on a Medium sized product which takes about 2 hours to build on a single dual-core machine.
As part of an effort to improve productivity I am looking for alternatives to improve our build process.
Currently we build C++ code and .NET code using VS2005 solutions, we also have some legacy code built using makefiles and we pack the products using installshield. We also have unit tests written in UnitTest++, Nunit, CPPUnit and some tests that we wrote ourselves without any testing framework. Everything is coordinated by a set of Perl scripts that we wrote.
I am looking for a product, or a suite of products, or a bunch of non-related products that will allow me to do the following:
Improve building time. I tried Incredibuild for parallelizing the build with some success. I will be happy to find more alternatives (cheaper ones???)
Improve the build process definition. I need something to replace our complicated perl scripts with something that will allow me to define the build process easily.
Improve our ability to discover problems in the build (maybe a web interface for looking at build outputs, highlight compilation errors, gather statistics on build times and build failures etc.
Any other nice features that can help us improve our build management.
Consider NAnt: http://nant.sourceforge.net/, CruiseControl: http://en.wikipedia.org/wiki/CruiseControl, and a faster build system. there are some .NET related links at: http://confluence.public.thoughtworks.org/dashboard.action
Edit: Gradle is probably the best choice now.
Try FinalBuilder it's very compreshesnsive build managment tool
Sakin,
I think you may be looking for two different tools - although occassionally you will find them packaged together.
One tool would be the build management system that schedules the builds, shows you the outputs, gathers statistics, etc. Tools like FinalBuilder, CruiseControl and AnthillPro play in that space.
The other tool would be the one that tears apart your build into parallelizeable pieces and speeds the build. Incredibuild is pretty good at what they do in that respect and certainly cheaper than some alternatives. There are free options built into some of the open source tools like gmake, but my understanding is that you'd need to do some pretty serious work to get that to work with solution files.
You should be able to mix and match at will. We have customers using Anthill with Incredibuild, gmake, and a number of other build speeding tools.
I used Automated QA's Automated Build Studio a few years ago at a previous job and it was pretty good and fairly cheap. I haven't used it since then, so I'm just recommending you look at it.
If you have an MSDN subscription you should look at TFS. TFS Build is pretty robust and is integrated into Visual Studio and works hand in hand with Team System/TFS Source Control (you didn't say what source control you're using).
We have a process similar to yours, although it's not a VS project. Up until recently we used perl scripts and tinderbox to build and track the status of the builds and unit tests. We've recently made the jump over to Hudson and it seems to work great.
https://hudson.dev.java.net/
Don't be put off by the java label, our projects are c/c++ and they work fine. Hudson can execute shell commands, so we can build our projects (makefiles) and run our unit tests fine. Right now we have it set on a timed build, which when it finishes (if successfully) kicks all our unit tests off in parallel.
It's a great system and worth checking out if it's what you're looking for. It logs console ouput and has tons of great addons (logs and charts warning counts, disk space used size, build on CM change) and a nice web-interface for the team to watch.
Oh yea it's free..
I wouldnt advise HUDSON .. there is unnecessary too much bragging about it ..
hudson is limited tools other than it can do much more and have a better bui8lt core architecture
The suggestion to go with anthillpro and incredibuild is a good one !!
With regard to the "Hudson has extensive plugins .. blah blah blah " its really is a sham ..
What ever hudson CANNOT achieve it is core architecture it does so as a plugin
Surprisingly if i have to mask a passsword or any credential i cannot do it without the "password masking" plugin ..
Ideally if a system is "enterprise scalable" this masking should be built into it core
This just one example .. i have a clear analysis for these tools but cannot upload a document here
Moving to the other side
Hudson "looks and feels" "nice & cute" and is relatively light weight