Some years ago I´ve tried to work with BDD and some tools of Acceptance tests, like Selenium/Web-driver, Fitnesse and JBehave.
I´d like to back to work with that in my current project, so I would like to know what the community is using of tools to perform that!
My project is based on Java.
Acceptance tests tools ?
BDD tools?
Should I consider a scripting language ?
The most successful teams are using conversations and not worrying too much about the tools!
Having said that, here are a few Java BDD tools that are in use in the community:
JBehave* (still)
Cucumber for the JVM (I don't think it's been as maven-ized as JBehave)
Fitnesse (though I recommend putting "Slim" behind it instead of "Fit")
Custom DSLs (it's not that hard).
Selenium is still the automation tool of choice for Java and the web.
*I helped write JBehave. One reason we got into it in the first place was because the acceptance tests we saw using scripts were such an astonishing mess. Meh. Also, you can't collaborate with the business or have conversations about scripts. I strongly recommend having conversations first**, then worrying about the tools!
** If you're working on your own, buy a rubber duck.
I strongly recommend spockframework + Geb. You need groovy support though. We have lot of tests running as part of CI every night. The reports are in junit format (being enhanced to be used by business users soon) and hence can be published to servers like Hudson or Sonar.
We are building an ASP.NET website using C#.NET language and VSTS 2008.
Would you please let me know which are all the third-party tools those can help us in the complete SDLC of this project?
Thank you..
Regards,
Karina.
Try Microsoft Team Foundation Server. It will handle your source control, work item tracking, bug tracking, reports, and provide you with a Sharepoint project portal.
Phew! There's lots that you can use, basically a complete eco-system has been constructed around this.
Much depends on
the process you'll use to develop the software (waterfall? agile? a mix?)
where your people are (if you have remote folks, you'll need good collaboration)
how many you are (small projects won't need quite as much project tracking software)
the kind of project you're on (a large complex site might need a comprehensive database management system, a small one can just use free tools)
what metrics you need to report
how experienced your developers are
the kind of graphic design expertise you have and need
your budget!
And that's just off the top of my head.
More specifically, here's some third-party tools I've used successfully that I'd not consider starting a project of any size without:
A source code repository: Subversion is a good one, TFS is expensive but does the job
A continuous integration server: I prefer Hudson, others like CruiseControl.NET
A refactoring tool for software developers (Refactor!, Resharper)
A virtual machine system for hosting test systems on a server (I use VMWare)
For larger teams where you're using Agile methodologies and need to report progress, consider Scrumworks.
Putting it another way, I'd suggest that you budget at least $500 per developer for tools, add-ons and geegaws beyond Visual Studio.
At home I use:
tortoises - Source Control - http://tortoisesvn.tigris.org/
ankhsvn - VS Integration - http://ankhsvn.open.collab.net/
CruiseControl.Net - Continuous Integration - http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET
Screwturn wiki - wiki for knowledge sharing - http://www.screwturn.eu/
At work I use TFS.
I'm not sure where to start when it comes to load-testing ASP.NET applications. My team doesn't use VSTS so that option is out. Does anyone have any good suggestions or experience they can share?
I'ved used Microsoft's free Web Application Stress Tool. It lets you record a browsing session, then replay it using multiple clients.
It seems that Microsoft has pulled this application from the download center, but you can use this alternative download (it may require 7-Zip to unpack).
Red Gate Software has a product called ANTS that can be used to do this.
Click Here for product link
Using BrowserMob.com is quite cheap for what it provides. You can listen to an interview with the founder of Browser Mob on Startup Success Podcast #18.
You can use Grinder.
Its a nice Java Based Opensource tool, to load test webapps. Pretty lean as well.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?
As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?
I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.
So no concerns there.
Jeroen van Menen
Lead dev WatiN
If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:
As of Friday, Google has over 50 teams
running over 51K tests per day on
internal Selenium Farm. 96% of these
tests are handled by Selenium RC and
the Farm machines correctly. The other
4% are partly due to RC bugs, partly
to test errors, but isolating the
cause can be difficult. Selenium has
been adopted as the primary technology
for functional testing of web
applications within Google. That's the
good news.
I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.
We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.
I've been trying 'em both out and here are my initial thoughts...
WatiN
The Good
Fast execution.
Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.
The Bad
Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
No native XPath support.
No out-of-the-box remote execution/grid based execution.
Script Example (C#). You can't do this with Selenium (not that I know off, at least):
class IEManager
{
IE _ie = null;
object _lock = new object();
IE GetInstance(string UrlFragment)
{
lock (_lock)
{
if (_ie == null)
{
var instances = new IECollection(true); //Find all existing IE instances
var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
_ie = match ?? new IE();
if (match==null) //we created a new instance, so we should clean it up when done!
_ie.AutoClose = true;
}
}
return _ie;
}
}
Selenium
Slower than WatiN (especially since a new process has to be created).
Built-in CSS selectors/XPath support.
Selenium IDE is good (can't say great, but it’s the best in class!).
Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
Support chrome and other browsers out of the box.
Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.
The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.
Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.
In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.
Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu
Update
Ye Oliver you're right. Ok why's it better?
Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.
Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.
Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,
browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");
... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.
There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara
I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.
The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.
Until now we are a pure Microsoft Shop for delivering solutions for the enterprise and went with WatiN. This may change in the future.
As a more recent source:
Microsoft printed in MSDN Magazine 12/2010 a BDD-Primer with the combination of SpecFlow with WatiN (cool BDD-Behavior Driven Development). Its author Brandon Satrom (msft Developer Evangelist) has also posted in December 2010 a Video Webcast teaching in detail 1:1 his above findings.
There is a Whitepaper from 04/2011 on Supporting ATDD/BDD with SpecLog, SpecFlow and Team Foundation Server (Acceptance Test Driven Development/Behavior Driven Development) from Christian Hassa, whose team built SpecFlow.
I use Watin, but haven't used Selenium. I can say I got up and running quickly on Watin and have had few to no problems. I can't think of anything I have wanted to do that I couldn't figure out with it. HTH
I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.
I recommend WebAii since that's what I've had any success with and when using it my gripes were few. I never tried Selenium and I don't remember using WaTiN much, at least not to the point where I could get it to succesfully work. I don't know of any framework that deals with Windows dialogs gracefully, although WebAii has an interface for implementing your own dialog handlers.
I considered using both. I used the recorder for Selenium to build some tests in FF. I tried to do the same in Watin and found that the Watin Recorder (2.0.9.1228) is completely worthless for our sites. It appeared to be rendering the site in IE6 -- making our site effectively unusable for recording. We don't support IE6. I couldn't find any way to change the browser it is using. I only found one Watin Recorder out there. If there's more than one, or one that is kept up to date, please comment.
The Selenium Recorder IDE for Firefox is simple to use and ports tests to C#. It isn't great at this. I couldn't get porting test suites to work, despite reading a blog post or two that had workarounds. So there's a bit of manipulation of the generated code. Still, it works 90% and that's better than the alternative.
For my money/time, Selenium is superior just for the ease of building new tests. IE doesn't have any good developer toolbars that are anywhere near as good as Firebug, so I'm doing my development in Firefox to begin with, so having a good working recorder in Firefox is a huge bonus.
My conclusion here was a lot like that democracy quote by Churchill: Selenium is the worst form of automated UI testing. Except for all the other ones.
At the risk of going off on a tangent, I'd recommend Axe/WatiN. Axe allows tests to be written in Excel by 'Manual' Testers with no knowledge of the underlying test 'language'. It does need a 'Technician' to write the bespoke actions (IE. Today I had to do a slightly complex Table lookup & cross-reference) but once written the actions can be used in tests by the non-techy testers.
I also heard that the UK Government Gateway project (which I believe has 6K+ tests automated tests) recently ported all their tests from Axe/Winrunner to Axe/Watin within a week!! And many of the tests are pretty complex - I know as I worked on it a few years ago...
I'm looking at Selenium at the moment, as a potential Client uses it. But i do suggest a wee look at Axe as a layer above the 'work horse' tool.
If you have to access iframes, modal dialogs and cross domain iframes WatiN is a way to go. Selenium couldn't handle the iframes it was throwing commandtimeout exceptions. WatiN you could do lot more things especially if the website uses IE specific stuff like ShowModalDialog etc.. WatiN handles all of them very well. I could even do cross domain iframe access.
You will have to do both if you need to do IE and FF testing, but they are only going to work so well for presentation testing. They cant detect if one element is slightly off, just that the elements are present. I dont know of anything that can replace the human eye for UI / presentation testing, though you could do a few things to assist it (take screenshots of the pages at each step for users to review).
What is the best way to test my webforms applications?
Looks like people are loving Watin, and selenium.
Just wondering, why would you call WatiN a unit testing tool? Last time I checked, it ran integration tests.
The best way would be to move all code that doesn't depend on HttpContext to a separate assembly and run unit tests as usual. The rest can be tested with Ivonna. She doesn't test the client behavior, that's where WatiN can be helpful; however, if you want to test your pages or controls in isolation, she's your only choice.
UPDATE: Given WatiN has been stagnant for over a year now, I would direct anyone that needs web ui tests towards selenium, it is in continuous use & development by many contributors, and is actively used by Google.
WatiN is the best that I've found. It integrates into Visual Studio unit testing or nunit & you can do pretty much anything you need in the browser (click links, submit forms, look for text/images, etc.)
See the following questions for similar answers:
What is the best way to do unit testing for ASP web pages (C#)?
Web Application Testing for .Net (watin Test Recorder)
How do you programmatically fill in a form and ‘POST’ a web page?
That's the biggest shortcoming of Webforms -- it's, for all practical reasons, untestable in terms of unit testing of testing controllers, etc.
That is one of the major advantages of the MVC framework.
I tend to favor the approach of separating the buisness logic out of the UI code.
Here's an article that describes a unit test friendly pattern (Model-View-Presenter)
http://www.unit-testing.net/CurrentArticle/How-To-Use-Model-View-Presenter-With-AspNet-WebForms.html
I would use a tool like WaitIn:
" WatiN is Web Application Testing in .NET, and this Test Recorder will generate chunks of source for you by recording your clicks in an embedded IE browser" (from Scott Hanselman's blog - which I found thanks to another post on StackOverflow
WaitIn website
I'd go with WATIR (Web Application Testing in Ruby) - http://wtr.rubyforge.org/. We (Acsys Interactive) have been using for about a year and the tool is great.
I developed a simple wrapper in .NET so that I can execute my WATIR scripts from Unit tests. The framework is incredible and you have entire Ruby power behind you. There's support for Firefox & Safari (FireWatir project).
It's very similar to WATIN (in fact I think WATIN was inspired by WATIR) but I find that WATIR community is much larger than WATIN one.
There're test recorders out there that you can use and tons of tutorials.
It's really your choice. If you feel like the tests need to be in .NET and you don't want to support any other language then your choice is WATIN. On the other hand, if you want to try a fun and quite powerful scripting language (that's what Ruby is) then go for WATIR.
Question to WATIN guys, does it support FireFox/Safari?
Here is a review of Watin,Watir and Selenium
http://adamesterline.com/2007/04/23/watin-watir-and-selenium-reviewed/
Apparently Selenium worked quite slow for the tester but if you'll notice, as one of the comments points out, that this is only the case due to its support of multiple browsers.
However there is a CTP (Community Technology Preview) release of WatiN which offers support for both Internet Explorer and FireFox automation.
I have had a great experience using Selenium. Web tests can be very fragile, but here is an article from my blog where I talk about how to make the tests less fragile.
http://www.unit-testing.net/CurrentArticle/How-To-Make-Web-Tests-Less-Fragile.html