I've been doing some research on the pros and cons of using a headless browser vs real browsers but am not sure how it all comes together.
I'm under the impression that:
tests in a headless browser run much faster
a headless browser will miss certain things that may appear differently on specific browsers
Knowing the above, how do I approach building a test suite? What are valuable uses of the headless browser?
About your impressions:
Tests in a headless browser run much faster
Maybe. Your mileage may vary here. Definitely spinning up a headless browser to perform a test should be faster than spinning up a full-fledged browser, but an overall performance improvement will depend on the tests and the system under test. If a test waits 30 seconds to perform an action, running it on a headless browser doesn't help; if the system under test takes a lot to answer to actions, running it on a headless browser doesn't help either.
A headless browser will miss certain things that may appear differently on specific browsers
True. It's an entire different browser with different engines. In a headless browser, tests will catch bugs that don't depend on the browser and bugs that only happen in a headless browser. It's worth considering the fact that the end users of your system won't use a headless browser normally, so you probably don't care about bugs that only happen in headless.
With this in mind, in order to approach building a test suite, take a closer look to your specific situation:
If your code has browser specific behaviour, you should validate that by writing tests that cover that specific behaviour and running them with the relevant browser.
However, having to run some tests in a specific browser doesn't mean your entire suite has to be run against that "real" browser. You can have generic tests run against a headless browser within your test suite, especially if your runs can benefit of a performance improvement that headless can provide in your case.
If your end users mostly use one browser and you don't expect that to change, maybe it's worth running the whole suite in that browser, taking it as your main focus and reference. You will catch bugs that matter most to your users, and that might compensate the performance loss you potentially get by not running headless.
Having said this, the more the browsers adhere to standards and behave similarly, the less browser specific issues and behaviour will happen, thus the more traction headless testing will get.
We have over 200 web applications. They are used both internally and by our external employees. Our company's standard has been IE8. We are planning on changing that and adopting IE 10 or 11.
Also, our external employees may use the application in their preferred browser (FF, Opera, Safari, diff. versions, etc.).
I know of services such as browershots, but these are mostly to check if layout is consistent across browsers and tell you nothing if the functionality is still working or not. Is there any other way to do an efficient preliminary check for compatibility with newer browsers instead of having to do a full out regression test? If checks fail, the regression tests can follow and then refactoring what is necessary.
We can use tools like Citrix AppDNA or modern.ie and IECT tool as a combination to explore the strategy and update the applications
So recently Microsoft threw the joke that was IE6 into the recycling bin, and said that users should automatically be upgraded to the latest version of IE, if they hadn't already. Unfortunately for web designers, this means that some people (running XP) will be stuck with IE8.
I recently started redesigning a clients website to bring it up to date, and in order to make loading times quicker, save on disk usage, and on bandwidth, I have implemented the use of CSS3 code and then a big but... IE8 Doesn't support CSS3... :(
Is there a significant amount of users on XP / Vista / 7 who still use IE8, or have they moved to alternative, more up to date browsers such as Chrome, Firefox, Safari etc?
Should I worry about trying to find a way to keep the website looking correct in all browsers? or is there a way for me to encourage visitors using IE8, to swap it for another browser?
I imagine this depends on how the website is used. Is it an intranet website, a public website, or an web application that the owner dictates the version. IMO the best way to figure this out is to have Google Analytics plugged into the web app.
I can tell you that the project I am currently on doesnt have the luxuary to rule out IE8. 70% of our users that use the web app I work on use IE8. It seems to still be the most used browser in the business world. (at least from my experience) Heck we still need to support IE7.
However if you can guarantee the web users are using an up to date browser, then thats a different story. Or of course you can say, this app doesnt support IE 8 or lower.
Thats my 2 cents.
IE8 is the default browser in Windows 7, so expect it to hang around forever, just as IE6 has because it was the default in Windows XP.
Use http://code.google.com/p/html5shiv/ or http://code.google.com/p/ie7-js/ to ease the pain.
Yes, but according to WC3 the percentage of usage is now only ~8% for IE8.
Thats still a pretty big number to turn your back on however.
I have no up to date mobile phone personally, and we don't have any in our team to allow us to test mobile versions of our ASP.Net websites.
We have now been asked to provide a mobile version of our website (which is a fairly busy site) but I have no means to check the mobile pages.
Any suggestions? Are there emulators or would it better to have the real thing?
It's always better to have the real thing, but if that's not an option, I have used a plugin for FireFox called "User Agent Switcher"
Another route you could go is to get the Android SDK and load up an emulator. It runs a full version of android, so you could open the browser from there (You could also change screen size/device type) - a little more heavy than the first solution, but potentially another route worth exploring.
You can use simulators/emulators - Android, iPhone etc.
Well Apple provide an iOS emulator, which you can download in the developers section of Apple's own website.
Outside of that, the only other testing environment I'm aware of is Ripple, which is a plugin for Chrome, designed to emulate a range of mobile devices. You can find that at: http://ripple.tinyhippos.com/.
As others have answered there is a lot of different tools to do the testing and that is all fine to a certain extent, for daily testing by developers and testers.
But with 15 years of testing behind me I would never let the site go untested with a few different real devices if the site is an important service - usually this can be done with "staff" phones at no cost if the cost is the problem. I would more or less say; can your company live with publishing a site out there and there might be a risk it won't work on some mobile platforms?
DeviceAnywhere is the tool that we have used a lot. It provides you access to numerous mobile devices using which you can test both apps and websites. They have placed several physical devices which you can see and operate remotely.
It is always better to test on real devices instead of emulators. During development phase, emulators are helpful but for final delivery it is best to test on real phones. Mobile web browsers may or may not support HTML,CSS,JS completely and their individual implementation could be different.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?
As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?
I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.
So no concerns there.
Jeroen van Menen
Lead dev WatiN
If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:
As of Friday, Google has over 50 teams
running over 51K tests per day on
internal Selenium Farm. 96% of these
tests are handled by Selenium RC and
the Farm machines correctly. The other
4% are partly due to RC bugs, partly
to test errors, but isolating the
cause can be difficult. Selenium has
been adopted as the primary technology
for functional testing of web
applications within Google. That's the
good news.
I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.
We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.
I've been trying 'em both out and here are my initial thoughts...
WatiN
The Good
Fast execution.
Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.
The Bad
Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
No native XPath support.
No out-of-the-box remote execution/grid based execution.
Script Example (C#). You can't do this with Selenium (not that I know off, at least):
class IEManager
{
IE _ie = null;
object _lock = new object();
IE GetInstance(string UrlFragment)
{
lock (_lock)
{
if (_ie == null)
{
var instances = new IECollection(true); //Find all existing IE instances
var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
_ie = match ?? new IE();
if (match==null) //we created a new instance, so we should clean it up when done!
_ie.AutoClose = true;
}
}
return _ie;
}
}
Selenium
Slower than WatiN (especially since a new process has to be created).
Built-in CSS selectors/XPath support.
Selenium IDE is good (can't say great, but it’s the best in class!).
Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
Support chrome and other browsers out of the box.
Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.
The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.
Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.
In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.
Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu
Update
Ye Oliver you're right. Ok why's it better?
Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.
Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.
Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,
browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");
... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.
There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara
I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.
The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.
Until now we are a pure Microsoft Shop for delivering solutions for the enterprise and went with WatiN. This may change in the future.
As a more recent source:
Microsoft printed in MSDN Magazine 12/2010 a BDD-Primer with the combination of SpecFlow with WatiN (cool BDD-Behavior Driven Development). Its author Brandon Satrom (msft Developer Evangelist) has also posted in December 2010 a Video Webcast teaching in detail 1:1 his above findings.
There is a Whitepaper from 04/2011 on Supporting ATDD/BDD with SpecLog, SpecFlow and Team Foundation Server (Acceptance Test Driven Development/Behavior Driven Development) from Christian Hassa, whose team built SpecFlow.
I use Watin, but haven't used Selenium. I can say I got up and running quickly on Watin and have had few to no problems. I can't think of anything I have wanted to do that I couldn't figure out with it. HTH
I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.
I recommend WebAii since that's what I've had any success with and when using it my gripes were few. I never tried Selenium and I don't remember using WaTiN much, at least not to the point where I could get it to succesfully work. I don't know of any framework that deals with Windows dialogs gracefully, although WebAii has an interface for implementing your own dialog handlers.
I considered using both. I used the recorder for Selenium to build some tests in FF. I tried to do the same in Watin and found that the Watin Recorder (2.0.9.1228) is completely worthless for our sites. It appeared to be rendering the site in IE6 -- making our site effectively unusable for recording. We don't support IE6. I couldn't find any way to change the browser it is using. I only found one Watin Recorder out there. If there's more than one, or one that is kept up to date, please comment.
The Selenium Recorder IDE for Firefox is simple to use and ports tests to C#. It isn't great at this. I couldn't get porting test suites to work, despite reading a blog post or two that had workarounds. So there's a bit of manipulation of the generated code. Still, it works 90% and that's better than the alternative.
For my money/time, Selenium is superior just for the ease of building new tests. IE doesn't have any good developer toolbars that are anywhere near as good as Firebug, so I'm doing my development in Firefox to begin with, so having a good working recorder in Firefox is a huge bonus.
My conclusion here was a lot like that democracy quote by Churchill: Selenium is the worst form of automated UI testing. Except for all the other ones.
At the risk of going off on a tangent, I'd recommend Axe/WatiN. Axe allows tests to be written in Excel by 'Manual' Testers with no knowledge of the underlying test 'language'. It does need a 'Technician' to write the bespoke actions (IE. Today I had to do a slightly complex Table lookup & cross-reference) but once written the actions can be used in tests by the non-techy testers.
I also heard that the UK Government Gateway project (which I believe has 6K+ tests automated tests) recently ported all their tests from Axe/Winrunner to Axe/Watin within a week!! And many of the tests are pretty complex - I know as I worked on it a few years ago...
I'm looking at Selenium at the moment, as a potential Client uses it. But i do suggest a wee look at Axe as a layer above the 'work horse' tool.
If you have to access iframes, modal dialogs and cross domain iframes WatiN is a way to go. Selenium couldn't handle the iframes it was throwing commandtimeout exceptions. WatiN you could do lot more things especially if the website uses IE specific stuff like ShowModalDialog etc.. WatiN handles all of them very well. I could even do cross domain iframe access.
You will have to do both if you need to do IE and FF testing, but they are only going to work so well for presentation testing. They cant detect if one element is slightly off, just that the elements are present. I dont know of anything that can replace the human eye for UI / presentation testing, though you could do a few things to assist it (take screenshots of the pages at each step for users to review).