I have to run the Testsuites across Windows (Chrome, Firefox, and IE) and macOS (Safari and Chrome).
Wanted to implement best strategies to Synchronize AUT with Robot test scripts while testing around Cross browser and Cross platform.
Somehow felt handling Safari browser on macOS is bit challenging. How best I try add some wait but still fail for each run with some 'ElementNotVisibleException'.
However Chrome on Mac OS is quite okay when the same is tested on Windows machine.
Any thoughts would be highly appreciated on best handling mechanisms for Safari on Mac OS.
Thanking in advance!!!
you can use Wait Until Element is Visible Keyword from selenium2Librray from robot framework. the best way is to do it on safari is try executing the scenario manually first and note the maximum time it takes to load the webElement. Once you note down the time, put this time as you max timeout. Below is one example
Wait Until Element Is Visible ${mention_locator_here} timeout=${element_timeout_in_seconds}
You can explore those options Here
Chrome is on the verge of definitly break compatability with NPAPI, and IE breaking with ActiveX the future of Java Applets is dark. Currenty we actively use a secure applet for out client organizations that enables their users to upload a bunch of files from their file system to our servers with the click of a button. The applet has full access to any configured drive, including network drives.
With the imminent death of the applet this functionality is going to be lost if we don't find an alternative. I have already tried to explore different solutions, including the chrome FileSystem API but that is currently only available for Chrome (http://caniuse.com/#feat=filesystem) and has limited access.
Does anybody know about an alternative to keep supporting the much appreciated functionality? Unfortunately we are obligated to support all browser down to IE8.
I've written a post about this here.
Once Google Chrome was the first to announce that they won’t be supporting NPAPI anymore, they were also the first to provide a new architecture in order to rewrite your code to work on their browser. You can take a look on Native Messaging, which “can exchange messages with native applications using an API that is similar to the other message passing APIs”. The problem is that this approach only works on Chrome, is not something that you can adapt to other browsers.
A more useful approach is FireBreath, a browser plugin in a post NPAPI world. Check the words below from one buddy of the project:
“FireBreath 2 will allow you to write a plugin that works in NPAPI, ActiveX, or through Native Messaging; it’s getting close to ready to go into beta. It doesn’t have any kind of real drawing support, but would work for what you describe. The install process is a bit of a pain, but it works. The FireWyrm protocol that the native messaging component uses could be used with any connection that allows passing text data; it should be possible to make it work with js-ctypes on firefox or plausibly WEB-RTC or even CORS AJAX in some way. For now the only thing we needed to solve was Chrome, but we did it in a way that should be pretty portable to other technologies.”
In light of the answer provided by Uly Marins I have researched the options suggested. Unfortunately these options weren't viable for our application, because the mayority of our users do not have sufficient rights to install third party plugins. Additionally the API is still in Beta which won't do any good in a stable production environment.
The main problem we wanted to solve was the abbility to delete files from the accessed folders. It seemed like one of the mayor goals of the removal of the NPAPI support was exactly to prevent this kind of possibility. Therefore we needed to reduce our goals to a simple solution that was still acceptable for our users, with the additional training on how to clear the selected folder manually (because most of our users are almost computer illiterate and needed to access network folders).
Long answer short. The requested solution is just not possible anymore and had to be replaced by a simpler solution and additional training.
We have over 200 web applications. They are used both internally and by our external employees. Our company's standard has been IE8. We are planning on changing that and adopting IE 10 or 11.
Also, our external employees may use the application in their preferred browser (FF, Opera, Safari, diff. versions, etc.).
I know of services such as browershots, but these are mostly to check if layout is consistent across browsers and tell you nothing if the functionality is still working or not. Is there any other way to do an efficient preliminary check for compatibility with newer browsers instead of having to do a full out regression test? If checks fail, the regression tests can follow and then refactoring what is necessary.
We can use tools like Citrix AppDNA or modern.ie and IECT tool as a combination to explore the strategy and update the applications
I need to setup a testing infrastructure to test combinations of cookies enabled/disabled, Sliverlight and Flash plug-ins enabled/disabled, and javascript enabled/disabled. For starters, I need to test on IE6, IE7, IE8, IE9, Chrome, FF 3.6, FF 4.0, Safari 5 and Opera 10 on their most likely OS.
Longer term, I get as close as possible to testing on every "relevant" browser including mobile devices. I'll also want to include testing IE, for example, on XP, Vista and Win 7.
Even if I pick a single OS per browser, I have more than 100 host setups. What tools or strategies can bring sanity to this process?
(My current thinking suggests that screenshots alone will not be sufficient. It could be that I can develop a basic smoke test which gives me some indication of a problem and is verifiable through screenshot alone.)
Have you looked at Selenium, specifically Selenium Grid? It's not going to cover every single one of your test setups, but it'll get you a lot of the way there.
Selenium Grid transparently distribute [sic] your tests on multiple machines so that you can run your tests in parallel, cutting down the time required for running in-browser test suites. This will dramatically speeds up in-browser web testing, giving you quick and accurate feedback you can rely on to improve your web application.
See also:
Which is the most comprehensive browser compatibility testing tool?
How to automate testing of a browser-based app?
https://stackoverflow.com/questions/2565505/whats-the-best-way-to-do-cross-browser-testing
All that said:
It really might not be worth testing every single browser/version/OS combination. Part of this is because, at least on desktop OSes, most variations will show up across browsers (and to a lesser extent, browser version), and have much less to do with the OS version.
Chances are that the users of your site will not have equally distributed browsers/versions/OSes. It's basically a case of the 80-20 rule: the majority of your users will be using the minority of browser/version/OS combinations. So, at least in the short term, you get much more for your effort if you focus on ensuring that the site works on only the most popular OSes and browsers.
I'd also expect that your site, like many mobile-friendly sites, has a separate mobile view (reduced JavaScript, less content per page, simpler CSS, etc.). In that case, you'd only need to test the mobile version on mobile browsers and OSes, and the full version of the site on desktop browsers/OSes.
Please take a look at the following tools:
Automated Testing Tools – TestComplete
Automated Testing Tools for Web Application Testing | Telerik
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?
As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?
I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.
So no concerns there.
Jeroen van Menen
Lead dev WatiN
If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:
As of Friday, Google has over 50 teams
running over 51K tests per day on
internal Selenium Farm. 96% of these
tests are handled by Selenium RC and
the Farm machines correctly. The other
4% are partly due to RC bugs, partly
to test errors, but isolating the
cause can be difficult. Selenium has
been adopted as the primary technology
for functional testing of web
applications within Google. That's the
good news.
I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.
We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.
I've been trying 'em both out and here are my initial thoughts...
WatiN
The Good
Fast execution.
Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.
The Bad
Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
No native XPath support.
No out-of-the-box remote execution/grid based execution.
Script Example (C#). You can't do this with Selenium (not that I know off, at least):
class IEManager
{
IE _ie = null;
object _lock = new object();
IE GetInstance(string UrlFragment)
{
lock (_lock)
{
if (_ie == null)
{
var instances = new IECollection(true); //Find all existing IE instances
var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
_ie = match ?? new IE();
if (match==null) //we created a new instance, so we should clean it up when done!
_ie.AutoClose = true;
}
}
return _ie;
}
}
Selenium
Slower than WatiN (especially since a new process has to be created).
Built-in CSS selectors/XPath support.
Selenium IDE is good (can't say great, but it’s the best in class!).
Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
Support chrome and other browsers out of the box.
Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.
The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.
Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.
In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.
Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu
Update
Ye Oliver you're right. Ok why's it better?
Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.
Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.
Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,
browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");
... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.
There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara
I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.
The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.
Until now we are a pure Microsoft Shop for delivering solutions for the enterprise and went with WatiN. This may change in the future.
As a more recent source:
Microsoft printed in MSDN Magazine 12/2010 a BDD-Primer with the combination of SpecFlow with WatiN (cool BDD-Behavior Driven Development). Its author Brandon Satrom (msft Developer Evangelist) has also posted in December 2010 a Video Webcast teaching in detail 1:1 his above findings.
There is a Whitepaper from 04/2011 on Supporting ATDD/BDD with SpecLog, SpecFlow and Team Foundation Server (Acceptance Test Driven Development/Behavior Driven Development) from Christian Hassa, whose team built SpecFlow.
I use Watin, but haven't used Selenium. I can say I got up and running quickly on Watin and have had few to no problems. I can't think of anything I have wanted to do that I couldn't figure out with it. HTH
I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.
I recommend WebAii since that's what I've had any success with and when using it my gripes were few. I never tried Selenium and I don't remember using WaTiN much, at least not to the point where I could get it to succesfully work. I don't know of any framework that deals with Windows dialogs gracefully, although WebAii has an interface for implementing your own dialog handlers.
I considered using both. I used the recorder for Selenium to build some tests in FF. I tried to do the same in Watin and found that the Watin Recorder (2.0.9.1228) is completely worthless for our sites. It appeared to be rendering the site in IE6 -- making our site effectively unusable for recording. We don't support IE6. I couldn't find any way to change the browser it is using. I only found one Watin Recorder out there. If there's more than one, or one that is kept up to date, please comment.
The Selenium Recorder IDE for Firefox is simple to use and ports tests to C#. It isn't great at this. I couldn't get porting test suites to work, despite reading a blog post or two that had workarounds. So there's a bit of manipulation of the generated code. Still, it works 90% and that's better than the alternative.
For my money/time, Selenium is superior just for the ease of building new tests. IE doesn't have any good developer toolbars that are anywhere near as good as Firebug, so I'm doing my development in Firefox to begin with, so having a good working recorder in Firefox is a huge bonus.
My conclusion here was a lot like that democracy quote by Churchill: Selenium is the worst form of automated UI testing. Except for all the other ones.
At the risk of going off on a tangent, I'd recommend Axe/WatiN. Axe allows tests to be written in Excel by 'Manual' Testers with no knowledge of the underlying test 'language'. It does need a 'Technician' to write the bespoke actions (IE. Today I had to do a slightly complex Table lookup & cross-reference) but once written the actions can be used in tests by the non-techy testers.
I also heard that the UK Government Gateway project (which I believe has 6K+ tests automated tests) recently ported all their tests from Axe/Winrunner to Axe/Watin within a week!! And many of the tests are pretty complex - I know as I worked on it a few years ago...
I'm looking at Selenium at the moment, as a potential Client uses it. But i do suggest a wee look at Axe as a layer above the 'work horse' tool.
If you have to access iframes, modal dialogs and cross domain iframes WatiN is a way to go. Selenium couldn't handle the iframes it was throwing commandtimeout exceptions. WatiN you could do lot more things especially if the website uses IE specific stuff like ShowModalDialog etc.. WatiN handles all of them very well. I could even do cross domain iframe access.
You will have to do both if you need to do IE and FF testing, but they are only going to work so well for presentation testing. They cant detect if one element is slightly off, just that the elements are present. I dont know of anything that can replace the human eye for UI / presentation testing, though you could do a few things to assist it (take screenshots of the pages at each step for users to review).