Please suggest which tool should I use - QTP, RIATest, Selenium for Flex Application Functional Automation Testing. Please help me compare them with different advantages and disadvantages.
Any help appreciated.
The big picture is:
QTP - sophisticated, expensive, can automate almost anything, not just Flex applications, works on Windows.
RIATest - simple, inexpensive, designed specifically and only for Flex application, works on Windows and Mac.
Sorry, no experience with Selenium.
I don't have direct experience with it, but there is a Selenium Flex API available; since Selenium is free and open source, it's a great option if your budget is tight.
I do have experience with Selenium in general and can highly recommend it; it's relatively easy to learn and works on any platform with a multiplicity of browsers.
Recently I evaluated the above tools for our Orgnisation.
I say go with QTP. The configuring of qtp for flex is easy.
I wrote a hub about it.
http://hubpages.com/hub/Automating-Flex-Applications-with-QTP
Please ask questions if you have any.
Follow me n my hub
QTP does work with Flex apps...just have to have the right versions.
I was using Selenium to test Flex for a while, then Sun came out with a new plugin for Java in the browser with 1.6.14 I think it was, and I wasn't able to get it to work after that, though I admit its been quite a while since I tried.
Related
I've just discovered TideSDK and it seems to be a really great tool, but I have one requirement : I need to use some native code (for managing USB devices for example) and so I need communication between this native code and the web app, is such a thing possible with TideSDK?
Yes, working with native code in TideSDK is possible. Our SDK is modular and we have been reorganizing the code structurally to make it easier to do the sort of thing you want. At a modular level, you will be contending with support for multiple platforms typically.
A module should extend to all platforms that you are supporting. We expect to have documentation to help developers (familiar with native code) to better understand the SDK. This should include some module boilerplate to help you get started. At this point, we have yet to prepare this more detailed documentation. We have much to do and sometimes progress seems slow despite all the great efforts going into TideSDK.
TideSDK is a large and complex SDK but don't let this frighten you off. It is extensible and we will be shining light on this soon with module development guides. It would be cool to talk more on IRC about this with you so feel free to drop by at any time. Perhaps the functionality you are speaking of is of general use ie. to extend the APIs for everyone.
There are possibilities to work together with the core developers of TideSDK on modules and to contribute to this great open source project. Other possibilities also include sponsoring module development if this something that you need more immediately for a project. Hope this helps.
We are in search of an automated testing tool for our project. As we are in testing department we prefer a tool which would have less programming in it. Please suggest some tools for us .Till now we are testing our application manually.
Our project is being developed in Java.
Is there any freeware tool that I could use or is it better to go for a paid tool?
Thanks in Advance.
Less programming? You'll need something like JUnit to write unit tests if you want to do serious regression testing, but unit tests require you to write some code
Here's a big list of open-source testing tools, some of them may offer what you want: http://java-source.net/open-source/testing-tools/junit
For example, T2 claims to be a random testing tool. As one, it is fully automatic, but one must keep in mind that the code coverage of random testing is in general very limited. It should be used as a complement to other testing methods. T2 checks for internal errors, run time exceptions, method specifications, and class invariant.
Not sure if you mean a CI tool or not, but we use Hudson at Zappos and it works pretty well.
http://hudson-ci.org/
..and there's also CruiseControl: http://cruisecontrol.sourceforge.net/
If you're not talking about CI, maybe you mean QA testing - in which case you should take a look at something like Selenium (for web apps):
http://seleniumhq.org/
If you're doing GUI testing? I'm not really familiar with that area, but I've heard about WinRunner and Rational:
http://en.wikipedia.org/wiki/HP_WinRunner
http://www-01.ibm.com/software/rational/offerings/quality/
..though neither are really free tools. Something like AutoIT might help you move widgets around, but it lacks the reporting parts:
http://www.autoitscript.com/autoit3/index.shtml
There could be two answer to you question:
Besides Selenium, though it has ample of advantages, I am reading about another tool which uses same API which Selenium use. The only changes in API I have seen so far is it reduces the complexity of functions thus making it more easier and simpler for user who is learning.
The tool is called 'Helium' and it has 50% (and more) less complex functions and code as Selenium has.
The only problem with this tool is it is paid tool for learning purpose and for implementing not-so-big scale project you can use it. But yeah after some time its gonna cost you.
I have implemented some code on Helium. Please let me know , if you face any issue initially or you are thinking to implement it.
Other being, you can use Selenium Builder(http://khyatisehgal.wordpress.com/2014/05/26/selenium-builder-exporting-and-execution/) which is an advanced form of Selenium IDE. It imports your command in different languages and does work more effectively and efficiently as Selenium IDE does(http://khyatisehgal.wordpress.com/2014/05/25/selenium-builder/) . So you can import scripts in Eclipse IDE and just execute them as is.
Please let me know , if you have any doubt in any of the tool.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?
As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?
I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.
So no concerns there.
Jeroen van Menen
Lead dev WatiN
If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:
As of Friday, Google has over 50 teams
running over 51K tests per day on
internal Selenium Farm. 96% of these
tests are handled by Selenium RC and
the Farm machines correctly. The other
4% are partly due to RC bugs, partly
to test errors, but isolating the
cause can be difficult. Selenium has
been adopted as the primary technology
for functional testing of web
applications within Google. That's the
good news.
I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.
We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.
I've been trying 'em both out and here are my initial thoughts...
WatiN
The Good
Fast execution.
Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.
The Bad
Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
No native XPath support.
No out-of-the-box remote execution/grid based execution.
Script Example (C#). You can't do this with Selenium (not that I know off, at least):
class IEManager
{
IE _ie = null;
object _lock = new object();
IE GetInstance(string UrlFragment)
{
lock (_lock)
{
if (_ie == null)
{
var instances = new IECollection(true); //Find all existing IE instances
var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
_ie = match ?? new IE();
if (match==null) //we created a new instance, so we should clean it up when done!
_ie.AutoClose = true;
}
}
return _ie;
}
}
Selenium
Slower than WatiN (especially since a new process has to be created).
Built-in CSS selectors/XPath support.
Selenium IDE is good (can't say great, but it’s the best in class!).
Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
Support chrome and other browsers out of the box.
Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.
The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.
Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.
In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.
Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu
Update
Ye Oliver you're right. Ok why's it better?
Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.
Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.
Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,
browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");
... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.
There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara
I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.
The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.
Until now we are a pure Microsoft Shop for delivering solutions for the enterprise and went with WatiN. This may change in the future.
As a more recent source:
Microsoft printed in MSDN Magazine 12/2010 a BDD-Primer with the combination of SpecFlow with WatiN (cool BDD-Behavior Driven Development). Its author Brandon Satrom (msft Developer Evangelist) has also posted in December 2010 a Video Webcast teaching in detail 1:1 his above findings.
There is a Whitepaper from 04/2011 on Supporting ATDD/BDD with SpecLog, SpecFlow and Team Foundation Server (Acceptance Test Driven Development/Behavior Driven Development) from Christian Hassa, whose team built SpecFlow.
I use Watin, but haven't used Selenium. I can say I got up and running quickly on Watin and have had few to no problems. I can't think of anything I have wanted to do that I couldn't figure out with it. HTH
I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.
I recommend WebAii since that's what I've had any success with and when using it my gripes were few. I never tried Selenium and I don't remember using WaTiN much, at least not to the point where I could get it to succesfully work. I don't know of any framework that deals with Windows dialogs gracefully, although WebAii has an interface for implementing your own dialog handlers.
I considered using both. I used the recorder for Selenium to build some tests in FF. I tried to do the same in Watin and found that the Watin Recorder (2.0.9.1228) is completely worthless for our sites. It appeared to be rendering the site in IE6 -- making our site effectively unusable for recording. We don't support IE6. I couldn't find any way to change the browser it is using. I only found one Watin Recorder out there. If there's more than one, or one that is kept up to date, please comment.
The Selenium Recorder IDE for Firefox is simple to use and ports tests to C#. It isn't great at this. I couldn't get porting test suites to work, despite reading a blog post or two that had workarounds. So there's a bit of manipulation of the generated code. Still, it works 90% and that's better than the alternative.
For my money/time, Selenium is superior just for the ease of building new tests. IE doesn't have any good developer toolbars that are anywhere near as good as Firebug, so I'm doing my development in Firefox to begin with, so having a good working recorder in Firefox is a huge bonus.
My conclusion here was a lot like that democracy quote by Churchill: Selenium is the worst form of automated UI testing. Except for all the other ones.
At the risk of going off on a tangent, I'd recommend Axe/WatiN. Axe allows tests to be written in Excel by 'Manual' Testers with no knowledge of the underlying test 'language'. It does need a 'Technician' to write the bespoke actions (IE. Today I had to do a slightly complex Table lookup & cross-reference) but once written the actions can be used in tests by the non-techy testers.
I also heard that the UK Government Gateway project (which I believe has 6K+ tests automated tests) recently ported all their tests from Axe/Winrunner to Axe/Watin within a week!! And many of the tests are pretty complex - I know as I worked on it a few years ago...
I'm looking at Selenium at the moment, as a potential Client uses it. But i do suggest a wee look at Axe as a layer above the 'work horse' tool.
If you have to access iframes, modal dialogs and cross domain iframes WatiN is a way to go. Selenium couldn't handle the iframes it was throwing commandtimeout exceptions. WatiN you could do lot more things especially if the website uses IE specific stuff like ShowModalDialog etc.. WatiN handles all of them very well. I could even do cross domain iframe access.
You will have to do both if you need to do IE and FF testing, but they are only going to work so well for presentation testing. They cant detect if one element is slightly off, just that the elements are present. I dont know of anything that can replace the human eye for UI / presentation testing, though you could do a few things to assist it (take screenshots of the pages at each step for users to review).
What tools, preferably open source, are recommended for driving an automated test suite on a FLEX based web application? The same tool also having built in capabilities to drive Web Services would be nice.
Adobe distributes a test framework themselves: FlexUnit.
I heard of people using selenium as a free/open source testing tool. A quick google revealed a FLEX API for it. Not sure if it works or is still in development, but it may be worth a look.
http://sourceforge.net/projects/seleniumflexapi/
Are you looking to script code-level unit tests? If so, dpuint is the bomb: http://code.google.com/p/dpuint/ . This library makes it really easy to do automated testing on all sorts of asynchronous events, on either non-visual ActionScript objects or visual components. They also have a nice multi-page tutorial on the Google Code project page.
If you are looking for functional testing tools along the lines of automated record-and-playback simulating an end user using a Flex app, HP's QuickTest Pro is the Adobe-endorsed solution. It works great, but costs about $4,000 - $6,000 per seat.
Check out FlexMonkey. It does automated testing via FlexUnit tests.
Try looking at Melomel. It has Cucumber support baked right in and comes packaged with steps for most Halo and Spark components.
http://melomel.info
There's an automated test tool called RIATest that might fit the bill for you.
Unfortunately only for Windows, and not open source, but if it does the job it might be well worth the price ($399 at time of writing).
FunFX is an option for automating UI testing. I haven't used it extensively, but I've heard of some having success with it. Here is the article where I first learned about it.
I've been extensively using FunFX for several months now on a Flex 3 + Rails project. Not only is it open source, it's also written in Ruby, so integration with web services should be fairly easy. There are a few screencasts out there covering the basics.
The Flex code that your Flex app needs is contained in the SeleniumFlexAPI distribution .swc file, SeleniumFlexAPI.swc. Just include this file as a library when you compile your Flex app.
Sikuli is good tool which can be used to test flex/flash based web applications.
-It can automate anything on graphical user interface.
-It works on Windows, MAC OSX and Linux as well as iPhone and Android.
-Here is the Sikuli link
My preferred tool is Selenium Remote Control. There is a plug-in I discovered a few months ago:
http://code.google.com/p/flash-selenium/
This required 'hooks' to be written on the server side (ActionScript/Flex). Once they were added, I was able to do some browser testing using Selenium RC.
FunFX is great. We've used it extensively and have been very happy with it. The community is also active and very responsive, so that is a big plus for me.
The new version of the Selenium-Flex API (0.2.5) works great.
Is it realistic to try and learn and code a Flex 3 application without purchasing FlexBuilder? Since the SDK and BlazeDS are open source, it seems technically possible to develop without Flex Builder, but how realistic is it.
I would like to test out Flex but don't want to get into a situation where I am dependent on the purchase of FlexBuilder (at least not until I am confident and competent enough with the technology to recommend purchase to my employer).
I am experimenting right now, so I'm taking a long time and the trial license on my Windows machine has expired. Also Linux is my primary development platform and there is only an alpha available for Linux.
Most of the documentation I've found seem to use Flex Builder.
Maybe I should use Laszlo...
IntelliJ IDEA works as a Flex IDE, if you happen to also be a Java developer. It's free if you contribute to open source projects.
Check out FlashDevelop for Windows. I like it better than Flex Builder.
I've been using Flex since version 2 and Flex3/BlazeDS since it came out of beta. I also have some experience with Lazzlo and the difference is day and night (Flex rocks!). I have not regretted once using Flex. Regarding FlexBuilder, it is worth every penny. While it is completely possible and reasonable to write Flex application without FlexBuilder, the productivity gains of using it will more than recoup the investment. Try the evaluation for 30 days and compare it to some of the other options suggested about (I'm going to try FlashDevelop).
Some things you get with FlexBuilder include:
Code completion
Visual editor
Debugger (it is fantastic!!)
Profiler (also very good)
Regarding Linux, the alpha version of FlexBuilder does not have a visual editor. Other than that, I understand it is reasonably feature complete, still free, and many of the Adobe employees I've talked with that use Linux are happy with it.
FlashDevelop is really easy to setup with the Flex SDK. Just download FlashDevelop, then download the Flex SDK. In FlashDevelop go to Tools > Program Options > AS3Context (under Plugins) > Set the "Flex SDK Location" to the root of the folder you extracted the SDK to and build away. FlashDevelop even has a basic MXML project that will get you going.
If you use ColdFusion for the backend, having FlexBuilder in Eclipse and CFEclipse can mean one less IDE to have to get familiar with.
I'm going to join the choir here and say FlashDevelop for an alternative. The only reasons you might want FlexBuilder are:
Flex charts
Step-through debugging.
Profiler (I haven't used it)
Visual style editor
However, the code-completion and general bloody-awesomeness of FlashDevelop's code-completion and syntax highlighting knocks the gimpy eclipse crap out of the water. So, pretty much what Todd said, except for the code-completion part. Flex Builder is very flakey in that department.
Short answer: Yes
I'm working on a team of developers and designers. We code our .MXML and .AS in FlashDevelop 3 and our designer creates .FLA with skins and widgets that get [Import()]ed in ActionScript.
I wrote a little more about this subject here:
Flash designer/coder collaboration best practices
I have been using FlashDevelop for along time (4/5 years), I am actively using it to develop Flex4.5 applications, it has built in support for code completion, it has a profiler and a debugger that work excellently. The IDE itself is responsive and require the .Net framework, in fact here, I'll list some stuff.
FlashDevelop Pros
Free IDE
Code completion feature
Very capable Debugger
Profiler
Documenting
Ability to build Air / Flex files
Templating
Plugins
FlashDevelop Cons
Lack of UI desing support
.Net support only (Won't work with Mono)
Everything else is pretty simple to get running with, the instructions are available at http://www.flashdevelop.org/
Absolutely. I've been a Flex developer since Flex 2 and until recently I've used my regular editor, TextMate, for coding and Ant for building. TextMate has some good extensions for ActionScript and Flex coding, but I think you could get that for any decent editor.
What's been missing from my setup is a usable debugger, the command line version is a pain to work with. Because of that I've been starting to use FlexBuilder on the side, using it in parallel with my regular setup.
Having a profiler doesn't hurt too.
I've been using FlexBuilder for awhile now and just started to switch to using Eclipse with Flex SDK. I work for a non-profit so the word FREE is huge.
Initially, it is fairly intimidating so if you have the money, you might want FlexBuilder.
There is a lot you need to know and do if you use the SDK. The learning and experience may pay off though... I am still undecided myself.
I second FlashDevelop. You don't get the visual design stuff for the MXML, but for the code (both MXML and AS) it's excellent.
I also use FlashDevelop when working on AS3 projects. For me, the ugliness (UI design) and sluggishness of Eclipse/Flex Builder is enough of a deterrent to stay away from Flex Builder.
In addition to the weaknesses of FlashDevelop pointed out previously, one of my biggest gripes is that it is not a true .NET only app and therefore will never work in mono and therefore can not be easily ported to the mac - which is my platform of choice for development web/javascript/AS3 development.
Amethyst is also a pretty good option to try. It is a plugin for MS Visual Studio, and takes advantage of a lot of the goodies there. It is significantly less sluggish than FlashBuilder, has a really good debugger, and a decent visual designer as well.
The personal version is free, but quite crippled. You have to buy the pro version after a 60 day free trial. However, it is (at time of writing) almost 1/3 the cost of Flash Builder.
As an added bonus you don't need to pay for Visual Studio since it works with the free (albeit hard to find) "shell version (integrated)" of Visual Studio. It won't work with any of the free Express editions, though.