I have to automate a Flex based rich internet application. We have tried out a few options like Ranorex and RIA Test. While the GUI based automation tools (both open source and commercial) do a decent job, the test scripts are hard to maintain and often result in flaky reports. PhantomJS has worked for me in previous projects very well but I am not sure if it works with Flex. Can anyone suggest if PhantomJS works at all with Flex?
I don't know anything about flex, but the tag excerpt suggests that it extensively uses Adobe Flash and Adobe Air. PhantomJS doesn't support Flash or other plugins (anymore). There are forks which enable flash, but they are a little behind with the versions.
Honestly i think this might have as much to do with how you build the automation tests that makes them hard to maintain. For instance with Ranorex instead of building 5 different recordings to test 5 different things build 15 tiny recordings that do only one thing, that can be pieced together in 5 different ways, then to maintain your scripts you just need to maintain those 15 recordings, and if all the scripts break at the same place the maintenance simply becomes ok why did that one tiny recording fail rather than ok let me maintain this suite of 5 recordings it's let me fix this one recording, and that will fix the larger recordings.
Try and use Testcomplete identifies most of the complex Flex controls not need for any compilation of helpers with your application libraries and if you have a team of dedicated automation engineers they should easily be able to create a reusable and maintainable GUI based automation pack around Testcomplete (vbscript)
Related
Suppose I develop an application with the logic as a CLI program that can be compiled to various platforms. To add a GUI, I could develop native versions separately for each platform or I could use a cross-platform tool like Tcl/Tk, wxLua, or PyGTK. Some people, like Bruce Eckel and James Ward, promote Flash/Flex on Adobe AIR for this.
I've looked into it, but haven't been able to find clear answers to these basic issues.
Native look and feel
The Spark architecture in Flex 4 improves support for skinning to the extent that you can match any L&F, it seems. What resources are available for implementing Windows/GTK+/Cocoa L&F and how much work is it?
Responsiveness
I would rank these popular toolkits from most to least snappy: FLTK, FOX, GTK+, Tcl/TK, Swing, Qt. Where does AIR fit in this range?
Stability
For a long-running app with extensive user interaction, how are the stability and memory management? I've read that memory leaks have been a significant issue with AIR.
Local system access
I think the biggest complaint about AIR 1 was its isolation from the local system. Now, AIR 2 adds communication via stdin/stdout with the NativeProcess class. How well is this working out for implementing the functionality you typically want in a GUI?
Package size
Ranging from a minor interface with a few buttons and textboxes up to something like an IDE, how big would an AIR version be, compared to say, something in Tcl/Tk or wxPython?
Hand coding
If you don't use Flash Builder, how hard is it, relatively, to create or maintain a nontrivial project?
Native look and feel: It's possible to skin a Flex app to make it look native. And load a different skin depending on the OS. There are already Windows and Mac skins (not sure if they are Flex 4 compatible yet) available on www.scalenine.com. But these skins will never be a 100% match since it's replicating the l&f, not actually using native widgets.
Responsiveness: I'd guess it's probably somewhere between Swing and GTK+. But I don't have benchmarks to back that up - just my personal experience.
Stability: As long as the code is good and objects are being GC'd then things should be fine. AIR 2 fixed some memory leaks that caused the issues you reference.
Local system access: NativeProcess allows you to do pretty much anything. But more complex use cases probably need something like Merapi.
Package size: The entire UI component set in Flex is around 2MB. Then any custom code will add to that number. Usually the bitmap assets are what really bloat SWF files. You can look at Apparat as a way to reduce the size of SWF files.
Hand coding: I hand code all of the time with just the open source Flex SDK, AIR SDK, and vim (my text editor of choice). That's all you really need. But you will lose out on code completion, visual debugging, etc. But there are some other open source tools out there for doing these things.
For an internship, I'm gonna have to develop a desktop application. The focus is creating a rich UI ( cool effects, sound etc ). Which tech should I go with ?
- Flash ? ( in this case, shoud I go with a flex project ? AIR ? what is the gain between this and a simple raw flash project )
- C#/Silverlight ?
Any other possibility ?
If experienced developers could give their advice, that would be great thanks.
The best technology to use is probably the one you're most familiar with, unless you mean you're going to be learning on the fly. But for the various options I'll take a stab.
Flash itself (the runtime) is a browser plugin, so that's not what you want, but if you build your application in Flash or Flex (the authoring tools), then you can publish it as an AIR application. AIR is also a runtime - basically it's Flash, plus a bunch of extra APIs for running outside the browser. The advantages to making an AIR app are:
The same app runs on win/mac/linux
You can make it with the same skills/tools you use for Flash (dubious utility if you don't have those skills/tools, of course)
Close tool integration with other Adobe tools (photoshop, after effects, etc.)
The main disadvantage is incomplete access to native (i.e. OS) capabilities. The way a runtime allows you to run the same app on win/mac/linux is by restricting you from doing some of the things that you could do with a native application.
As for Flex, whether you use Flex or not is a separate question. Flex is a framework within Flash - an AIR app can be made using Flex or not using it. Basically, if you expect your app to have a lot of controls, slider bars, charts, accordion panels, etc., you might look into Flex. If you expect it to have a lot of animations and pictures and sounds, plain-old-Flash may be better.
Apart from Flash, the other main choice for a runtime is Silverlight. The advantages there are, to be honest, basically the same. You use .NET instead of ActionScript, and the tooling integration is with MS' Expression suite instead of Adobe's creative suite, but the development process and the results are quite similar. The choice between AIR and Silverlight is really a choice between Adobe tools and MS tools, and between two programming languages of entirely comparable power. Most everyone simply uses the one they're more familiar with.
Finally, the other set of options is not to use a runtime, and to simply compile an app from any language/development environment you like - C, C++, .NET, Python, anything. The advantage here is you get (in general) full access to OS capabilities. The disadvantages are, you (in general) need to do extra development to run on multiple platforms, and also I don't think you'll find a language/environment that lets you work with multimedia as easily as you can in AIR and Silverlight.
Depends what programming languages are you proficient in. If you are .NET/C#/VB.NET guy go with SilverLight or WPF. If you know ActionScript, go with Adobe line of products.
My preference goes to the (slightly) most opened universe. Assumed that you know how to write Object-oriented programs, Adobe is using Eclipse as a base for his tools, using plugins.
Actionscript is just like Javascript syntax. You just have to learn the flex 4 libraries.
Then, publish you app as an Air desktop application that can run everywhere. Really everywhere
The only thing to know about MS tools and languages and libraries is that they are very well designed and usable. and it runs on Win and mac also. and Only.
If the main request is to have a rich UI the simplest approach is to build a Flex application running under Adobe AIR platform. The answer to the questions Flash or Flex is very simple: if you are a designer go with Flash, if you are a developer stay away from Flash and use Flex. The Flex framework was created for developers, and is really hard to replicate all its features in Flash (think only what you have to do in Flash if you want to resize your app).
On the other hand if you are coming from a .NET background it will make more sense to stick with Silverlight/C#
I'm not sure if this helps but I'm developing in Flex and now I have started to learn Silverlight and I can say that developing in Flex Builder is much more faster then developing in Silverlight( Visual Studios ) . For me Flex Builder is much better because you use only one IDE to develop both back and front end. And that's when it comes to web development. If you want to do desktop applications then you do the same thing and a plus is that you get everything in only one file. You also have a small SQL database that comes with it.
But on the other hand when you develop in Silverlight you have to use several IDE's or tools outside Visual Studios to make awesome graphics. But if you want to have more time for design then I would suggest to use Flex Builder and not Silverlight.
I work in a company which is primarily concerned with desktop apps not served over the internet.
Part of my value is I have a web based background and proficiencies in ASP.NET, JavaScript, JQuery etc.
The issue I'm having is that compared to a traditional desktop application building a rich web based app is more time consuming. This is understandable in terms of the hoops that need to be jumped through for web development. This is a cause of frustration for those not familiar with the ways of the web.
However because people are used to the non web world where I work I need to utilise every possible tool and technique to be productive as possible while building web based applications.
As such I'm looking for what other people do to be as productive as possible while making web based applications. I'm primarily thinking of ASP.NET (not using MVC) but apart from that everything is open.
The best tools for web development, in my opinion, are:
.Net Reflector
Firebug
IE Developer Toolbar
IE Tester and MyDebugBar
Fiddler
You should also look into NHibernate as an option.
The best choice of tools tends to vary quite a bit from one developer to another. It's also very dependent on the details of the type of application you're building.
For me, Visual Studio Team Suite is indispensable. It has a rich set of visual designers and other features that have a big impact on my productivity. Deep integration with bug reporting and source control is another huge time saver (I use Team Foundation Server).
With web forms, you can of course often use drag-and-drop components, which improves productivity for some people (although not me). I can't say I'm a fan of most third-party components. They can be quirky, take a long time to learn well, and then still not do exactly what I want them to. It's also a skill that often isn't portable from one job to the next.
You can leverage jQuery to help simplify scripting -- although even with jQuery I find scripting to be one of the most time-consuming and error prone aspects of web development.
If it's suitable for your environment, you might explore Silverlight. That way, you can often have the best of both worlds, building desktop-like apps using desktop-like tools, but with web-based deployment (Expression Blend is an awesome tool). You can also use Silverlight as a replacement for JavaScript in many cases, with code that's compiled and type-safe.
Good tools on the data side are critical, such as SQL Server Profiler. Visual Studio Team Data (part of Team Suite) is invaluable for it's data generation, unit testing and deployment and management aspects.
Something very underrated by many web developers is building up an appropriate infrastructure: things like logging and performance counters, that can help you track down problems quickly when they occur. A solid configuration system is also important.
In case it helps, I put together a longer list of tools. I also write about something I call the "ultra-fast approach" in my book, which is about more than just building web sites that run fast; it's also about how to build them quickly and reliably: Ultra-Fast ASP.NET.
You should use third party tools.
Telerik RAD Controls and the AJAXToolkit are two such tools available to you. They make much of the 'fancy' stuff built in.
Then, and probably preferably, use JQuery.
With those three options, you can make a fancy web application in the quickest way possible.
Finally be sure your UI and business logic is properly segregated so you can reuse your business logic easier.
There are several ways to increase productivity while building web applications, and they're not too different from desktop programming.
Code re-use. As a shop that is primarily desktop-based, this may not be an immediate option. But as you build your code-base, you'll find that you can use some of the code over for new applications, and even begin building your own customized frameworks.
Development tool knowledge. If a lot of the tools are similar, then you may see immediate benefit from this. If not, it may take more time as the team gets used to new IDEs, etc.
Third-party tools. UI toolkits, backend frameworks. These exist in both worlds, and could be utilized to help jump start development, until items 1 and 2 can be worked in.
A few must-have tools on the client side (i.e. the browser)
FireBug - let's you debug your javascript, DOM, headers, etc
WebDeveloper - provides all sorts of information about the rendering of the page
Selenium - Web UI functional testing framework + tool
(all are firefox add-ons)
Here's some of the tools I use for IE and FireFox, http://www.learn-ajax.com/blog/2008/04/debugging-tools-for-web-applications_03.aspx . Chrome, Opera and Safari also have their own built in client-side web development tools as well.
Most important tools for ANY productive development
Dual large screen monitors
keyboard and mouse that works for YOU.
Comfortable high-backed chair that will save you taking 2 weeks off for physiotherapy down the road.
We're in the initial stages of a large project, and have decided that some form of automated UI testing is likely going to be useful for us, but have not yet sorted out exactly how this is going to work...
The primary goal is to automate a basic install and run-through of the app, so if a developer causes a major breakage (eg: app won't install, network won't connect, window won't display, etc) the testers don't have to waste their time (and get annoyed by) installing and configuring a broken build
A secondary goal is to help testers when dealing with repetitive tasks.
My question is: Who should create these kinds of tests? The implicit assumption in our team has been that the testers will do it, but everything I've read on the net always seems to imply that the developers will create them, as a kind of 'extended unit test'.
Some thoughts:
The developers seem to be in a much better position to do this, given that they know control ID's, classes, etc, and have a much better picture of how the app is working
The testers have the advantage of NOT knowing how the app is working, and hence can produce tests which may be much more useful
I've written some initial scripts using IronRuby and White. This has worked really well, and is powerful enough to do literally anything, but then you need to be able to write code to write the UI tests
All of the automated UI test tools we've tried (TestComplete, etc) seem to be incredibly complex and fragile, and while the testers can use them, it takes them about 100 times longer and they're constantly running into "accidental complexity" caused by the UI test tools.
Our testers can't code, and while they're plenty smart, all I got were funny looks when I suggested that testers could potentially write simple ruby scripts (even though said scripts are about 100x easier to read and write than the mangled mess of buttons and datagrids that seems to be the standard for automated UI test tools).
I'd really appreciate any feedback from others who have tried UI automation in a team of both developers and testers. Who did what, and did it work well? Thanks in advance!
Edit: The application in question is a C# WPF "rich client" application which connects to a server using WCF
Ideally it should really be QA who end up writing the tests. The problem with using a programmatic solution is the learning curve involved in getting the QA people up to speed with using the tool. Developers can certainly help with this learning curve and help the process by mentoring, but it still takes time and is a drag on development.
The alternative is to use a simple GUI tool which backs a language (and data scripts) and enables QA to build scripts visually, delving into the finer details of the language only when really necessary - development can also get involved here also.
The most successful attempts I've seen have definitely been with the latter, but setting this up is the hard part. Selenium has worked well for simple web applications and simple threads through the application. JMeter also (for scripted web conversations for web services) has worked well... Another option which is that of in house built test harness - a simple tool over the top of a scripting language (Groovy, Python, Ruby) that allows QA to put test data into the application either via a GUI or via data files. The data files can be simple properties files, or in more complex cases structured (something like YAML or even Excel) data files. That way they can build the basic smoke tests to start, and later expand that into various scenario driven tests.
Finally... I think rich client apps are way more difficult to test in this way, but it depends on the nature of the language and the tools available to you...
In my experience, testers who can code will switch jobs for a pay raise as developers.
I agree with you on the automated UI testing tools. Every place I've worked that was rich enough to afford WinRunner or LoadRunner couldn't afford the staff to actually use it. The prices may have changed, but back then, these were in the high 5-digit to low 6-digit price tags (think of the price of a starter home). The products were hard to use, and were usually kept uninstalled in a locked cabinet because everyone was afraid of getting in trouble for breaking them.
I worked over 7 years as an application developer before I finally switched to testing and test automation. Testing is much more challenging than coding, and any automation developer who wants to succeed should master testing skills.
Some time ago I put my thoughts on skill matrices in a couple of blog posts.
If interested to discuss:
http://automation-beyond.com/2009/05/28/qa-automation-skill-matrices/
Thanks.
I think having the developers write the tests will be of the most use. That way, you can get "breakage checking" throughout your dev cycle, not just at the end. If you do nightly automated builds, you can catch and fix bugs when they're small, before they grow into huge, mean, man-eating bugs.
What about the testers proposing the tests, and the developers actually writing it ?
I believe at first it largely depends on the tools you use.
Our company currently uses Selenium (We're a Java shop).
The Selenium IDE (which records actions in Firefox) works OK, but developers need to manually correct mistakes it makes against our webapps, so it's not really appropriate for QA to write tests with.
One thing I tried in the past (with some success), was to write library functions as wrappers for Selenium functions. They read as plain english:
selenium.clickButton("Button Text")
...but behind the scenes check for proper layout and tags on the button, has an id etc.
Unfortunately this required a lot of set up to allow easy writing of tests.
I recently became aware of a tool called Twist (from Thoughtworks, built on the Eclipse engine), which is a wrapper for Selenium, allowing plain English style tests to be written. I am hoping to be able to supply this to the testers, who can write simple assertions in plain English!
It automatically creates stubs for new assertions too, so the testers could write the tests, and pass them to developers if they need new code.
I've found the most reasonable option is to have enough specs such that the QA folks can stub out the test, basically figure out what they want to test at each 'screen' or on each component, and stub those out. The stubs should be named such that they're very descriptive as to what they're testing. This also offers a way to crystalize functional requirements. In fact, doing the requirements in this fashion are particularly easy, and help non-technical people really work through the muddy waters of their own though process.
The stubs can be filled in via a combination of QA/dev people. This allows you to CHEAPLY train QA people as to how to write tests, and they typically slurp it up as it furthers their job security.
I think it depends mostly on the skill level of your test team, the tools available, and the team culture with respect to how developers and testers interact with each other. My current situation is that we have a relatively technical test team. All testers are expected to have development skills. In our case, testers write UI Automation. If your test team doesn't have those skills they will not be set up for success. In that case, it may be best for developers to write you UI automation.
Other factors to consider:
What other testing tasks are on the testers' plate?
Who are your customers and what are their expectations related to quality?
What is the skill level of the development team and what is their willingness to take on test automation work?
-Ron
What tools, preferably open source, are recommended for driving an automated test suite on a FLEX based web application? The same tool also having built in capabilities to drive Web Services would be nice.
Adobe distributes a test framework themselves: FlexUnit.
I heard of people using selenium as a free/open source testing tool. A quick google revealed a FLEX API for it. Not sure if it works or is still in development, but it may be worth a look.
http://sourceforge.net/projects/seleniumflexapi/
Are you looking to script code-level unit tests? If so, dpuint is the bomb: http://code.google.com/p/dpuint/ . This library makes it really easy to do automated testing on all sorts of asynchronous events, on either non-visual ActionScript objects or visual components. They also have a nice multi-page tutorial on the Google Code project page.
If you are looking for functional testing tools along the lines of automated record-and-playback simulating an end user using a Flex app, HP's QuickTest Pro is the Adobe-endorsed solution. It works great, but costs about $4,000 - $6,000 per seat.
Check out FlexMonkey. It does automated testing via FlexUnit tests.
Try looking at Melomel. It has Cucumber support baked right in and comes packaged with steps for most Halo and Spark components.
http://melomel.info
There's an automated test tool called RIATest that might fit the bill for you.
Unfortunately only for Windows, and not open source, but if it does the job it might be well worth the price ($399 at time of writing).
FunFX is an option for automating UI testing. I haven't used it extensively, but I've heard of some having success with it. Here is the article where I first learned about it.
I've been extensively using FunFX for several months now on a Flex 3 + Rails project. Not only is it open source, it's also written in Ruby, so integration with web services should be fairly easy. There are a few screencasts out there covering the basics.
The Flex code that your Flex app needs is contained in the SeleniumFlexAPI distribution .swc file, SeleniumFlexAPI.swc. Just include this file as a library when you compile your Flex app.
Sikuli is good tool which can be used to test flex/flash based web applications.
-It can automate anything on graphical user interface.
-It works on Windows, MAC OSX and Linux as well as iPhone and Android.
-Here is the Sikuli link
My preferred tool is Selenium Remote Control. There is a plug-in I discovered a few months ago:
http://code.google.com/p/flash-selenium/
This required 'hooks' to be written on the server side (ActionScript/Flex). Once they were added, I was able to do some browser testing using Selenium RC.
FunFX is great. We've used it extensively and have been very happy with it. The community is also active and very responsive, so that is a big plus for me.
The new version of the Selenium-Flex API (0.2.5) works great.