I am looking for recommendations for tools for automated testing of a web application with some flex components.
To provide some background we have a web application that was entirely developed in AJAX+HTML and we were somewhat successful in using Selenium for testing that application end to end. We recently added some flex components into the mix and it got complicated.
We tried using Selenium Flex but we are disappointed with what it can do. So now we are looking for some alternatives. Ideally the tool would be able to drive both the web and the flex parts simultaneously, but we can also settle for just testing the flex components on their own. We prefer open source but good commercial tool is also an option.
I've heard about Fluint and FlexMonkey but haven't tried them yet. Anybody has any experience with using those? Any advice for someone who is just starting? Any gotchas?
Are there some other tools that could be used?
Thanks.
i'v try FlexMonkey.
It was great, until he get lost in the recording of interaction with huge and complex Flex Component.
For me, and for now, it's great in the demo or with small app, but not ready for production.
BUT : seems to evolve quickly.
I've had much success using FunFX to automate a fairly complex Flex application. FunFX is built off FireWatir, which is another web automation framework similar to Selenium. So together that should cover the entire AJAX/HTML + Flex bits of your application.
Since both FunFX and FireWatir are Ruby-based, I also recommend something like Cucumber as your testing framework to glue everything together.
We've had some success using Fluint for unit testing and Quick Test Professional (with the Flex plug-in) for functional end-to-end testing. QTP is pretty heavy-weight, but once you've got it set up and have created a few tests, it works pretty well. It works with Flex as well as vanilla HTML/Javascript, so even if your app is a mix of the two, it should be able to handle it.
Fluint works great for unit testing, but given the amount of code required to test a component, I wouldn't advocate it for functional testing (it's too low-level). FlexMonkey is more of an automation framework for Flex Unit. You basically use it to record test cases, and it generates code using Flex Unit to perform the test. I've tinkered with it, but ultimately decided to use QTP.
As for tips to using Fluint: Learn how to use Sequences if you want to test a Flex component - it will make your life considerably easier.
RIATest. This thing rocks. It blew away Flex Monkey (at least in my evaluation)
Our app is fairly large and complicated with custom components and all. RIATest could recognize and manipulate almost everything. Others needed small tweaks since they were custom composite components.
RIA Test has the right balance between features, price and ease of use.
Down side. Paid commercial product.
Related
We have a flex based UI whose functionality (eg: login, logout) needs to be tested periodically in IE 7. Manual testing takes a lot of time and hence we want to automate it.
EDIT: We dont have the source code of this app uder test so cannot use something like FlexUnit. Is there a way to automate this testing?
I have heard of FleXmonkey, but have read negative reviews of it and hence wary of trying it.
EDIT: source of negative review:
Automated testing of FLEX based applications
(Comment by Ryan H)
Please suggest if theres a way to go about it.
EDIT: An open-source solution is preferred.
Thank you.
You won't get very far in your development adventures if you avoid reviewing things just because you read negative reviews. ( Please provide a source for said negative reviews ). I've also read bad things about Flex and Flash Builder and the Flash Player. Yet, you still ended up using Flex?
That said, I strongly suggest checking out Flex Monkey.
RIATest is another option.
I believe you can also do this sort of testing with QTP. I couldn't find a specific source for QTP info, but there are a few links if you google.
http://www.learnqtp.com/does-quicktest-professional-support-flashadobe-flex-applications/
http://vishnuagrawal.blogspot.com/2009/04/flex-automation-testing-with-qtp.html
Without recompiling the source code, it is not possible to test Flex in all cases. Saucelabs IDE claims that it can test Flex without recompiling the code. You may check that.
I think FlexMonkey is pretty cool and worth judging on your own. Most of these tools use the same underlying automation framework provided by Adobe, and thus have many of the same pains and challenges. We've actually done a lot the last few months to make it easier to work with and have been working to provide more documentation on dealing with common challenges.
If you do have problem, you will find that our Forums are very active with questions and answers: http://www.gorillalogic.com/forum/7
-Jon Rose -
Gorilla Logic, Inc
Some Flex automation tools support testing of Flex applications without requiring to recompile it.
RIATest definitely supports it (called runtime loading) and if I am not wrong QTP does it as well. AFAIK there are no open source tools that support it.
Beware though that runtime loading is not for every application. If for example you are using custom html wrapper you cannot use it with runtime loading.
We are currently thinking of getting our apps automatically built and tested through a CI server.
My major concern is regarding UI tests. I've been playing with FlexMonkium/Selenium IDE in Firefox but it doesn't really play nice with my app. I'm also testing RIATest 3 but I would have liked the Firefox approach better because it's easy to use for non-technical users.
A bigger concern is that my app is really complex, using custom events, custom components, ExternalInterface calls.. in order to make it all work smoothly, should I give an automationName to every single component? How can I handle custom events? RPC responses time?
More generally, starting fresh, what is the best "way" to design your app to be Automation Compliant?
In my company, one Flex developer researched several tools for automating testing UI in Flex. His conclusion was that RIATest has the best support of non-trivial UI. Although, you can have problems even with it, if your UI is very complex.
As an alternative approach, you can reduce number of tests for UI, that is usually more expensive, and test on a lower level, that is cheaper. You will not have a headache with maintaining tests when UI changes, and so on.
If you can test behind UI, it will be better. Besides, as I understand, there is no an enough good tool for testing complex Flex UI.
Has anyone built application to compare the performances of them?
(performances like speed, different between GUI etc.)
If no, Can anyone recommend me which function (in each of them) should I use to compare them?
Thanks
Try the Bubblemark animation test: http://bubblemark.com/
It runs with multiple versions of Silverlight, JavaFX, Flash/Flex, and even includes DHTML and some other frameworks.
I used all three.
From my experience I can tell that Flash output (Flex produces) is fastest. Silverlight is also OK. JavaFX is again slow like applets.
The best gui is provided by Flex. But this is about taste. JavaFx also has great features about ui. But it is really slow. Noone would want to wait for it.
If you use Flex the output will be flash so nearly everyone will see it. But some may not see silverlight since they don't install the plugin or some does not have jre so they can't see the javafx.
And also there is "Open Laszlo" It is opensource and may output flash or dhtml. Does good job.
You should try RIABench: http://www.timo-ernst.net/riabench-start/
It's much more sophisticated than Bubblemark because it's split into multiple tests and is based on true scientific methods of performance testing.
Has anyone had any luck testing a Flex app without static Automation IDs attached to components? All of the elements in the apps are generated .....
We've investigated FlexMonkey but it appears to be incompatible with any app that utilizes the ExternalInterface. RIATest's scripting language leaves much to be desired...
Thanks-
Jonathan
Unfortunately I don't know much about this kind of stuff, but I went to a talk that presented these tools for TDD:
- Hudson
- Flex Unit 4
I guess there are tutorials online, don't know if it helps with ExternalInterface testing.
Is there anything that prevents you from generating appropriate automationNames for your generated components? This way you should be able to refer to them properly in automation tools.
Are there any other non-changing properties that your generated components have, like maybe 'id'? If so you can use these properties to address the components. This is definitely possible in RIATest.
FunFX is a Flex automation tool that allows you to access components via ID, "automation name", "automation value" or index. While using something like the component index might be less than ideal for robust tests, if that's all that's stable, it might be worth a try. And it's written in Ruby, so that should satisfy any "real programming language" related requirements. :)
We added an "automationPrefix" property to many of our custom controls (particularly those that are reused many times on a single screen), and wrote code to append the beginning of the automationName property on any child controls. Setting the automationName was the most important parting of enabling automation testing on our Flex apps. There are several ways you could modify the automationName to be unique without making it completely static at the level that most test automation packages need it. We are currently using QTP as the test automation tool of choice.
From what I understand a framework reduces complexity in areas that are common, like a log-in system. I use ASP.NET MVC at work and have done some work in Zend Framework but do not get how a framework helps with client side development. The reason ASP.NET MVC is used at work is for Unit Testing - does a Flex framework help with this too?
Please let me know why I should or should not use a framework with Flex?
The short answer is: it depends on the framework. :) My thoughts below:
Flex is itself a framework, and you can write reasonable applications without the need for any additional framework. Flash has a built-in event model that allows for bubbling events, so you can dispatch an event in a deeply-nested user interface component and have a listener higher up in the hierarchy that handles the event. The event handler can delegate to your model, which retrieves data from the server, and Flex's binding support can ensure that your views are appropriately updated from the model. I think it's important to understand that Flex applications can and should be written more or less according to that approach, and that any additional framework should help facilitate that approach, rather than providing its own way of doing things that ends up coupling you to the framework.
That being said, an extra framework that helps facilitate this approach can absolutely provide value. I would recommend Mate or Swiz because I think they achieve this goal. They don't try to reinvent the wheel or replace parts of the Flash / Flex APIs; instead they complement them. The dependency injection features make it much easier to provide data to your views, but without coupling them to any framework. There are a number of utilities available to make it easier to work with remote services. They also have a utils to facilitate testing or even persisting data in shared objects.
I have also worked with Cairngorm in the past and I would not recommend it. CG is notorious for requiring you to create a ton of classes that adhere to CG-specific APIs and requiring you to use many of their Singleton implementations which make your app brittle and hard to test in isolation. It is based on a number of J2EE patterns which fell out of favor in the Java community at least 5 years ago.
I've read a little about PureMVC and while I can't speak to its invasive nature, I think that reinventing the event model (called "notifications") is silly and couples you to their framework. Sure, you can say it "insulates" you from the Flash event model in case it changes, but I'd say the odds of PureMVC changing their notification model is far more likely than Adobe changing the event model. :)
If you've ever tried building a slightly large application, or one at all complex, things can quickly get out of control. I don't know how many projects I bailed on when I was first starting out because I didn't know patterns, or how to make parts of the system communicate without being tied to each other, or dependent on each other.
So, basically a framework is a collection of patterns put together. Theoretically, if you learn to follow the "rules" of a (tried and true) framework, your app will not get out of control to the point where you find yourself fixing one bug and causing two. I've been there and it's not fun.
I also found that by learning to use a framework you initially don't have to know so much about the patterns behind what you're doing. But before long you'll get a good handle on the patterns used and you'll be able to apply them in new situations or find a better pattern. So it's a good learning tool as well.
I'm sure people will have arguments against using frameworks -- this is just my experience. But if you become familiar with a few, you will likely find that one might be suitable for one project but not another.
As far as Flex frameworks, I personally like PureMVC. In all honesty the only other one I've given a good amount of time to is Cairngorm. But I like PureMVC because it feels right to me, but also it's generally not dependent on so much on the built-in Actionscript classes. For example, it uses it's own notification system. So if the notifications change in Flex, they'll still work in your PureMVC app. Also, the creator Cliff is super helpful in his forums, and he's really passionate about it. And the documentation is great.
I recommend coming up with a super basic app, and building it without any frameworks, then again with a couple others. You don't have to finish the app, but just get a feel for what's behind the frameworks.
You may find value in using a framework if:
You work in a team environment and want to be sure everyone uses the same approach
You have a large, sophisticated app and want to communicate effectively between parts of the app
You want to be able to hire additional programmers and have them be productive quickly
You want to ensure your application is built with a proven architecture
You want to leverage the knowledge of design patterns and solutions that are known to work well
You want to simplify a complex application
Here's a great article on Flex frameworks.
Flex Framework Comparison
And, I agree with the conclusion...Mate is a great Flex Framework.
Another interesting framework not mentioned in this article is Spicefactory's Parsley.
I use PureMVC (which is an MVC-framework for Flex) for my larger applications. Like 99miles stated, it depends on the application size.
If you just have to make a (relatively) small tool/application, you can get it going with just Flex. But if you want to create a large, complex application a Framework might come in handy.
The main reason I use PureMVC in Flex is because of the Model-View-Controller architecture. This separates the presentation layer from the business layer and works a lot with events and notifications. This gives you a lot of control in creating reusable classes/controls in Flex in a relatively simple way.
As I said before, for a small tool/app/website I wouldn't recommend using a framework, because in that case it might be to time consuming and complex. But for the larger applications, frameworks are quite handy.
More information on the MVC architecture can be found here.
Hope this helps :)