I have been asked to check how to load test a flex application.
I've though about using Selenium, since it's precisely its job. I've also found that testing Flex with Selenium requires extensions to Selenium. I've seen that there some of these :
sfapi
flex-ui-selenium
First, it seems that both requires me to rebuild my Flex application to test it. is it true ? Isn't there any way to automatically add handles ?
Second, when doing so, will I be able to do all kind of tests ? Load tests ? Performance tests ? All these in a continous way ?
Thanks
Yes there is one another way, but that is too complex to care or to implement. It's called OCR or http://en.wikipedia.org/wiki/Optical_character_recognition
Also you could use image recognition.
Alas, of course you can't automate a flash or a flex you need to rebuild it to use ExternalInterfaces to make some methods publicly available. That is how it is. The flash is one big chunk of an object there are no tiny parts that you could catch with anything else.
So you have these options only, nothing else i'm afraid.
Adobe Genie can be used for testing flex apps without rebuilding the flex application, and its free.
However, only Actionscript 3 is supported. AIR, AS1 & AS2 is not supported. Currently MX and Spark components cannot be tested, the doc says it can be done, but doesn't say how.
Another option is Sikuli, this is a image based testing tool. Its strength is its image heuristics, and does not use coordinates for comparison, unlike most image based testing tools.
No you don't need to rebuild your application. There is an option to do this, but there are also options not too. The manuals clearly describe both approaches.
Check out also:
RiaTest
Flex Monkey
Neo4J
Related
We have a flex based UI whose functionality (eg: login, logout) needs to be tested periodically in IE 7. Manual testing takes a lot of time and hence we want to automate it.
EDIT: We dont have the source code of this app uder test so cannot use something like FlexUnit. Is there a way to automate this testing?
I have heard of FleXmonkey, but have read negative reviews of it and hence wary of trying it.
EDIT: source of negative review:
Automated testing of FLEX based applications
(Comment by Ryan H)
Please suggest if theres a way to go about it.
EDIT: An open-source solution is preferred.
Thank you.
You won't get very far in your development adventures if you avoid reviewing things just because you read negative reviews. ( Please provide a source for said negative reviews ). I've also read bad things about Flex and Flash Builder and the Flash Player. Yet, you still ended up using Flex?
That said, I strongly suggest checking out Flex Monkey.
RIATest is another option.
I believe you can also do this sort of testing with QTP. I couldn't find a specific source for QTP info, but there are a few links if you google.
http://www.learnqtp.com/does-quicktest-professional-support-flashadobe-flex-applications/
http://vishnuagrawal.blogspot.com/2009/04/flex-automation-testing-with-qtp.html
Without recompiling the source code, it is not possible to test Flex in all cases. Saucelabs IDE claims that it can test Flex without recompiling the code. You may check that.
I think FlexMonkey is pretty cool and worth judging on your own. Most of these tools use the same underlying automation framework provided by Adobe, and thus have many of the same pains and challenges. We've actually done a lot the last few months to make it easier to work with and have been working to provide more documentation on dealing with common challenges.
If you do have problem, you will find that our Forums are very active with questions and answers: http://www.gorillalogic.com/forum/7
-Jon Rose -
Gorilla Logic, Inc
Some Flex automation tools support testing of Flex applications without requiring to recompile it.
RIATest definitely supports it (called runtime loading) and if I am not wrong QTP does it as well. AFAIK there are no open source tools that support it.
Beware though that runtime loading is not for every application. If for example you are using custom html wrapper you cannot use it with runtime loading.
Has anyone built application to compare the performances of them?
(performances like speed, different between GUI etc.)
If no, Can anyone recommend me which function (in each of them) should I use to compare them?
Thanks
Try the Bubblemark animation test: http://bubblemark.com/
It runs with multiple versions of Silverlight, JavaFX, Flash/Flex, and even includes DHTML and some other frameworks.
I used all three.
From my experience I can tell that Flash output (Flex produces) is fastest. Silverlight is also OK. JavaFX is again slow like applets.
The best gui is provided by Flex. But this is about taste. JavaFx also has great features about ui. But it is really slow. Noone would want to wait for it.
If you use Flex the output will be flash so nearly everyone will see it. But some may not see silverlight since they don't install the plugin or some does not have jre so they can't see the javafx.
And also there is "Open Laszlo" It is opensource and may output flash or dhtml. Does good job.
You should try RIABench: http://www.timo-ernst.net/riabench-start/
It's much more sophisticated than Bubblemark because it's split into multiple tests and is based on true scientific methods of performance testing.
I am looking for recommendations for tools for automated testing of a web application with some flex components.
To provide some background we have a web application that was entirely developed in AJAX+HTML and we were somewhat successful in using Selenium for testing that application end to end. We recently added some flex components into the mix and it got complicated.
We tried using Selenium Flex but we are disappointed with what it can do. So now we are looking for some alternatives. Ideally the tool would be able to drive both the web and the flex parts simultaneously, but we can also settle for just testing the flex components on their own. We prefer open source but good commercial tool is also an option.
I've heard about Fluint and FlexMonkey but haven't tried them yet. Anybody has any experience with using those? Any advice for someone who is just starting? Any gotchas?
Are there some other tools that could be used?
Thanks.
i'v try FlexMonkey.
It was great, until he get lost in the recording of interaction with huge and complex Flex Component.
For me, and for now, it's great in the demo or with small app, but not ready for production.
BUT : seems to evolve quickly.
I've had much success using FunFX to automate a fairly complex Flex application. FunFX is built off FireWatir, which is another web automation framework similar to Selenium. So together that should cover the entire AJAX/HTML + Flex bits of your application.
Since both FunFX and FireWatir are Ruby-based, I also recommend something like Cucumber as your testing framework to glue everything together.
We've had some success using Fluint for unit testing and Quick Test Professional (with the Flex plug-in) for functional end-to-end testing. QTP is pretty heavy-weight, but once you've got it set up and have created a few tests, it works pretty well. It works with Flex as well as vanilla HTML/Javascript, so even if your app is a mix of the two, it should be able to handle it.
Fluint works great for unit testing, but given the amount of code required to test a component, I wouldn't advocate it for functional testing (it's too low-level). FlexMonkey is more of an automation framework for Flex Unit. You basically use it to record test cases, and it generates code using Flex Unit to perform the test. I've tinkered with it, but ultimately decided to use QTP.
As for tips to using Fluint: Learn how to use Sequences if you want to test a Flex component - it will make your life considerably easier.
RIATest. This thing rocks. It blew away Flex Monkey (at least in my evaluation)
Our app is fairly large and complicated with custom components and all. RIATest could recognize and manipulate almost everything. Others needed small tweaks since they were custom composite components.
RIA Test has the right balance between features, price and ease of use.
Down side. Paid commercial product.
Has anyone had any luck testing a Flex app without static Automation IDs attached to components? All of the elements in the apps are generated .....
We've investigated FlexMonkey but it appears to be incompatible with any app that utilizes the ExternalInterface. RIATest's scripting language leaves much to be desired...
Thanks-
Jonathan
Unfortunately I don't know much about this kind of stuff, but I went to a talk that presented these tools for TDD:
- Hudson
- Flex Unit 4
I guess there are tutorials online, don't know if it helps with ExternalInterface testing.
Is there anything that prevents you from generating appropriate automationNames for your generated components? This way you should be able to refer to them properly in automation tools.
Are there any other non-changing properties that your generated components have, like maybe 'id'? If so you can use these properties to address the components. This is definitely possible in RIATest.
FunFX is a Flex automation tool that allows you to access components via ID, "automation name", "automation value" or index. While using something like the component index might be less than ideal for robust tests, if that's all that's stable, it might be worth a try. And it's written in Ruby, so that should satisfy any "real programming language" related requirements. :)
We added an "automationPrefix" property to many of our custom controls (particularly those that are reused many times on a single screen), and wrote code to append the beginning of the automationName property on any child controls. Setting the automationName was the most important parting of enabling automation testing on our Flex apps. There are several ways you could modify the automationName to be unique without making it completely static at the level that most test automation packages need it. We are currently using QTP as the test automation tool of choice.
I recently started to develop using Flex 3 and Adobe Air and I wanted to know what features you want to be in futures releases of Adobe Air ?
The ones that I miss are:
Cross-systems way of launching a local file (shellExec) right from an Air application (although you can do this using workaround at least under Windows)
Ability to setup dynamic paths for Embed statement (e.g. Embed[(variable+"/path/to/file")] ). I didn't find any way to do this properly.
Some way of setting Flex object's positions with absolute values from CSS (that sounds more Flex related by the way)
Don't hesitate to add your workarounds to theses limitations if you know somes.
Ability to call out to native code - you currently have to ship a server written in another language and make calls to that to do anything more than Air gives you.
Modal windows. There's a hack you can do which involves setting Application.application.enabled = false, setting dialog.nativeWindow.alwaysInFront = true and then re-enabling the application when the dialog closes, but this is long-winded and doesn't disable any native menus you may have! It's crazy that something so simple is made so difficult.
A usable way to set the application's icon. I spent 2 hours trying to do this the other day and gave up after persistent "303" errors that gave me no idea of what I was doing wrong. Again, really basic stuff.
"Call native code", "Modal windows", agree-agree.
loadLibrary would be really great :)
ps.:
if we can not have this features in AIR, if would be awesome to have an opportunity to embed flash(AIR) graphics engine into your own app. in that case we'll get great performance improvement and liberty of choice what to write (not only small gadgets).