Related
We are evaluating and looking for a workflow engine which support .NET-Core and I'd really appreciate the community input. I would like to hear on the suggestion based on your guys implementation stories.
My main evaluation criteria, so far, are below;
open source and OEM friendly license
production installations (success stories are a great help)
technical support available
open standards support - BPMN
dynamic creation/assembly of the workflow based on input
embeddable
Currently I am evaluating Elsa, Workflow Core, Argo, and Airflow. Elsa seems like a good candidate as well but never used it.
Do you guys have any successful deployments on Elsa workflow engine?
Full disclosure: I am the project lead of Elsa, but I will try and be as objective as I can.
Elsa does not currently support BPMN, so if this is a hard requirement then Elsa might not be suitable for your project. At least not until it implements BPMN in the future.
As for technical support, there is no official paid support available as of yet, but the community is very friendly & helpful, though still relatively small.
Dynamic creation based on input is possible since you can programmatically define workflows. But you cannot update workflows while it executes (which would be more or less similar to being able to update your C# program statements as your program runs). Not sure if this is what you are looking for or not?
Other than that, Elsa is OEM friendly, runs in production successfully at several companies that I know of and is embeddable.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
Looking to add UI testing to my WinForms 3.5 project. Currently using MSTest for unit testing and MSBuild to build it.
One option I am looking at is Test Automation FX.
The product seems to be a bit new and not fully polished, but it seems to work. So, I'm curious if anyone else is using and has good or bad things to say about it.
It is quite a bit cheaper in price ($450) than Test Complete ($2000), so I also am trying to figure out what is lacking or missing, if anything, from Test Automation FX.
I have gone recently through the process of choosing a GUI testing solution, and finally decided to go to TestAutomationFX. Here are the main reasons I made this choice:
It's creating real code (in my case C#), which is invaluable for me: for maintenability, archivability, flexibility and so on. It is much easier to write in C# (I can ask my developers for support) than in a proprietary script language I would have to learn from scratch (or worse: endless grids of non-maintanable dropboxes). It also lets me build a good testing framework
It has seamless integration with NUnit (that my team uses for unit and integration tests). My data driven test come from the same CSVs, and GUI test reports are just appended to unit test reports, granting easy archiving and maintenance
It has much better recognition of the complex UI objects my developers use (Telerik, Infragistic, home-made): 25% of my clics are in x/y mode, versus 67% with TestComplete or Ranorex
Their sales engineers gave me excellent support (at least during the evaluation period)
It has no major bugs nor complex license setup (yes, I'm looking at you, TestComplete guys, see my other post), no runtime license issue, no virtual machine licensing problems either
(though this was not that important to me), it's four times cheaper than other commercial solutions
On the other hand, there is a medium flaw in the application:
The mapping system (ie. mapping AUT-object properties to Test-application-objects) is really touchy: code refactoring needs special attention. I overcome this by commiting to my VCS before every code refactoring. Anyway, does testComplete provide the option of code refactoring.
OK, as you can see, I'm pretty ethusiast with this solution. I've been using it for only a few days, and may run into bigger problems later. But right now it gives me exactly what I wanted, so let me be happy :)
The company I work for uses SilkTest, which works very good. In general, when using automated testing, you would be doing lots of regression testing. What is more important is when you've modified an existing project, then the test software must still be able to run those tests without any errors. (Or, with the errors you'd expect.)
But the market does have lots and lots of other test solutions. In the past, I even saw a test setup which required two computers and additional hardware. The hardware would connect to the monitor, mouse and keyboard of the test system. The other end would connect to a special extension card in the test server. The hardware was there so the server could send keyboard commands to the test system and record anything that happened on the screen. With some additional OCR software, it was very well capable of analysing any errors. Then again, it had a price of six digits and to be honest, I'd rather buy a Porsche for that price and probably would have some cash left to bring two beautiful dates with me while driving through the boulevards in Nice, France...
There's a Wiki page with an overview of all kinds of test software. It doesn't compare them but you can find Test Automation FX there, although it doesn't provide much information. It seems limited to testing Windows GUI's only.
TestComplete provides more information. Then again, comparing the Wiki's it also supports a lot more. Really a lot more. Enough to explain why it's that expensive...
I have just starting to evaluate different GUI automate testing tool. I have looked at Test Automation FX, Ranorex and TestComplete. And the price for the software are in that order.
This is some of my conclusions:
Test Automation FX - Coded in C#, Fully VS integrated. But very slow in finding components and takes much memory and don't fully support DevExpress components
Ranorex - Coded in C#, Have a studio for maintating test but can be fully integrated into VS. Has better object support. And you can find objects in your software by regex expresseion on several thing. Have some problem with DevExpress components but is rather fast to work with.
TestComplete - Uses its on script language. VBscript is the easiest one (C#Script is just awkward notation). This have really good support for DevExpress components and runs the test really fast. But is very expemsive
Right now I don't know which I should use. Ranorex is alite better than Test Automation FX but both lack the full support for DevExpress components. TestComplete is nice but it introduce a new language to the development and is very expensive. But the test scripts are small and the program have more logic in finding very to click.
I have evaluated Test Automation Fx, Although it recognizes all the controls of my application (we use 3rd party controls from infragistics ie netAdvantage controls for WPF)
It is very slow in recognizing the controls and even playback time is quite slow compared to QTP or Ranorex. I would recommend Ranorex over Test Automation Fx.
We're in the initial stages of a large project, and have decided that some form of automated UI testing is likely going to be useful for us, but have not yet sorted out exactly how this is going to work...
The primary goal is to automate a basic install and run-through of the app, so if a developer causes a major breakage (eg: app won't install, network won't connect, window won't display, etc) the testers don't have to waste their time (and get annoyed by) installing and configuring a broken build
A secondary goal is to help testers when dealing with repetitive tasks.
My question is: Who should create these kinds of tests? The implicit assumption in our team has been that the testers will do it, but everything I've read on the net always seems to imply that the developers will create them, as a kind of 'extended unit test'.
Some thoughts:
The developers seem to be in a much better position to do this, given that they know control ID's, classes, etc, and have a much better picture of how the app is working
The testers have the advantage of NOT knowing how the app is working, and hence can produce tests which may be much more useful
I've written some initial scripts using IronRuby and White. This has worked really well, and is powerful enough to do literally anything, but then you need to be able to write code to write the UI tests
All of the automated UI test tools we've tried (TestComplete, etc) seem to be incredibly complex and fragile, and while the testers can use them, it takes them about 100 times longer and they're constantly running into "accidental complexity" caused by the UI test tools.
Our testers can't code, and while they're plenty smart, all I got were funny looks when I suggested that testers could potentially write simple ruby scripts (even though said scripts are about 100x easier to read and write than the mangled mess of buttons and datagrids that seems to be the standard for automated UI test tools).
I'd really appreciate any feedback from others who have tried UI automation in a team of both developers and testers. Who did what, and did it work well? Thanks in advance!
Edit: The application in question is a C# WPF "rich client" application which connects to a server using WCF
Ideally it should really be QA who end up writing the tests. The problem with using a programmatic solution is the learning curve involved in getting the QA people up to speed with using the tool. Developers can certainly help with this learning curve and help the process by mentoring, but it still takes time and is a drag on development.
The alternative is to use a simple GUI tool which backs a language (and data scripts) and enables QA to build scripts visually, delving into the finer details of the language only when really necessary - development can also get involved here also.
The most successful attempts I've seen have definitely been with the latter, but setting this up is the hard part. Selenium has worked well for simple web applications and simple threads through the application. JMeter also (for scripted web conversations for web services) has worked well... Another option which is that of in house built test harness - a simple tool over the top of a scripting language (Groovy, Python, Ruby) that allows QA to put test data into the application either via a GUI or via data files. The data files can be simple properties files, or in more complex cases structured (something like YAML or even Excel) data files. That way they can build the basic smoke tests to start, and later expand that into various scenario driven tests.
Finally... I think rich client apps are way more difficult to test in this way, but it depends on the nature of the language and the tools available to you...
In my experience, testers who can code will switch jobs for a pay raise as developers.
I agree with you on the automated UI testing tools. Every place I've worked that was rich enough to afford WinRunner or LoadRunner couldn't afford the staff to actually use it. The prices may have changed, but back then, these were in the high 5-digit to low 6-digit price tags (think of the price of a starter home). The products were hard to use, and were usually kept uninstalled in a locked cabinet because everyone was afraid of getting in trouble for breaking them.
I worked over 7 years as an application developer before I finally switched to testing and test automation. Testing is much more challenging than coding, and any automation developer who wants to succeed should master testing skills.
Some time ago I put my thoughts on skill matrices in a couple of blog posts.
If interested to discuss:
http://automation-beyond.com/2009/05/28/qa-automation-skill-matrices/
Thanks.
I think having the developers write the tests will be of the most use. That way, you can get "breakage checking" throughout your dev cycle, not just at the end. If you do nightly automated builds, you can catch and fix bugs when they're small, before they grow into huge, mean, man-eating bugs.
What about the testers proposing the tests, and the developers actually writing it ?
I believe at first it largely depends on the tools you use.
Our company currently uses Selenium (We're a Java shop).
The Selenium IDE (which records actions in Firefox) works OK, but developers need to manually correct mistakes it makes against our webapps, so it's not really appropriate for QA to write tests with.
One thing I tried in the past (with some success), was to write library functions as wrappers for Selenium functions. They read as plain english:
selenium.clickButton("Button Text")
...but behind the scenes check for proper layout and tags on the button, has an id etc.
Unfortunately this required a lot of set up to allow easy writing of tests.
I recently became aware of a tool called Twist (from Thoughtworks, built on the Eclipse engine), which is a wrapper for Selenium, allowing plain English style tests to be written. I am hoping to be able to supply this to the testers, who can write simple assertions in plain English!
It automatically creates stubs for new assertions too, so the testers could write the tests, and pass them to developers if they need new code.
I've found the most reasonable option is to have enough specs such that the QA folks can stub out the test, basically figure out what they want to test at each 'screen' or on each component, and stub those out. The stubs should be named such that they're very descriptive as to what they're testing. This also offers a way to crystalize functional requirements. In fact, doing the requirements in this fashion are particularly easy, and help non-technical people really work through the muddy waters of their own though process.
The stubs can be filled in via a combination of QA/dev people. This allows you to CHEAPLY train QA people as to how to write tests, and they typically slurp it up as it furthers their job security.
I think it depends mostly on the skill level of your test team, the tools available, and the team culture with respect to how developers and testers interact with each other. My current situation is that we have a relatively technical test team. All testers are expected to have development skills. In our case, testers write UI Automation. If your test team doesn't have those skills they will not be set up for success. In that case, it may be best for developers to write you UI automation.
Other factors to consider:
What other testing tasks are on the testers' plate?
Who are your customers and what are their expectations related to quality?
What is the skill level of the development team and what is their willingness to take on test automation work?
-Ron
We have 4 different environments:
Staging
Dev
User Acceptance
Live
We use TFS, pull down the latest code and code away.
When they finish a feature, the developers individually upload their changes to Staging. If the site is stable (determined by really loose testing), we upload changes to Dev, then UserAcceptance and then live.
We are not using builds/tags in our source control at all.
What should I tell management? They don't seem to think there is an issue as far as I can tell.
If it would be good for you, you could become the Continuous Integration champion of your company. You could do some research on a good process for CI with TFS, write up a proposed solution, evangelize it to your fellow developers and direct managers, revise it with their input and pitch it to management. Or you could just sit there and do nothing.
I've been in management for a long time. I always appreciate someone who identifies an issue and proposes a well thought-out solution.
Whose management? And how far removed are they from you?
I.e. If you are just a pleb developer and your managers are the senior developers then find another job. If you are a Senior developer and your managers are the CIO types, i.e. actually running the business... then it is your job to change it.
Tell them that if you were using a key feature of very expensive software they spent a lot of money on, it would be trivial to tell what code got pushed out when. That would mean in the event of a subtle bug getting introduced that gets passed user acceptance testing, it would be a matter of diffing the two versions to figure out what changed.
One of the most important parts of using TAGS is so you can rollback to a specific point in time. Think of it as an image backup. If something bad gets deployed you can safely assume you can "roll" back to a previous working version.
Also, developers can quickly grab a TAG (dev, prod or whatever) and deploy to their development PC...a feature I use all the time to debug production problems.
So you need someone to tell the other developers that they must label their code every time a build is done and increment a version counter. Why can't you do that?
You also need to tell management that you believe the level of testing done is not sufficient. This is not a unique problem for an organisation and they'll probably say they already know. No harm in mentioning it though rather than waiting for a major problem to arrive.
As far as individuals doing builds or automated build processes this depends on whether you really need this based on how many developers there are and how often you do builds.
What is the problem? As you said, you can't tell if management see the problem. Perhaps they don't! Tell them what you see as the current problem and what you would recommend to fix the problem. The problem has to of the nature of "our current process has failed 3 out of 10 times and implementing this new process would reduce those failures to 1 out of 10 times".
Management needs to see improvements in terms of: reduced costs, icreased profits, reduced time, reduced use of resources. "Because it's widely used best practice" isn't going to be enough. Neither is, "because it makes my job easier".
Management often isn't aware of a problem because everyone is too afraid to say anything or assumes they can't possibly fail to see the problem. But your world is a different world than theirs.
I see at least two big problems:
1) Developers loading changes up themselves. All changes should come from source control. Do you encounter times where someone made a change that went to production but never got into source control and then was accidentally removed on the next deploy? How much time (money) was spent trying to figure out what went wrong there?
2) Lack of a clear promotion model. It seems like you guys are moving changes between environments rather than "builds". The key distinction is that if two changes work great in UAT because of how they interact, if only one change is promoted to production it could break there. Promoting consistent code - whether by labeling it or by just zipping up the whole web application and promoting the zip file - should cause fewer problems.
I work on the continuous integration and deployment solution, AnthillPro. How we address this with TFS is to retrieve the new code from TFS based on a date-time stamp (of when someone pressed the "Deliver to Stage" button).
This gives you most (all?) the traceability you would have of using tags, without actually having to go around tagging things. The system just records the time stamp, and every push of the code through the testing environments is tied to a known snapshot of code. We also have customers who lay down tags as part of the build process. As the first poster mentioned - CI is a good thing - less work, more traceability.
If you already have TFS, then you are almost there.
The place I'm at was using TFS for source control only. We have a similar setup with Dev/Stage/Prod. I took it upon myself to get a build server installed. Once that was done I added in the ability to auto deploy to dev for one of my projects and told a couple of the other guys about it. Initially the reception was luke warm.
Later I added TFS Deployer to the mix and have it set to auto deploy the good dev build to stage.
During this time the main group of developers were constantly fighting the "Did you get latest before deploying to Stage or Production?" questions; my stuff was working without a hitch. Believe me, management and the other devs noticed.
Now (6 months into it), we have a written rule that you aren't even allowed to use the Publish command in visual studio. EVERYTHING goes through the CI build and deployments. When moving to prod, our production group pulls the appropriate copy off of the build server. I even trained our QA group on how to do web testing and we're slowly integrating automated tests into the whole shebang.
The point of this ramble is that it took awhile. But more importantly it only happened because I was willing to just run with it and show results.
I suggest you do the same. Start using it, then show the benefits to get everyone else on board.
I am developing applications for 9 years now - meanly Java. Now am asked to participate in the SVT team for the next release. Overall this means installing complex system setups and running specific user scenarios on these setups as well as doing long runs and load runs.
Overall I am positive about it as I will learn something new. But I am also affraid to loose some grip and knowledge with programming, because of not doing it a lot then.
I know doing programming in side projects such as helping with open source projects will be one alternative, but finding the time on top of a familiy life and a fulltime jop is not that easy.
What do you think, is doing concrete testing work helping getting a better software engineer?
Thanks in advance,
Michael
Testing isn't asside of programming.
You can still program automated systems so you can have recursion testing. From unit tests to real complex automated systems, the best i know is selenium which generates code you can use to build testing scripts in most languages.
There are other tools for non webapps. But I personaly believe that testing is a bit far away from "stoping coding. Unless you're just doing user point-of-view testing.
You can also do error injections which will make you write small singletons to inject them in the memory of your application.
So you can code while testing ;) and learn new stuff also.
Having been in a testing team i think it really helps, because you'll learn to exploit code easily, which will reflect when you build your own API or App at a later date.
I would say it depends on your skill and temparament. Programming knowledge will serve you well while testing. At the same time, I know that it needs a different approach and mindset and is on a completely different career track. You can always keep up your programming skills by writing code for a project you like (even if you have to make one up).