I am trying to test our source tree using PHPUnit with old, web based legacy code, trying to make as few changes as possible to begin. Once testing is in place, I can then change the library functions for better use, and better unit testing. However, I need the tests done to allow me to change it.
Question: We share the library code across many projects of our application, but they all share a common directory structure. When the website runs, local directories are available when we require files in the library.
Consider this:
APP1
APP2
Library
Library/COMM
Library/UTIL
...
When you launch the application, you point to either APP1 or APP2 for the different applications. They have the common code (messaging, DB access, etc...). in Library. The problem is, that the functions in the library need special parameters to work, as they are coded today. These libraries simply require('Config.php'); since it will be found in either APP1 or APP2 (they both have one with application specific settings) and the web server is using the APP1 or APP2 as the directory when the Library files were require()'d.
While this works, it fails when attempting to run the code in PHPUnit. My question is how to include the Config.php file without having to change the legacy code too much before the testing is in place.
I know this is the wrong format, but this is what I inherited.
I can not simply require('../../APP1/Config.php'); since both applications share this library.
Any suggestions are greatly appreciated.
Note: We are trying to test the library and all projects as we begin writing tests, so not sure if the include_path will solve it. I am contemplating different PHPUnit.xml.dist files for each application, but trying to avoid this right now due to corporate influence of testing all applications right away.
From phpunit.xml (<phpunit bootstrap="./bootstrap.php">...) and from the cli (--bootstrap ./bootstrap.php) you can specify a bootstrap file. In that file you could do this inclusion you are looking to do.
As word of advice, when stating to test a legacy code base don't start with Unit test. Your first goal should be "get some kind of automated tests in place". For most people, this will be system tests. That is testing the stack/site as a whole. A common tools for this is Selenium.
This is still not small task. What you are going to have to do it work out "how do I put my system in to a consistent state". The first thing you may need to do is automate importing and emptying test data in your database. Once you can do that you will be able to reliably run automated tests. You will need to get many other things to be consistent also, date and times being a good example.
My point is that, from experience, starting with Unit tests will not give you the value you need to prove that automated testings is worth the effort.
Good luck!
Related
I'm new to DevOps, so forgive me if this is trivial, but given the following workflow, what is the purpose of the integration server?
I've been given the following steps as an example of an approach to DevOps at my organisation :
Developers check in changes to source control (TFS).
Build server checks for changes.
Artefacts of the build are deployed to an "integration server" which has a copy of our ERP on it.
A release management application takes the output from this ERP environment and moves it to test, pre-production, and production environment as and when.
Is this approach correct, and if so, is the purpose of an integration server merely to provide a working implementation of code, that isn't accessed for any means other than moving code onto other servers?
My answer is making some assumptions on what it sounds like is going on in your environment.
When you check in changes to source control with AX, it's adding *.xpo text files of the code/objects that are your changes only.
It sounds like your "integration server" is a build/staging server. Imagine these two scenarios:
You have a customization with 3 objects, and you add 2 of the objects to source control and forget one. When you build on the integration server, it could have compile errors because that missing dependent object.
In your development environment, you create test forms and jobs that are basically junk you are experimenting with. You do not add these objects to source control. You wouldn't want this code to be deployed to your other environments, so the integration environment ensures the code is strictly from the repo.
Doing full compiles/syncs against the integration will also help identify issues. Then you can deploy the environment in its entirety to your other environments.
The big thing to realize is that your repo is really only your changes to the base (sys/syp) code. So part of the integration/build process is your code & base code combining.
I'm looking for suggestions to improve the process of automating functional testing of a website. Here's what I've tried in the past.
I used to have a test project using WATIN. You effectively write what look like "unit tests" and use WATIN to automate a browser to click around your site etc.
Of course, you need a site to be running. So I made the test actually copy the code from my web project to a local directory and started a web server pointing to that directory before any of the tests run.
That way, someone new could simply get latest from our source control and run our build script, and see all the tests run. They could also simply run all the tests from the IDE.
The problem I ran into was that I spent a lot of time maintaining the code to set up the test environment more than the tests. Not to mention that it took a long time to run because of all that copying. Also, I needed to test out various scenarios including installation, meaning I needed to be able to set the database to various initial states.
I was curious on what you've done to automate functional testing to solve some of these issues and still keep it simple.
MORE DETAILS
Since people asked for more details, here it is. I'm running ASP.NET using Visual Studio and Cassini (the built in web server). My unit tests run in MbUnit (but that's not so important. Could be NUnit or XUnit.NET). Typically, I have a separate unit test framework run all my WATIN tests. In the AssemblyLoad phase, I start the webserver and copy all my web application code locally.
I'm interested in solutions for any platform, but I may need more descriptions on what each thing means. :)
Phil,
Automation can just be hard to maintain, but the more you use your automation for deployment, the more you can leverage it for test setup (and vice versa).
Frankly, it's easier to evolve automation code, factoring it and refactoring it into specific, small units of functionality when using a build tool that isn't
just driving statically-compiled, pre-factored units of functionality, as is the case with NAnt and MSBuild. This is one of the reasons that many people who were relatively early users of toole like NAnt have moved off to Rake. The freedom to treat build code as any other code - to cotinually evolve its content and shape - is greater with Rake. You don't end up with the same stasis in automation artifacts as easily and as quickly with Rake, and it's a lot easier to script in Rake than NAnt or MSBuild.
So, some part of your struggle is inherently bound up in the tools. To keep your automation sensible and maintained, you should be wary of obstructions that static build tools like NAnt and MSBuild impose.
I would suggest that you not couple your test environment boot-strapping from assembly load. That's an inside-out coupling that only serves brief convenience. There's nothing wrong (and, likely everything right) with going to the command line and executing the build task that sets up the environment before running tests either from the IDE or from the command line, or from an interactive console, like the C# REPL from the Mono Project, or from IRB.
Test data setup is simply just a pain in the butt sometimes. It has to be done.
You're going to need a library that you can call to create and clean up database state. You can make those calls right from your test code, but I personally tend to avoid doing this because there is more than one good use of test data or sample data control code.
I drive all sample data control from HTTP. I write controllers with actions specifically for controlling sample data and issue GETs against those actions through Selenium. I use these to create and clean up data. I can compose GETs to these actions to create common scenarios of setup data, and I can pass specific values for data as request parameters (or form parameters if needs be).
I keep these controllers in an area that I usually call "test_support".
My automation for deploying the website does not deploy the test_support area or its routes and mapping. As part of my deployment verification automation, I make sure that the test_support code is not in the production app.
I also use the test_support code to automate control over the entire environment - replacing services with fakes, turning off subsystems to simulate failures and failovers, activating or deactivating authentication and access control for functional testing that isn't concerned with these facets, etc.
There's a great secondary value to controlling your web app's sample data or test data from the web: when demoing the app, or when doing exploratory testing, you can create the data scenarios you need just by issuing some gets against known (or guessable) urls in the test_support area. Really making a disciplined effort to stick to restful routes and resource-orientation here will really pay off.
There's a lot more to this functional automation (including test, deployment, demoing, etc) so the better designed these resources are, the better the time you'll have maintaining them over the long hall, and the more opportunities you'll find to leverage them in unforseen but beneficial ways.
For example, writing domain model code over the semantic model of your web pages will help create much more understandable test code and decrease the brittleness. If you do this well, you can use those same models with a variety of different drivers so that you can leverage them in stress tests and load tests as well as functional test as well as using them from the command line as exploratory tools. By the way, this kind of thing is easier to do when you're not bound to driver types as you are when you use a static language. There's a reason why many leading testing thinkers and doers work in Ruby, and why Watir is written in Ruby. Reuse, composition, and expressiveness is much easier to achieve in Ruby than C# test code. But that's another story.
Let's catch up sometime and talk more about the other 90% of this stuff :)
We used Plasma on one project. It emulates a web server in process - just point it at the root of your web application project.
It was surprisingly stable - no copying files or starting up an out of process server.
Here is how a test using Plasma looks for us...
[Test]
public void Can_log_in() {
AspNetResponse response = WebApp.ProcessRequest("/Login.aspx");
AspNetForm form = response.GetForm();
form["UserName"] = User.UserName;
form["Password"] = User.Password;
AspNetResponse loggedIn = WebApp.ProcessRequest(Button.Click(form, "LoginUser"));
Assert.IsTrue(loggedIn.IsRedirect());
AspNetResponse homePage = WebApp.ProcessRequest(loggedIn.GetRedirectUrl());
Assert.AreEqual(homePage.Status, 200);
}
All the "AspNetResponse" and "AspNetForm" classes are included with Plasma.
We are currently using an automated build process for our asp.net mvc application.
We use the following tools:
TeamCity
SVN
nUnit
Selenium
We use an msbuild script that runs on a build agent which can be any amount of machines.
The msbuild script gets the latest version of code from svn and builds it.
On success it then deploys the artifacts to a given machine/folder and creates the virtual site in IIS.
We then use MSBuild contrib tasks to run sql scripts to install the database and load data, you could also do a restore.
On success we kick off the nUnit tests. The test setup ensures that selenium is up and running and then drives the selenium tests much in the same way that Watin does. Selenium has a good recorder for tests which can be exported to c#.
The good thing about Selenium is that you can drive FF, Chorme and IE rather than being restricted to IE which was the case with Watin the last time i looked at it. You can also use Selenium to do load testing with the Selenium Grid therefore you can reuse the same tests.
On success msbuild then tags the build in svn. TeamCity has a job that runs overnight that will deploy the latest tag to a staging environment ready for the business users to check the project status the following morning.
In a previous life we had nant & msbuild scripts to fully manage the environment (installing java, selenium etc) however this does take a lot of time so as a pre req we assume each build agent has these installed. In time we will include these tasks.
Why do you need to copy code? Ditch Cassini and let Visual Studio create a virtual directory for you. Sure the devs must remember to build before running web tests if the web app has changed. We have found that this is not a big deal, especially if you run web tests in CI.
Data is a big challenge. As far as I can see, you must choose between imperfect alternatives. Here's how we handle it. First, I should explain that we are working with a large complex legacy WebForms app. Also I should mention that the domain code is not well-suited for creating test data from within the test project.
This left us with a couple of choices. We could: (a) run data setup scripts under the build, or (b) create all data via web tests using the actual web site. The problem with option (a) is that tests become coupled with scripts at a minute level. It makes my head throb to think about synchronizing web test code with T-SQL. So we went with (b).
One benefit of (b) is that your setup also validates application behavior. The problem is...time.
Ideally tests should be independent, without temporal coupling (can run in any order) and not sharing any context (e.g., common test data). The common way to handle this is to set up and tear down data with every test. After some careful thought, we decided to break this rule.
We use Gallio (MbUnit 3), which provides some nice features that support our strategy. First, it lets you specify execution order at the fixture and test level. We have four "setup" fixtures which are ordered -4, -3, -2, -1. These run in the specified order and before all "non setup" fixtures, which by default have an order of 0.
Our web test project depends on the build script for one thing only: a single well-known username/password. This is a coupling I can live with. As the setup tests run they build up a "data context" object that holds identifiers of data (companies, users, vendors, clients, etc.) that is later used (but never changed) throughout other all fixtures. (By identifiers, I don't necessarily mean keys. In most cases our web UI does not expose unique keys. We must navigate the app using names or other proxies for true identifiers. More on this below.)
Gallio also allows you to specify that a test or fixture depends on another test or fixture. When a precedent fails, the dependent is skipped. This reduces the evil of temporal coupling by preventing "cascading failures" which can reap much confusion.
Creating baseline test data once, instead of before each test, speeds things up a lot. However, the setup tests still might take 10 minutes to run. When I'm working on new tests I want to run and rerun them frequently. Enter another cool Gallio feature: Ambience. Ambience is a wrapper around DB4 that provides a very simple way to persist objects. We use it to persist the data context automatically. Thus setup tests must only be run once between rebuilds of the database. After that you can run any or all other fixtures repeatedly.
So what about cleaning up test data? Don't we need to start from a known state? This is a rule we have found it expedient to break. A strategy that is working for us is to use long random values for things like company name, username, etc. We have found that it is not very difficult to keep a test run inside a logical "data space" such that it does not bump into other data. Certainly I fear the day that I spend hours chasing down a phantom failing test only to find that it's some data collision. It's a trade off that is working for us currently.
We are using Watin. I quite like it. Another key to success is something Scott Bellware alluded to. As we create tests we are building up an abstract model of our UI. So instead of this:
browser.TextField("ctl0_tab2_newNote").TypeText("foo");
You will see this in our tests:
User.NotesTab.NewNote.TypeText("foo");
This approach provides three benefits. First, we never repeat a magic string. This greatly reduces brittleness. Second, tests are much easier to read and understand. Last, we hide most of the the Watin framework behind our own abstractions. In the second example, only TypeText is a Watin method. This will make it easier to change as the framework changes.
Hope this helps.
It was difficult, but not impossible, to build an integration test phase into the build process using maven. What happened was essentially this:
Ignore all JUNit tests in a specific directory unless the integration-test phase fires.
Add a maven profile to execute the integration tests.
For the pre-integration test phase -
Start Jetty running the application hitting a test database.
Start the selenium server
Run selenium integration tests in integration test phase
Stop selenium server
Stop selenium
The difficulty in this step was really setting up jetty - we couldn't get it to just launch from a war, so we actually have to have jetty unpack the war, then run the server - but it works, well, and is automated - all you have to do is type mvn -PintegrationTest (that was our integration test profile name) and off it went.
Do you mean automatically starting testing after build finished?
You could write automated scripts to copy the build files to a working IIS while the build complied successfully. And then start the automated BVT by call mstest.exe or other methods.
You could get a try with autoitx or some function language,such as Python,ruby.
Where I work we use a set schedule to build our applications. What is involved with builds? What is involved with getting an application to build somewhere other than at the local host?
I came across this just the other day... It helps clarify some things regarding building ASP.NET websites. Not a definitive answer, but worth a look.
http://msdn.microsoft.com/en-us/library/399f057w.aspx:
ASP.NET offers two options for precompiling a site:
Precompiling a site in place. This option is useful for existing sites where you want to enhance performance and perform error checking.
Precompiling a site for deployment. This option creates a special output that you can deploy to a production server.
Additionally, you can precompile a site to make it read-only or updatable. The following sections provide more details about each option.
The basic purpose of a build is to ensure that consistent set of all files needed for the website are deployed together.
This is typically accomplished by retrieving the files from source control, packaging them in an appropriate archive or archives (called artifacts of the build process) and then having scripts which send these files to the production or QA servers where the application is then launched.
Depending on the technology and the project, the build script might compile files, run tests, create archives, copy files to different locations and things of that nature.
There are different tools to support the build process to provide the underlying scripting ability. Build scripting can sometimes have unique properties, such as ensuring that a given task is only run once even though different parts of a larger build need to ensure that it is run (such as a task to create a directory to deploy artifacts, the directory is only created once and the build script engine (such as NAnt) makes sure that even if the task is called multiple times it is only run once per build.
If you want to understand more about builds, look around for some open source projects that use a technology you are familiar with, and look at their build proceedure (how you change code and deploy it).
This Joel On Software article is my favorite explanation of builds, daily builds, build servers of all time.
His central thesis in this and many other articles is the more effort it takes for you to create a build (which means any compilation or transformation of source files, including changing their location so that they will be production/test ready) the less productive and lower quality your software will be and the more mistakes you'll be prone to during deployment/builds. Ideally, with a web application, you have a single script that you can double click, it grabs the most current files from source control, compiles them, and deploys everything to the target location.
Many commercial products exist that can help you with this end. I like Team Foundation Server but it may not fit your Budget/Culture/Programming language choice. Having a custom build script is not the worst idea either because it gives your programming a team a great view of your process.
There's a wikipedia article on the subject:
In the field of computer software, the
term software build refers either to
the process of converting source code
files into standalone software
artifact(s) that can be run on a
computer, or the result of doing so.
One of the most important steps of a
software build is the compilation
process where source code files are
converted into executable code.
While for simple programs the process
consists of a single file being
compiled, for complex software the
source code may consist of many files
and may be combined in different ways
to produce many different versions.
The process of building a computer
program is usually managed by a build
tool, a program that coordinates and
controls other programs. Examples of
such a program are make, ant, maven
and SCons. The build utility needs to
compile and link the various files, in
the correct order. If the source code
in a particular file has not changed
then it may not need to be recompiled
(may not rather than need not because
it may itself depend on other files
that have changed). Sophisticated
build utilities and linkers attempt to
refrain from recompiling code that
does not need it, to shorten the time
required to complete the build. Modern
build utilities may be partially
integrated into revision control
programs like Subversion. A more
complex process may involve other
programs producing code or data for
the build process.
As part of my overall development practices review I'm looking at how best to streamline and automate our ASP.net web development practices.
At the moment, our process goes something like this:
Designer builds frontend as static HTML/CSS on a network share. This gets tweaked until signed off. (e.g. http://myserver/acmesite_design)
Once signed off, developer takes over and copies over frontend HTML/CSS to a new directory on the same server (e.g. http://myserver/acmesite_development)
Multiple developers work on local copy until project is complete.
Developer publishes code to an external publicly accessible server for a client to review/signoff.
Edits made locally based on feedback.
Republish to external server.
Signoff
Developer publishes to live public server
What goes wrong? Lots of things!
Version Control — this is obviously a must and is being introduced
Configuration errors — many many times, there are environment specific paths and variables (such as DB names, image upload directories, web server paths etc. etc.) which incorrectly get copied from local to staging to live etc. etc. with very embarrassing results.
I'm pretty confident I've got no.1 under control. What about configuration management? Does anyone have any advice as to how best to manage an applications structure within asp.net apps to minimize these kinds of problems?
I found that using SVN, NAnt and NUnit with Cruise Control.net solves a lot of the issues you describe. I think it works well for small groups and it's all free. Just need to learn how to use them.
CruiseControl.net helps you put together builds and continuous integration.
Use NAnt or MSBuild to do different environment builds (DEV, TEST, PROD, etc).
http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET
You got the most important part right. Use version control. Subversion is a good choice.
I usually store configuration along with the site; i.e. when coding a PHP-based site I have a file named config.php-dist. If you want the site to work at all you'll have to copy + edit in all the required parameters (this avoids storing passwords in version control). The -dist file should have reasonable defaults.
Upload directories should be relative if possible; actually all directories should be relative. I'm not experienced in ASP.net, but if it's anything like PHP the current directory is always the directory of the file being requested. If you channel all requests through a single file (i.e. index.asp), then this can even be found programmatically. Or you could find it programmatically by using the equivalent of dirname(____FILE____) in your configuration file.
I also recommend installing IIS (or whatever webserver you are using) on all development workstations (including the designers). Makes life easier as noone can step on each others toes. What one has to do is simply add test hosts to the hosts file (\windows\system32\drivers\etc\hosts iirc) in addition to adding a site to the local IIS. This plays well with version control (checkout, add site to IIS and hosts-file, edit edit edit commit).
One thing that really helps is making sure you keep your paths relative where you can and centralise them where you can't, so when I've been working with ASP.Net I have tended to use web.config to store any configuration and path related data that can't be found programmatically. It is quite possible to find information like your current application path programmatically through the Request object - it's worth looking in some detail over what the environment makes available to you.
One way to make sure you don't end up on something that is dependent on the path name is having a continuous integration server executing your test suite against your application. Each time this happens you create a random filepath. As soon as someone introduces a dependency on the filepath it will fail.
Or, actually establishing a build process when there isn't much of one in place to begin with.
Currently, that's pretty much the situation my group faces. We do web-app development primarily (but no desktop development at this time). Software deployments are ugly and unwieldy even with our modest apps, and we've had far too many issues crop up in the two years I have been a part of this team (and company). It's past time to do something about that, and the upshot is that we'll be able to kill two Joel Test birds with one stone (daily builds and one-step builds, neither of which exists in any form whatsoever).
What I'm after here is some general insight on the kinds of things I need to be doing or thinking about, from people who have been in software development for longer than I have and also have bigger brains. I'm confident that will be most of the people currently posting in the beta.
Relevant Tools:
Visual Build
Source Safe 6.0 (I know, but I can't do anything about whether or not we use Source Safe at this time. That might be the next battle I fight.)
Tentatively, I've got a Visual Build project that does this:
Get source and place in local directory, including necessary DLLs needed for project.
Get config files and rename as needed (we're storing them in a special sub directory that isn't part of the actual application, and they are named according to use).
Build using Visual Studio
Precompile using command line, copying into what will be a "build" directory
Copy to destination.
Get any necessary additional resources - mostly things like documents, images, and reports that are associated with the project (and put into directory from step 5). There's a lot of this stuff, and I didn't want to include it previously. However, I'm going to only copy changed items, so maybe it's irrelevant. I wasn't sure whether I really wanted to include this stuff in earlier steps.
I still need to coax some logging out of Visual Build for all of this, but I'm not at a point where I need to do that yet.
Does anyone have any advice or suggestions to make? We're not currently using a Deployment Project, I'll note. It would remove some of the steps necessary in this build I presume (like web.config swapping).
When taking on a project that has never had an automated build process, it is easier to take it in steps. Do not try to swallow to much at one time, otherwise it can feel overwhelming.
First get your code compiling with one step using an automated build program (i.e. nant/msbuild). I am not going to debate which one is better. Find one that feels comfortable to you and use it. Have the build scripts live with the project in source control.
Figure out how you want your automated build to be triggered. Whether it is hooking it up to CruiseControl or running a nightly build task using Scheduled Tasks. CruiseControl or TeamCity is probably the best choice for this, because they include a lot of tools you can use to make this step easier. CruiseControl is free and TeamCity is free to a point, where you might have to pay for it depending on how big the project is.
Ok, by this point you will be pretty comfortable with the tools. Now you are ready to add more tasks based on what you want to do for testing, deployment, and etc...
Hope this helps.
I have a set of Powershell scripts that do all of this for me.
Script 1: Build - this one is simple, it is mostly handled by a call to msbuild, and also it creates my database scripts.
Script 2: Package - This one takes various arguments to package a release for various environments, such as test, and subsets of the production environment, which consists of many machines.
Script 3: Deploy - This is run on each individual machine from within the folder created by the Package script (the Deploy script is copied in as a part of packaging)
From the deploy script, I do sanity checks on things like the machine name so things don't accidentally get deployed to the wrong place.
For web.config files, I use the
<appSettings file="Local.config">
feature to have overrides that are already on the production machines, and they are read-only so they don't accidentally get written over. The Local.config files are not checked in, and I don't have to do any file switching at build time.
[Edit] The equivalent of appSettings file= for a config section is configSource="Local.config"
We switched from using a perl script to MSBuild two years ago and haven't looked back.
Building visual studio solutions can be done by just specifying them in the main xml file.
For anything more complicated (getting your source code, executing unit tests, building install packages, deploying web sites) you can just create a new class in .net deriving from Task that overrides the Execute function, and then reference this from your build xml file.
There is a pretty good introduction here:
introduction
I've only worked on a couple of .Net projects (I've done mostly Java) but one thing I would recommend is using a tool like NAnt. I have a real problem with coupling my build to the IDE, it ends up making it a real pain to set up build servers down the road since you have to go do a full VS install on any box that you want to build from in the future.
That being said, any automated build is better than no automated build.
Our build process is a bunch of homegrown Perl scripts that have evolved over a decade or so, nothing fancy but it gets the job done. One script gets the latest source code, another builds it, a third stages it to a network location. We do desktop application development so our staging process also builds install packages for testing and eventually shipping to customers.
I suggest you break it down to individual steps because there will be times when you want to rebuild but not get latest, or maybe just need to re-stage. Our scripts can also handle building from different branches so consider that also with whatever solution you develop.
Finally we have a dedicated build machine that rebuilds the trunk and maintenance branches every night and sends out an email with any problems or if it completed successfully.
One thing I would suggest ensure your build script (and installer project, if relevant in your case) is in source control. I tend to have a very simple script that just checks out\gets latest the "main" build script then launches it.
I say this b/c I see teams just running the latest version of the build script on the server but either never putting it in source control or when they do they only check it in on a random basis. If you make the build process to "get" from source control it will force you to keep the latest and greatest build script in there.
Our build system is a makefile (or two). It has been rather fun getting it working as it needs to run on both windows (as a build task under VS) and under Linux (as a normal "make bla" task). The really fun thing is that the build gets the actual file list from a .csproj file, builds (another) makefile from that, and run that. In the processes the make file actually calls it's self.
If that thought doesn't scare the reader, then (either they are crazy or) they can probably get make + "your favorite string mangler" to work for them.
We use UppercuT.
UppercuT uses NAnt to build and it is extremely easy to use.
http://code.google.com/p/uppercut/
Some good explanations here: UppercuT