I'm trying to set up a deployment chain for some of our ASP.NET applications. The tool of choice is Web Deploy (msdeploy) - for now. Unfortunately I'm stuck on a problem.
A high level overview of the chain is thus:
Web developer creates the code and checks it in SVN;
Buildserver sees the update and builds the msdeploy .zip package of the website;
The .zip package is automatically put inside our installer and sent to various clients;
The clients run the installer on their webserver(-s);
The installer uses msdeploy internally to deploy the .zip package and create a new website or upgrade an existing one.
Msdeploy makes it easy to deploy a new instance, but I'm stumped about how to perform an "upgrade" install. The main problem is the web.config file. Each client will most certainly have made some customizations there to suit their specific environment. The installer itself offers to set some more critical parameters at the first-time installation (achieved by msdeploy's parameter mechanism), but they can do others by hand.
On the other hand, we developers also occasionally make changes to web.config, adding some new settings or removing obsolete ones. So I can't just tell msdeploy to ignore the file entirely. I need some kind of advanced XML modification mechanism. It could be a script that the developers maintain, but then it needs to be run ONLY at upgrades, not new installs.
I've no idea how to accomplish this.
Besides that, sometimes there's also some completely weird upgrade logic. For example, the application comes with our company logo, but some clients have replaced that .png file to show their own logo. Recently we needed to update the logo - but only for clients that hadn't replaced it with their own.
Similarly, there might be some cache folders that might need to be cleaned at SOME upgrades but not at others. Or folders with user content that may not be touched (but come with default content at the initial installation). Etc.
How do you normally achieve this dual behavior for msdeploy packages? Do I really need to create 2 distinct packages for every application?
Suggestion from personal experience:
Isolate customisations
Your customers should have the ability to customise their set up and the best way is to provide them with something like an override file. That way you install the new package and follow by superimposing your customer's customisations on top of your standard setup. If its a brand new install then there will be nothing to superimpose.
> top-level --
> standard files |
images | This will never be touched or changed by customer
settings.txt |
__
> customer files --
images | Customer hacks this to their heart's content
settings.txt_override |
--
Yes, this does mean that some kind of merging process needs to happen and there needs to be some script that does that but this approach has several advantages.
For settings that suddenly become redundant just issue a warning to that effect
If a customer has their own logo provide the ability to specify this in the override file
The message is clear to customers. Stay off standard files.
If customers request more customisable settings then write the default if it does not exist into the override file during upgrades.
Vilx, in answer to your question, the logic for knowing whether it is an upgrade or not must be contained in the script itself.
To run an upgrade script before installation
msdeploy -verb:sync -source:contentPath="C:\Test1" -dest:contentPath="C:\Test2" -preSync:runcommand="c:\UpgradeScript.bat"
Or to run an upgrade script after installation
msdeploy -verb:sync -source:contentPath="C:\Test1" -dest:contentPath="C:\Test2" -postSync:runcommand="c:\UpgradeScript.bat"
More info here
As to how you know its an upgrade your script could check for a text file called "version.txt" and if it exists the upgrade bat script will run. Version to be contained within the text file. Bit basic but it should work.
This also has the added advantage of giving you the ability of more elegantly merging customer's custom settings between versions as you know which properties could be overriden for that particular version.
There are some general suggestions (not specific to msdeploy), but I hope that helps:
I think you'll need to provide several installers anyway: for the initial setup and for each version-to-version upgrade.
I would suggest to let your clients to merge the config files themselves. You could just provide them either detailed desciption of waht was added/changed/removed, and/or include the utility that simplifies the merge. Maybe this and this links will give you some pointers.
as for merging the replaced logos, other client's customization, I think the best approach would be to support branding your application. I mean - move all branding details to the place where your new/upgrade installers won't touch that.
as for the rest of the adjustments made by your clients, they do that on their own risk, so the only help you could provide them is to include the detailed list of changes (maybe even the list of changed files since the previous version) and the How-To article about merging the sources with tools like Araxis Merge or similar
Or.. you could create a utility and include it to the installer, which will try to do all the tricky merging stuff on client's machine. I would not recommend this way as it requires a lot of efforts/resources to maintain.
One more thing: you could focus on backup-ing the previous client copy before upgrade. So even client will have troubles with upgrading - that will be always possible to roll back. The only thing here for you is to provide a good feedback channel which your clients can use to shoot their troubles. This feedback will allow you to figure out what the troubles your clients have and how to make their upgrade process more comfortable.
I would build on what the above have said, but I would do it with transformations, and strict documentation about who configures what. The way you have it now relies on customer intervention against a config that is mission critical to the app deploy process.
Create three config file areas. One for development, one for the "production generic" build, and one that is an empty template for the customer to edit.
The development instance should be self explanatory. This is the transform that takes the production generic template and creates a web config for your development server. (it sounds like you are shooting for a CI type process here)
The "production generic" transform should set the app up for a hypothetically perfect instance of the app. This is what the install would look like if the architect had his way.
The customer transform is used by the customers to set up the web config as required to meet their own needs. Write some documentation and see what happens. Edit the docs as you help customers through the process.
It that what you were looking for? Thoughts?
Related
I'm not a WordPress developer, but I'm trying to determine the optimal way for a team of folks to work with WordPress. For a Rails project or most anything else, it's easy to work locally and deploy upstream, but my understanding is that WordPress doesn't make this quite as easy. Maybe this is a myth?
From what I gather, it's not uncommon for URLs and file paths to be stored in the database which seems like it would make it difficult to deploy a WP project from dev --> stg --> prd (where each environment has it's own URL and possibly different file paths), much less for individual developers to have their own dev environment that would need to be "merged" into a unified copy for deployment.
I could configure all developer sandboxes to use a single database, but here again, if URLs and file paths are stored then nothing is gained.
There are a series of smaller questions here, but the more I think about those, the more I realize that what I'm really asking for is advice about how to structure things for optimal development of a WordPress site that will be hacked on by a team of developers. I'd prefer the sandboxed approach we use for other projects, but I have no idea if/how things can be unified once all development is complete.
Warning: incoming wall of text..
#Rob, WP is hell when it comes to working in teams; however, with a little work (and some symlink magic) you can set up your WP projects so that your working files for your themes or plugins can reside separately from the WP core. Some of this uses WP's built in mechanisms, some of it is related to SVN externals (hint). I'll let you google that since it's outside the scope of your question.
A note on WP GUIDs
WARNING: DO NOT replace guids. WP GUIDs are there for external feed readers. Feed readers use the GUID to determine if the content is recent. Changing it basically tells those readers that every entry in the feed is new (especially for posts.) That introduces a lot of extra overhead for legacy content that you just don't need. GUID are a legacy feature that should have been changed a long time ago to UUIDs. Technically, you can use anything int he guid field, but WP uses the permalink to populate that field -- legacy.
The only time it is ever acceptable to change the GUID is for new wp projects where content is brand-spankin' new.
To answer your question:
WP stores explicit references to the current domain in a dozen places in it's DB. These locations are a pain to track down and change, and the last thing you want to do is deal with manual edits to a *.sql dump file that you're going to import into production. It just smacks of bad development practices.
There's a couple ways to get around this, but it means a little bit of work if you're already further down your development lifecycle. I'll address the first case.
Case 1: Project Onset
When you're starting the project, you'll likely have a development sandbox and DB ready. You'll likely have WP already installed by now, so it's essentially clean for all intents and purposes.
The first thing you're going to want to do is change how your config file works. Most folks keep with the standard wp-config.php file (beyond a team production project, there's not really any reason to edit it.) However, you can set it up with some logic to include developer-specific or environment-specific config files. For example:
wp-config.php
switch( $current_environment )
{
case 'jack.local' : include( 'wp-config-jack.php' ) break; // Jack's sandbox
case 'jill.local' : include( 'wp-config-jill.php' ) break; // Jill's sandbox
default : ... break; // Staging & Production
}
The next thing you're going to want to do is include the normal contents of the wp-config.php file in a wp-config-remote.php file for use on staging/production. Next, edit your wp-config-remote.php file so that you can use 1 config file across multiple environments (staging,production). An if(...) or switch(...) block is all you need, e.g.
if( (strpos( $_SERVER[ "HTTP_HOST" ], "localhost" ) !== false) || (strpos( $_SERVER[ "HTTP_HOST" ], "local" ) !== false) )
(There are better ways to write that condition... this is just a crude example.)
Configure all of your WP settings specific to each of your remote environments. Hopefully you'll be checking this into a source control repository.
That basically frees you up to let your team have config settings specific to their environment, while letting you check in settings for each of the remote environments once.
The second thing you're going to want to do is build a mechanism to intercept and filter domain-specific links. The intent behind this mechanism is to replace any references to the current domain with a token/placeholder. I've outline the technique to do this here: http://www.farfromfearless.com/2010/09/07/url-token-replacement-techniques-for-wordpress-3-0/
It basically amounts to creating a filter that acts on the content before it's submitted to the DB and before the content is rendered to the page. The technique is transparent in that it won't affect normal editing practices. You can still create your content in the editor, reference other pages, posts, images, etc. and they'll show up just fine while editing in different environments.
In recent projects, I've wrapped all of this and a few other WP "normalization" features into a single bootstrap plugin that I set & forget.
Case 2: Project Ongoing
Now, in your case, you're further along in your development lifecycle. It's going to take some work to replace those domain references, but if you follow the steps I've outline above you should only ever have to do this once. The link I supplied above gives you the SQL you'll need to do that job. It's important to note that in a multi-site environment, you'll need to do this for every "sub-site" you've created.
Once you've updated your DB, I suggest implementing the steps in CASE 1 so you don't have to repeat the steps again.
Bonus: synchronizing content
Synchronizing content is a pain. What I've done in recent projects is had clients work on the staging server and promote changes upstream to production. So then, that leaves you with synchronizing downstream to your sandbox(es). Write a shell script that dumps a copy of SPECIFIC content tables from your staging DB, and imports them into your sandbox DB (effectively replacing content tables.) You should be able to see the benefit of the domain-token-replacement technique.
Images that aren't checked into source control, e.g. client images should be pushed to a common location, e.g. an S3 Bucket. There are WP plugins that can help you with that. That'll save a lot of time having to synchronize assets across environments.
I hope this helps you out -- if not, there's always SilverStripe ;)
It easy we build on a dev server and move to live server with this sql query:
UPDATE wp_posts SET guid = REPLACE(guid, 'devserver.com', 'liveserver.com');
UPDATE wp_posts SET post_content = REPLACE(post_content, 'devserver.com', 'liveserver.com');
UPDATE wp_options SET option_value = REPLACE(option_value, 'devserver.com', 'liveserver.com');
Last year I wrote a bash script to mirror a live MU installation into a sandbox. It's not perfect and not ideal, but a good starting point. It consists of mirroring the databases, files and rewriting the mirrored database to reflect the sandbox.
See http://pp19dd.com/2011/01/bash-script-to-mirror-wordpress-mu-installation-into-a-sandbox/
It's important for developers to be able to take live and exact content snapshots to replicate conditions.
I just ran into this issue myself launching a new website. My solution was to use Vagrant. Vagrant is also platform agnostic so you could be developing on a Mac and a teammate is using Windows. Same Vagrant project runs on both.
I wrote a guide on how to set Vagrant up with Wordpress from a production environment running locally on your machine. I don't use Wordpress that often but every time I do its always a hassle to setup Apache and PHP on my Mac, then make sure all the Wordpress site urls are updated in the database.
Once you've configured your Vagrant project, its a single command for any developer on your team to be up and running with a local instance of Wordpress. In short, Vagrant will mount your project directory from your host machine in the guest machine and run Apache, MySQL, PHP through the guest machine. You still use your host machine's IDE (as you normally would) and your host machines browser. There is no uploading of files anywhere, its just code, save, refresh browser all on your local machine.
http://www.distilnetworks.com/wordpress-development-with-vagrant/ <- Explains how to set it all up with Vagrant
https://gist.github.com/markmalek/fd2e6e65385400d9cd47 <- the shell script when provisioning Vagrant
The shell script could probably be a lot better, this is what worked for me but I would love to hear some better suggestions or ideas. I'm new to Vagrant and now use it for some of our other projects so thought it would be a good fit here as well.
I do update the GUID in the shell script, which I've read in another answer that you shouldn't do because feed readers use this. In this case its irrelevant since its just for your local instance of Wordpress but I wouldn't make this change in production. See this answer for better explanation.
Not a big deal, simply backup you whole site with all in one wp migration plugin and import on live server installation, plugin will replace all url automatically.
I write a lot of code, most of it I throw away eventually when I am done with it; recently I was thinking that if I just kept every small piece of utility script I wrote, named it, tagged it and filed it in a dev shell, I will never loose the code, and on top of that I won't need to redo something I have done already, which is the main motivation, as I keep finding myself writing something I've done earlier.
Is there a ASP.NET shell style environment anywhere?
If not, what would be the best way to go about this?
I am looking to be able to do the following:
Write big or small bits of code.
Derive from or chain together alread written code/libraries/services.
Ability to have everything on my desktop (would that mean IIS on the desktop? or is there an lighter weight mechanism?), sync'ed with the server at home, so if I am on the move I can still access this and make this part of my day-to-day workflow.
You could build a unique solution, with many class library projects inside. Each project would address a specific scenario, something like this:
MyStuff (Solution)
MyStuff.Common
MyStuff.Validation
MyStuff.Web
MyStuff.Encryption
etc.
Then you can put this solution on an online versioning service like bitbucket or assembla, so you can access your source code from anywhere, edit it and commit it back to the server. This way you get the advantages of versioning and you store your code on a remote server so even if your harddisk breaks it's not a problem, cause what's on the server is what matters.
You should either look into a source control system (Git perhaps?) or into a file storage / syncing / sharing service like DropBox.
DropBox would allow you to access code snippets from wherever you are and works really easily (just drop a file into a folder).
If you need versioning and branching you're going to have to look into a source control system. Since you have a server at home, that should be no problem.
For an ASP.NET web application that is packaged and sold to customers for deployment, what would be the best location for a "read me" file with notes about setup and configuration on the target system?
Requirements:
The file should not be accessible by
users of the web application, only
the person doing setup and
configuration.
The file should be
consumable by the MSI installer
program, so that it can be displayed
as part of the setup wizard UI.
The solution should be simple and very
low cost. (I don't want an elaborate
solution for just a simple text
file.)
Some thoughts I have are to copy the file to *App_Data* or to bin as those are protected folders by default, and then pull the file in from one of those locations in the setup program.
The readme should be a separate file that sits beside the MSI on the media you distribute the web app on. This is a standard practice dating from generations ago the dark ages. If you distribute as a download from the web then have a link for the MSI, and a link for the readme.
You could also include the same file into the MSI, but arguably that is the wrong place for it as the user has yet to reach the configuration stage, and unless they print it they won't be able to refer to it later in the MSI process (if you have any configuration steps in the MSI).
Having the instructions available via the web app is also arguably wrong, as the user may have to do some initial configuration in order to reach the page telling them how to configure the app....
So ship the instructions separately to the MSI, and make sure they look okay and are easily readable when printed out. Remember these pointers:
Instructions are not always read
Instructions are not always read at the time of installation
Instructions are not always read by the same person that does the installation
Instructions are not always read from the screen
Instructions are not always read correctly, even when they are simple
Instructions are not always read (I know that is a duplicate of the first point...)
Don't forget to clearly distinguish between pre-install and post-install configuration instructions (even if they are in the same document) - you want to minimize the risk of the end user getting it wrong (which some of them will do no matter how hard you try).
Build the important message into your application. Do it like Apache where it says "this is a new installation of...." and don't allow that screen to go away until they go in and do all the things that you consider important.
This isn't a problem for your installer to solve.
As part of my overall development practices review I'm looking at how best to streamline and automate our ASP.net web development practices.
At the moment, our process goes something like this:
Designer builds frontend as static HTML/CSS on a network share. This gets tweaked until signed off. (e.g. http://myserver/acmesite_design)
Once signed off, developer takes over and copies over frontend HTML/CSS to a new directory on the same server (e.g. http://myserver/acmesite_development)
Multiple developers work on local copy until project is complete.
Developer publishes code to an external publicly accessible server for a client to review/signoff.
Edits made locally based on feedback.
Republish to external server.
Signoff
Developer publishes to live public server
What goes wrong? Lots of things!
Version Control — this is obviously a must and is being introduced
Configuration errors — many many times, there are environment specific paths and variables (such as DB names, image upload directories, web server paths etc. etc.) which incorrectly get copied from local to staging to live etc. etc. with very embarrassing results.
I'm pretty confident I've got no.1 under control. What about configuration management? Does anyone have any advice as to how best to manage an applications structure within asp.net apps to minimize these kinds of problems?
I found that using SVN, NAnt and NUnit with Cruise Control.net solves a lot of the issues you describe. I think it works well for small groups and it's all free. Just need to learn how to use them.
CruiseControl.net helps you put together builds and continuous integration.
Use NAnt or MSBuild to do different environment builds (DEV, TEST, PROD, etc).
http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET
You got the most important part right. Use version control. Subversion is a good choice.
I usually store configuration along with the site; i.e. when coding a PHP-based site I have a file named config.php-dist. If you want the site to work at all you'll have to copy + edit in all the required parameters (this avoids storing passwords in version control). The -dist file should have reasonable defaults.
Upload directories should be relative if possible; actually all directories should be relative. I'm not experienced in ASP.net, but if it's anything like PHP the current directory is always the directory of the file being requested. If you channel all requests through a single file (i.e. index.asp), then this can even be found programmatically. Or you could find it programmatically by using the equivalent of dirname(____FILE____) in your configuration file.
I also recommend installing IIS (or whatever webserver you are using) on all development workstations (including the designers). Makes life easier as noone can step on each others toes. What one has to do is simply add test hosts to the hosts file (\windows\system32\drivers\etc\hosts iirc) in addition to adding a site to the local IIS. This plays well with version control (checkout, add site to IIS and hosts-file, edit edit edit commit).
One thing that really helps is making sure you keep your paths relative where you can and centralise them where you can't, so when I've been working with ASP.Net I have tended to use web.config to store any configuration and path related data that can't be found programmatically. It is quite possible to find information like your current application path programmatically through the Request object - it's worth looking in some detail over what the environment makes available to you.
One way to make sure you don't end up on something that is dependent on the path name is having a continuous integration server executing your test suite against your application. Each time this happens you create a random filepath. As soon as someone introduces a dependency on the filepath it will fail.
Or, actually establishing a build process when there isn't much of one in place to begin with.
Currently, that's pretty much the situation my group faces. We do web-app development primarily (but no desktop development at this time). Software deployments are ugly and unwieldy even with our modest apps, and we've had far too many issues crop up in the two years I have been a part of this team (and company). It's past time to do something about that, and the upshot is that we'll be able to kill two Joel Test birds with one stone (daily builds and one-step builds, neither of which exists in any form whatsoever).
What I'm after here is some general insight on the kinds of things I need to be doing or thinking about, from people who have been in software development for longer than I have and also have bigger brains. I'm confident that will be most of the people currently posting in the beta.
Relevant Tools:
Visual Build
Source Safe 6.0 (I know, but I can't do anything about whether or not we use Source Safe at this time. That might be the next battle I fight.)
Tentatively, I've got a Visual Build project that does this:
Get source and place in local directory, including necessary DLLs needed for project.
Get config files and rename as needed (we're storing them in a special sub directory that isn't part of the actual application, and they are named according to use).
Build using Visual Studio
Precompile using command line, copying into what will be a "build" directory
Copy to destination.
Get any necessary additional resources - mostly things like documents, images, and reports that are associated with the project (and put into directory from step 5). There's a lot of this stuff, and I didn't want to include it previously. However, I'm going to only copy changed items, so maybe it's irrelevant. I wasn't sure whether I really wanted to include this stuff in earlier steps.
I still need to coax some logging out of Visual Build for all of this, but I'm not at a point where I need to do that yet.
Does anyone have any advice or suggestions to make? We're not currently using a Deployment Project, I'll note. It would remove some of the steps necessary in this build I presume (like web.config swapping).
When taking on a project that has never had an automated build process, it is easier to take it in steps. Do not try to swallow to much at one time, otherwise it can feel overwhelming.
First get your code compiling with one step using an automated build program (i.e. nant/msbuild). I am not going to debate which one is better. Find one that feels comfortable to you and use it. Have the build scripts live with the project in source control.
Figure out how you want your automated build to be triggered. Whether it is hooking it up to CruiseControl or running a nightly build task using Scheduled Tasks. CruiseControl or TeamCity is probably the best choice for this, because they include a lot of tools you can use to make this step easier. CruiseControl is free and TeamCity is free to a point, where you might have to pay for it depending on how big the project is.
Ok, by this point you will be pretty comfortable with the tools. Now you are ready to add more tasks based on what you want to do for testing, deployment, and etc...
Hope this helps.
I have a set of Powershell scripts that do all of this for me.
Script 1: Build - this one is simple, it is mostly handled by a call to msbuild, and also it creates my database scripts.
Script 2: Package - This one takes various arguments to package a release for various environments, such as test, and subsets of the production environment, which consists of many machines.
Script 3: Deploy - This is run on each individual machine from within the folder created by the Package script (the Deploy script is copied in as a part of packaging)
From the deploy script, I do sanity checks on things like the machine name so things don't accidentally get deployed to the wrong place.
For web.config files, I use the
<appSettings file="Local.config">
feature to have overrides that are already on the production machines, and they are read-only so they don't accidentally get written over. The Local.config files are not checked in, and I don't have to do any file switching at build time.
[Edit] The equivalent of appSettings file= for a config section is configSource="Local.config"
We switched from using a perl script to MSBuild two years ago and haven't looked back.
Building visual studio solutions can be done by just specifying them in the main xml file.
For anything more complicated (getting your source code, executing unit tests, building install packages, deploying web sites) you can just create a new class in .net deriving from Task that overrides the Execute function, and then reference this from your build xml file.
There is a pretty good introduction here:
introduction
I've only worked on a couple of .Net projects (I've done mostly Java) but one thing I would recommend is using a tool like NAnt. I have a real problem with coupling my build to the IDE, it ends up making it a real pain to set up build servers down the road since you have to go do a full VS install on any box that you want to build from in the future.
That being said, any automated build is better than no automated build.
Our build process is a bunch of homegrown Perl scripts that have evolved over a decade or so, nothing fancy but it gets the job done. One script gets the latest source code, another builds it, a third stages it to a network location. We do desktop application development so our staging process also builds install packages for testing and eventually shipping to customers.
I suggest you break it down to individual steps because there will be times when you want to rebuild but not get latest, or maybe just need to re-stage. Our scripts can also handle building from different branches so consider that also with whatever solution you develop.
Finally we have a dedicated build machine that rebuilds the trunk and maintenance branches every night and sends out an email with any problems or if it completed successfully.
One thing I would suggest ensure your build script (and installer project, if relevant in your case) is in source control. I tend to have a very simple script that just checks out\gets latest the "main" build script then launches it.
I say this b/c I see teams just running the latest version of the build script on the server but either never putting it in source control or when they do they only check it in on a random basis. If you make the build process to "get" from source control it will force you to keep the latest and greatest build script in there.
Our build system is a makefile (or two). It has been rather fun getting it working as it needs to run on both windows (as a build task under VS) and under Linux (as a normal "make bla" task). The really fun thing is that the build gets the actual file list from a .csproj file, builds (another) makefile from that, and run that. In the processes the make file actually calls it's self.
If that thought doesn't scare the reader, then (either they are crazy or) they can probably get make + "your favorite string mangler" to work for them.
We use UppercuT.
UppercuT uses NAnt to build and it is extremely easy to use.
http://code.google.com/p/uppercut/
Some good explanations here: UppercuT