Related
I have a wordpress site that I am finally getting into git. Previously I was editing changes locally and uploading via ftp. I want to come up with a strategy to be able to do the following:
Update my git repository with any changes made on the ftp server (plugin updates etc). That I will then sync that down into my local copy
Push locally tested changes into git and then deploy via ftp (I know how to do this part)
I don't know the best way to approach 1 and I'm very new to using git (usually use svn).
Thanks
Using a conventional development workflow on Wordpress is challenging.
Because WP puts too much in the database, including site config, it makes syncing changes from one environment to the other a real pain.
We have a number of WP installs (against my better judgement) and organise our workflow like this.
You might lighten this process if you're not part of a dev team and are going alone.
Local Dev
codebase changes only. Including shortcode development and templating. This lives under a GIT workflow (branches, tags, etc). Plugin/framework updates are installed here, and tested before adding to the gitball.
Beta Live
All content is built here, with any code based changes being requested as needed. Testing is carried out here, responsive, UAT etc.
Code changes are pushed up via deployment scripts which trigger either a GIT pull, or by syncing files via ssh:rsync. heres a good resource on how to use git hooks for this.
Live
For live deployment, we use a WP database migration plugin to get content across, and the git repository to push tagged releases up to the live server. We also use the sync tool to pull down up to date copies of the database to our local dev machines.
I'm sure there are lots of other opinions on what's best, and how to implement them.
My opinion
The fact is is that WP is a bad choice for websites that require a lot of under the hood attention. It was never designed for this, and consequently is a real pain to force into a proper development cycle. If I must use a CMS, I generally use Drupal for sites that need more functionality, its a steeper learning curve, but more than worth it.
You could follow a git flow pattern. This consist of three main branches:
master
release
develop
Preform all of your development in the develop branch which you would build and test locally. Then when you are ready to test on a server merge your changes into the release branch for testing. When everything checks out you can then merge into your master branch. From here you can either manually ftp the files from the master branch like you have been or set up a deployment hook to automate deployment to your server when changes are detected.
As your project grows you can implement a couple of other branches to further maintain your build. You can base a new branch feature off of develop when you need to add a new feature then merge it back into develop following the flow back up again. As well as a hotfix branch to take care of any bugs "on the fly" straight off the master branch.
Here is a good example of the git flow pattern:
[http://nvie.com/posts/a-successful-git-branching-model/][1]
This article goes into a lot more detail about the git flow pattern and is an awesome resource!
Be sure that whenever you start a new development phase you merge from master to develop so you know you are starting with the most recent version of your project.
I am working on a ASP.net MVC4 project where a same project needs to be deployed to many clients on daily basis, each client will have its own domain / sub domain and a separate app pool and db (MSSSQL).
Doing each deployment manually could take at least 1-2 hours if everything goes well. Is there anyway using which I can do this in some automated way?
Moreover, we also need to update all of the apps when a new version is released.. may be one by one or all of them at same time. However, doing this manually could take weeks and once we have more clients then it will not possible doing this update manually.
The update involves, suspending app for some time, taking a full backup of files and db, update application code/ files in app folder, upgrade db with a script and then start app, doing some diagnosis script to check if update was successful or not, if not we need to check what went wrong?
How can we automate this updates? Any idea would be great on how to approach this issue.
As a developer for BuildMaster, I can say that this scenario, known as the "Core Version" pattern, is a common one. If you're OK with a paid solution, you can setup your deployment plans within the tool that do exactly what you described.
As a more concrete example, we experience this exact situation in a slightly different way. BuildMaster has a set of 60+ extensions that rely on a specific SDK version. In our recent 4.0 release, we had to re-deploy every extension because of breaking API changes within the SDK. This is essentially equivalent to having a bunch of customers and deploying to them all at once. We have set up our deployment plans such that any time we create a new release of the SDK application, we have the option to set a variable that says to build every extension that relies on the SDK:
In BuildMaster, the idea is to promote a build (i.e. an immutable object that travels through various environments like Dev, Test, Staging, Prod) to its final environment (where it becomes the deployed build for the release). In your case, this would be pushing your MVC application to its final environment, and that would then trigger the deployments of all dependent applications (i.e. your customers' instances of your application). For our SDK, the plan looks like this:
For your scenario, you would only need the single action, "Promote Build". As I mentioned before, any dependents would then be promoted to their final environments, so all your customer deployments would kick off once that action is run during deployment. As an example, our Azure extension's deployment plan for its final environment looks like this (internal URLs redacted):
You may have noticed that these plans are marked "Shared", which means every extension we have has the exact same deployment plan, but utilizes different variables to handle the minor differences like names, paths, etc.
Since this is such an enormous topic I could go on for ages, but I think that should be sufficient for your use-case if you wanted to try it out.
There are others but you could setup Team Server Foundation to deploy automated builds.
http://msdn.microsoft.com/en-us/library/ff650529.aspx
I find the easiest way to do this from an MVC project is to create a publish profile.
This is done by right-clicking your project selecting publish and then configuring it to your needs.
Then from TFS you create a new build definition, this kicks of a wizard which takes you through it.
There are quite a few options which would be too long to go into for every scenario.
The main change I usually find the most important is to set an MSBuild Argument to deploy with the publish profile.
This can be found at Process > Advanced > MSBuild Arguments.
Once this is configured correctly it's a simple case of right-clicking and queue new build to build and deploy.
You wil need different PublishProfile/Build configuration per deployment environment.
For backups I use a powershell script which can be called manually or from TFS.
You also have a drop folder in TFS which keeps a backup of x many releases.
The datbases are automatically configured via Sql server to backup, TBH I didn't set that up it was a DB admin guy who is also involved with releases.
From a dev testing side I use jMeter (http://jmeter.apache.org/) to run some automated scripts that check that users can login and view certain screens, just to confirm nothing major has gone wrong. However there is usually a testing team to run more detailed tests, again not setup by me.
All of the above will probably take you sometime to setup but in the long run it will literally save you weeks of time over a year.
A free alternative to TFS is http://www.cruisecontrolnet.org/, I have used this in the past too and is pretty good.
You can automate your .Net deployments with Beanstalk, which will give you a way to trigger deployments with a single click, watch progress, manage permissions and see history of deployments. Check out this guide on the topic:
http://guides.beanstalkapp.com/deployments/deploy-dotnet.html
I hope you will find it useful.
P.S. - I work at Beanstalk.
I am new to web development and am unfamiliar with some of the methods for best testing the web end information. On my current project I am using Weblogic to deploy my JSP project through Eclipse.
The webpages build fine and everything works, but currently I am restarting my localhost server every time I make a change to the css or the jsp, which a single round trip can be up to 3-5 min. is there a more efficient way of testing the css other than restarting my server every time? What about jsp?
Edit: Followup:
For any that stumble past this: After reading up on different staging methods as #better_use_mkstemp suggested, I couldn't figure out how to explicitly set eclipse up to stage or nostage deploy, however I did find some nifty things.
You can do an Incremental push (expand drop down in the servers view in eclipse, right click project that is deployed on a running server > click "Incremental Publish". This only pushes what has changed since last push) which helps a bit.
The other thing I found was an automatic deploy when a file changes, but that had it pushing while I was still editing the file which was a bit annoying. Using this feature you can also have it auto deploy each time there is a new version, but I didn't play around with that since we're not changing the version number yet.
Check how you are deploying your application. If you deploy using nostage mode, the server(s) will keep looking in the original deployment directory and will automatically detect jsp changes for refreshes.
This is as opposed to stage mode, where the original war/ear deployment is actually copied to each individual server.
Read more here: http://docs.oracle.com/cd/E11035_01/wls100/deployment/deploy.html#wp1027044
Sounds like if you keeps dropping your updated deployment files in the same directory with nostage you won't need a restart.
For our company websites we develop in a style that would be most accurately called Continuous Deployment (http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/)
We have a web farm and every site is located on two servers for fail over purposes. We wish to automate the process of deploying to our dev server, test server, and prod servers. Out websites are ASP.Net websites (not web applications) so we don’t do builds, we just push the pages. We generally change our main site anywhere from 2-20 times a day depending on many variables. These changes can be simple text/html changes to adding new pages/functionality.
Currently we use TFS for our source control, and each site is its own repository. We don’t have branches, you manually push your changes to dev (we develop on local machines) when it needs to be reviewed, then after sign off you check in your changes and push them to prod manually (this ensures that the source files are what is actually on prod and merges happen right then). Obviously there is a lot of room for error in here and it happens, infrequently, where a dev forgets a critical file and brings down the entire site.
It seems that most deployment tools are built upon a build process, which isn’t something we are likely to move to since we need to be extremely agile in meeting our business needs. What we’d really like, we think, is something like the following:
Dev gets latest version of website from source control, which is essentially a copy of production
Dev does work on his own box.
When it is ready to be reviewed by task owner he creates some form of changeset (either checking in the change or something else) and a tool would then push that change to dev
After review (and further possible changes) dev then is able to have the tool push all the changes for this task to test (UAT)
After this sign off the changes are automatically pushed to prod and checked into the prod branch, or whatever, for anyone else to get
We’ve toyed around with the idea of having three branches of the code in TFS, but we aren’t sure how you would pull from one branch (prod) but check in to a dev branch (none of us are TFS gurus).
So, how have other people handled this scenario? We are not wedded to TFS as a source control provider if there is something out there that would do what we wanted, nor do we require a vertical solution, we are more than happy to plug in different components as long as they can be plugged in, or even develop our own if we have to.
Thanks in advance.
I'm sure you can do this with TFS, but I've successfully done what you're describing with Mercurial, and it works pretty well.
With Mercurial (I'd assume Git is the same way), you can pull from one repo, and push to another. So you'd pull from Prod, do your change, commit locally, push to Dev, and then whenever you're happy, push the changesets into Prod (either from Dev or from Local, doesn't matter).
To get started, you'd have a Prod repo, then clone it into a Dev repo - both of these are on your server. At this point they're identical.
TortoiseHG on Windows gives you the ability to easily choose which repository you're synchronizing with. You can keep both the Prod and Dev repos in your list, and decide which one to use when it's time to pull or push.
The process of creating a new build and releasing it to production is a critical step in the SDLC but it is often left as an afterthought and varies greatly from one company to the next.
I'm hoping people will share improvements they have made to this process in their organisation so we can all takes steps to 'reduce the pain'.
So the question is, specify one painful/time consuming part of your release process and what did you do to improve it?
My example: at a previous employer all developers made database changes on one common development database. Then when it came to release time, we used Redgate's SQL Compare to generate a huge script from the differences between the Dev and QA databases.
This works reasonably well but the problems with this approach are:-
ALL changes in the Dev database are included, some of which may still be 'works in progress'.
Sometimes developers made conflicting changes (that were not noticed until the release was in production)
It was a time consuming and manual process to create and validate the script (by validate I mean, try to weed out issues like problem 1 and 2).
When there were problems with the script (eg the order in which things were run such as creating a record which relies on a foreign key record which is in the script but not yet run) it took time to 'tweak' it so it ran smoothly.
It's not an ideal scenario for Continuous Integration.
So the solution was:-
Enforce a policy of all changes to the database must be scripted.
A naming convention was important for ensuring the correct running order of the scripts.
Create/Use a tool to run the scripts at release time.
Developers had their own copy of the database do develop against (so there was no more 'stepping on each others toes')
The next release after we started this process was much faster with fewer problems, indeed the only problems found were due to people 'breaking the rules', eg not creating a script.
Once the issues with releasing to QA were fixed, when it came time to release to production it was very smooth.
We applied a few other changes (like introducing CI) but this was the most significant, overall we reduced release time from around 3 hours down to a max of 10-15 minutes.
We've done a few things over the past year or so to improve our build process.
Fully automated and complete build. We've always had a nightly "build" but we found that there are different definitions for what constitutes a build. Some would consider it compiling, usually people include unit tests, and sometimes other things. We clarified internally that our automated build literally does everything required to go from source control to what we deliver to the customer. The more we automated various parts, the better the process is and less we have to do manually when it's time to release (and less worries about forgetting something). For example, our build version stamps everything with svn revision number, compiles the various application parts done in a few different languages, runs unit tests, copies the compile outputs to appropriate directories for creating our installer, creates the actual installer, copies the installer to our test network, runs the installer on the test machines, and verifies the new version was properly installed.
Delay between code complete and release. Over time we've gradually increased the amount of delay between when we finish coding for a particular release and when that release gets to customers. This provides more dedicated time for testers to test a product that isn't changing much and produces more stable production releases. Source control branch/merge is very important here so the dev team can work on the next version while testers are still working on the last release.
Branch owner. Once we've branched our code to create a release branch and then continued working on trunk for the following release, we assign a single rotating release branch owner that is responsible for verifying all fixes applied to the branch. Every single check-in, regardless of size, must be reviewed by two devs.
We were already using TeamCity (an excellent continuous integration tool) to do our builds, which included unit tests. There were three big improvements were mentioning:
1) Install kit and one-click UAT deployments
We packaged our app as an install kit using NSIS (not an MSI, which was so much more complicated and unnecessary for our needs). This install kit did everything necessary, like stop IIS, copy the files, put configuration files in the right places, restart IIS, etc. We then created a TeamCity build configuration which ran that install kit remotely on the test server using psexec.
This allowed our testers to do UAT deployments themselves, as long as they didn't contain database changes - but those were much rarer than code changes.
Production deployments were, of course, more involved and we couldn't automate them this much, but we still used the same install kit, which helped to ensure consistency between UAT and production. If anything was missing or not copied to the right place it was usually picked up in UAT.
2) Automating database deployments
Deploying database changes was a big problem as well. We were already scripting all DB changes, but there were still problems in knowing which scripts were already run and which still needed to be run and in what order. We looked at several tools for this, but ended up rolling our own.
DB scripts were organised in a directory structure by the release number. In addition to the scripts developers were required to add the filename of a script to a text file, one filename per line, which specified the correct order. We wrote a command-line tool which processed this file and executed the scripts against a given DB. It also recorded which scripts it had run (and when) in a special table in the DB and next time it did not run those again. This means that a developer could simply add a DB script, add its name to the text file and run the tool against the UAT DB without running around asking others what scripts they last ran. We used the same tool in production, but of course it was only run once per release.
The extra step that really made this work well is running the DB deployment as part of the build. Our unit tests ran against a real DB (a very small one, with minimal data). The build script would restore a backup of the DB from the previous release and then run all the scripts for the current release and take a new backup. (In practice it was a little more complicated, because we also had patch releases and the backup was only done for full releases, but the tool was smart enough to handle that.) This ensured that the DB scripts were tested together at every build and if developers made conflicting schema changes it would be picked up quickly.
The only manual steps were at release time: we incremented the release number on the build server and copied the "current DB" backup to make it the "last release" backup. Apart from that we no longer had to worry about the DB used by the build. The UAT database still occasionally had to be restored from backup (eg. since the system couldn't undo the changes for a deleted DB script), but that was fairly rare.
3) Branching for a release
It sounds basic and almost not worth mentioning, yet we weren't doing this to begin with. Merging back changes can certainly be a pain, but not as much of a pain as having a single codebase for today's release and next month's! We also got the person who made the most changes on the release branches to do the merge, which served to remind everyone to keep their release branch commits to an absolute minimum.
Automate your release process whereever possible.
As others have hinted, use different levels of build "depth". For instance a developer build could make all binaries for runnning your product on the dev machine, directly from the repository while an installer build could assemble everything for installation on a new machine.
This could include
binaries,
JAR/WAR archives,
default configuration files,
database scheme installation scripts,
database migration scripts,
OS configuration scripts,
man/hlp pages,
HTML documentation,
PDF documentation
and so on. The installer build can stuff all this into an installable package (InstallShield, ZIP, RPM or whatever) and even build the CD ISOs for physical distribution.
The output of the installer build is what is typically handed over to the test department. Whatever is not included in the installation package (patch on top of the installation...) is a bug. Challenge your devs to deliver a fault free installation procedure.
Automated single step build. The ant build script edits all the installer configuration files, program files that need changed ( versioning) and then builds. No intervention required.
There is still a script run to generate the installers when it's done, but we will eliminate that.
The CD artwork is versioned manually; that needs fixed too.
Agree with previous comments.
Here is what has evolved where I work. This current process has eliminated the 'gotchas' that you've described in your question.
We use ant to pull code from svn (by tag version) and pull in dependencies and build the project (and at times, also to deploy).
Same ant script (passing params) is used for each env (dev, integration, test, prod).
Project process
Capturing requirements as user 'stories' (helps avoid quibbling over an interpretation of a requirement, when phrased as a meaningful user interaction with the product)
following an Agile principles so that each iteration of the project (2 wks) results in demo of current functionality and a releasable, if limited, product
manage release stories throughout the project to understand what is in and out of scope (and prevent confusion abut last minute fixes)
(repeat of previous response) Code freeze, then only test (no added features)
Dev process
unit tests
code checkins
scheduled automated builds (cruise control, for example)
complete a build/deploy to an integration environment, and runs smoke test
tag the code and communicate to team (for testing and release planning)
Test process
functional testing (selenium, for example)
executing test plans and functional scenarios
One person manages the release process, and ensures everyone complies. Additionally all releases are reviewed a week before launch. Releases are only approved if there are:
Release Process
Approve release for a specific date/time
Review release/rollback plan
run ant with 'production deployment' parameter
execute DB tasks (if any) (also, these scripts can be version and tagged for production)
execute other system changes / configs
communicate changes
I don't know or practice SDLC, but for me, these tools have been indispensible in achieving smooth releases:
Maven for build, with Nexus local repository manager
Hudson for continuous integration, release builds, SCM tagging and build promotion
Sonar for quality metrics.
Tracking database changes to development db schema and managing updates to qa and release via DbMaintain and LiquiBase
On a project where I work we were using Doctrine's (PHP ORM) migrations to upgrade and downgrade the database. We had all manner of problems as the generated models no longer matched with the database schema causing the migrations to completely fail half way.
In the end we decided to write our own super basic version of the same thing - nothing fancy, just up's and down's that execute SQL. Anyway it worked out great (so far - touch wood). Although we were reinventing the wheel slightly by writing our own, the fact that the focus was on keeping it simple meant that we have far less problems. Now a release is a cinch.
I guess the moral of the story here is that it is sometimes OK to reinvent the wheel some times as long as you are doing so for a good reason.