I worked with Plone 2 several years ago, but its workflow and security systems (not to mention the new theming engine) seem the best fit for a current project. Plone 4 looks much better from a usability standpoint, but in its drive not to force a particular setup on users, it has become a bit of a maze as to how best to host it on our public-facing Ubuntu servers. I’ve got Plone installed using the Unified Installer (in /home/plone/Plone/zinstance), and have been running it in the foreground for development, but I’m now going to need to set things up for the move to making this website live.
I want to achieve a development instance on the project server that we can use for our own customisations (with additional local instances on our desktops for testing before checking into the development instance), then a site that our client can test things on before they go live, and finally the live website.
In particular:
I’m assuming that the live site will be based on ZEO, while testing and development will be standalone; does that make sense?
At present, everything lives in /home/plone/Plone, while all our other (generally PHP CMS) sites are hosted from /srv/<domain>/www; everything for the domain is then backed up in one go. What’s the best way to layout the different Plone instances to fit into this?
What’s the best way to move our changes to buildouts and products through the cycle from development to testing and then to a live site? We currently have a basic deployment process using Mercurial for other systems.
Current best practice is to use buildout to define a fully reproducible deployment.
Generally, you use a production.cfg and a development.cfg configuration that share configuration via inclusion, to tweak the deployment to the specific needs of either environment; you can expand on this model as needed. A big project might have a staging.cfg to deploy new features to a test environment first, for example.
Whether or not you use ZEO in development depends entirely on your use cases. Most development setups certainly do not need it, but in a large deployment that includes async workers you may find you need a ZEO server when developing anyway. Again, buildout will make switching your development environment around for changing needs easier, especially when combined with supervisord to manage the various daemons used in the setup.
For the biggest deployment I manage, we use a combination of subversion and git, but only for historic reasons. This is all being moved to one git repository. In the end, the repository will hold the complete buildout with it's various configurations, development.cfg, staging.cfg and per-production-machine configuration files (one for each server in the production cluster). For a 3-machine ZEO setup, that'd be one zeo.cfg and instances-1.cfg and instances-2.cfg, for example. The core development eggs are stored in the same repository in src/.
Development takes place exclusively in per-feature and per-issue branches. Once complete, a branch is merged into the staging branch, and the staging server is updated and restarted. Once the customer has signed off on each merged branch, and a roll-out has been planned, we merge the approved branches to the main branch and tag a release. The production cluster machines are then switched to that new tag and restarted.
We also use Jenkins for continuous integration testing; both the main and the staging branches are put through their tests at least once a day, letting us catch problems early.
All daemons are managed by a dedicated supervisord, which is started via a crontab entry (#reboot), not the native OS init.d structure. This way we can manage the running daemons entirely with buildout. The buildouts even generate logrotate configurations and munin monitoring plugins; these are symlinked into the OS locations as needed.
Because a buildout is entirely self-contained, it doesn't matter where you set up the production buildout on a machine. Do pick something consistent, document it and stick to that. Make sure you have sufficient disk space for your daemon needs and otherwise not worry too much about it.
ZEO or not - for production or development - depends on personal needs and preferences. Many devs use ZEO for production and development and many do not. ZEO is needed for scaling (multiple app servers)
Nobody cares...install Plone where you want it or need it...re-running buildout after moving an installation fixes relocation issues. It is your system, not ours...you decide.
Usually buildout configurations live in a repository and can be pulled on the production server from the repository....otherwise you need to copy over the related buildout files yourself some. Typical installation works like this:
run virtualenv
checkout the buildout configuration from git/subversion
run bootstrap
run buildout
Related
I'm looking to implement failover (master/slave) for Karaf. Our current
server setup has two application servers that have a shared SAN disk where
our current Java applications are installed in a single location and can
be started on either machine or both machines at the same time.
I was looking to implement Karaf master/slave failover in a similar way
(one install being shared by both app servers), however I'm not sure that
this is really a well beaten path and would appreciate some advice on
whether the alternatives (mentioned below) are significantly better.
Current idea for failover:
Install Karaf once on the shared SAN and setup basic file locking on this
shared disk.
Both application servers will effectively initiate the Karaf start script,
however only one (the first) will fully start (grabbing the lock) and the
second will remain in standby until it grabs the lock (if the master falls
over)
The main benefit I can see from this is that I only have to manage
deploying components to one Karaf installation and I only need to manage
one Karaf installation.
Alternatives:
We install Karaf in two separate locations on the shared SAN and setup to
lock to the same lock file.
Each application server will have their own Karaf instance, thus start
script to run.
This will make our deployment slightly more complicated (2 Karaf
installations to manage and deploy to).
I'd be interested if anyone can indicate any specific concerns that they
have with the current idea.
Note: I understand that Karaf-cellar can simplify my Karaf instance
management, however we would need to undertake another round of PoCs etc..
to approve our company use of cellar (as a separate product). Something
that I'd like to migrate to in the future.
Take a look at the documentation
This is from the documentation on how to set a lockfile for HA:
karaf.lock=true
karaf.lock.class=org.apache.karaf.main.lock.SimpleFileLock
karaf.lock.dir=<PathToLockFileDirectory>
karaf.lock.delay=10000
as can be seen there, you can also set a level for the bundle start-levels to start or not to start:
karaf.lock.level=50
So I do create websites since some years already but I never cared about a good workflow. So I did bad things like working on the production server etc.
I want to improve all that and so I came across git and tools like wordmove (for wordpress).
I tried to visualize what I want or what I think could work:
workflow visualization
Now I think something there is "wrong" or "not so good" and can be done better but I dont really know what or how to do it.
So I have my local machine where I develop, then I have a bitbucket repository, a staging server to show the customer the current status and a production server which is the live server of the customer.
I'd appreciate some help :P How it can be understood.
From git 2.3, a feature named "Push to deploy" was added, you can search document for it or read here
What type of websites are you making? WordPress, Drupal, ect? It looks like you're on the right track.
As that diagram shows, I'd recommend creating a development, staging, and production branch for each project and setting up a webhook for the repo that listens for pushes and deploys (and builds) accordingly. This way you can deploy to private servers to "stage" your project/features for the client first before merging in to production.
stackahoy.io is built to do exactly this. It's free for 1 repo and unlimited branches. Some benefits to using Stackahoy are:
Maintain deployments for your git repositories in one place
Maintain static configuration files (stuff you keep in the .gitignore file)
Preform post-deployment scripts
Securely and instantly deploy your code based on the branch that was pushed and see real-time logs as it deploys.
Deploy to multiple servers at once (good for load-balanced applications)
Disclaimer: I work for Stackahoy and would be happy to answer any Q's.
The Setup:
I'm setting up a Wordpress-powered application using Elastic Beanstalk from Amazon Web Services. All development is being done locally under a MAMP apache2/php5 server environment with a GIT repository controlling the entire application root.
Deployment Workflow:
After committing any code changes (edits, new plugins, etc) to the repo the application is deployed using AWS EB CLI's eb deploy command which pushes the latest version out to any running EC2 instances managed by Elastic Beanstalk.
My Issue:
Sometimes the code changes aren't exactly syncing up between my development/production environments and I'm not sure how to overcome it. Especially when trying to install and setup plugins like W3 Total Cache or WP Super Cache.
Since my local environment doesn't have things like a memcahced server installed, but my production environment does (ElastiCache) I'm unable to save the proper settings file and deploy it for use in my production environment. These plugins won't allow me to select the needed services because it sees them as not available...
It seems I can only get W3 Total Cache to work if I install it directly onto a live production environment, which seems like a bad idea.
Given the above:
Am I going about deployments the wrong way?
Should plugins like W3 Total Cache be installed and configured on
local development environments and pushed to production environments?
I cannot comment on the issues specific to Elastic Beanstalk, but based on experience I can make a suggestion about the second part of your issue statement:
You are better off running a development environment that mirrors your production environment as closely as possible. I suggest that you convert from MAMP to a VM environment like VirtualBox. You might want to check out puphpet.com for help in getting it set up. It requires some startup effort, but gives you an environment similar to or the same as your production servers. For example, you could run memcached yourself so you could actually test it with W3 Total Cache.
As for your second question, just installing a plugin in the production environment without testing it beforehand has obvious risks (but then again clients do that all the time). I would prefer to test first. To a certain extent it probably depends on how critical it is if the site experiences downtime or weirdness.
I would suggest you to create another environment on Beanstalk.
It's easy, fast and more reliable than a VM in your case because it will allow you to test your deployment process as well.
I usually have 3 environment for a every website. Each environment is on its own branch. If your configuration is different between environment (url and database access for example), just store your wp-config and other config files into S3 (you may not want production password in your git repository), and through ebextensions you can download them into your website automatically.
I use AWS Beanstalk that way for 16 websites and some are wordpress one. All with autoscaling and able to get thousands of users simultaneously.
Don't hesitate to ask me for further details.
I'm accustomed to writing desktop applications in C#.NET. Where I have a nice little solution folder which is also under version control. So at any time, on any computer I can check out whatever version of my software I want run the compiler and have a working copy of my program.
Now I'm looking into developing websites, where the files and data are a lot more dispersed. I'm using ASP.NET, but really my question is more general and could apply to any website framework.
I'm trying to understand the proper work-flow between developing my website, a version control server, and the actual live website that users will see. Obviously this can vary a lot depending on the type and scale of the website, but I'm only considering a pretty simple site. I'm just getting started with this stuff.
The diagram below shows my current idea. All the source files for the site would be stored on a subversion server, which I would check out onto my local computer. My local computer would have a local database which I would use for development of the site. Next I would publish to a test version on my hosted server, which would point to a separate test database. This test database may periodically be replaced by a copy of the live database.
If all goes well I would then publish to a beta version of the site which points to the live data. Users could then check out the beta version to provide feedback. Finally if there are still no problems the source files for the live site would be updated.
Does this make sense? Does anyone have any comments on how this could be improved? Are there any good books or online tutorials available on developing these kind of workflows?
Also the one thing that I'm really not sure about is how to manage changes to the actual schema of the database. I figure with each version I could generate a SQL script that can be use to update the Test and Live databases on the host. However, I'd also like to be able to easily setup a new database for any version of my site with out having to run every update SQL script for every version up to the desired version. Is the best solution to use an ORM like NHibernate or Subsonic so I could always generate my database schema directly from my code?
I currently use a workflow very similar to this. It's not flawless, but for the most part it works well. It sounds like you pretty much have the important parts figured out.
Where I work we have a powerful web server. Our subversion repos also live on this server. For myself, as a LAMP developer, I do all of my development on my local Linux machine (or VM) with a local MySQL database, operating on a local working copy.
At all times I maintain two primary branches of the application: A Dev branch (trunk) and a Live branch (which reflects the current production version). My repo looks like this:
/repo/trunk/ [My current active development version]
/repo/archive/ [All other versions not in active development live here]
/repo/archive/2010.12/ [This happens to be the current production version]
On the web server we maintain three separate instances of the software: Live (or Production), Beta, and Dev. The following should illustrate how we use these versions to
Dev - Always points to our development version at `/repo/trunk'. Uses a non-vesioned config file to point to the development database. Development is never physically done here however; instead we work on our local machines, where our working copies point to the Trunk, and do all our development testing on our local machines. After several commits from multiple developers, though, it's important to test on the Dev server to make sure we're on the right track and no one is breaking someone else's changes.
Live - Always points to the most recent stable production version. In this case, it points to /repo/archive/2010.12/. This version is world-accessible.
Beta - Most of the time, Beta will be a mirror of whats on Live and points to the same production version in our repository (/repo/archive/2010.12). Anytime we need bug fixes on Live, we make them here, test them, and then commit & update live. (We also merge them into Trunk, if necessary.)
When the version in Trunk is deemed complete and ready for testing, I create a new archive branch in the repository for the upcoming new release (i.e. 2011.01) by svn copying the existing production branch. Then I merge the Trunk version into the new branch and commit, so we have a version that mirrors whats currently on Dev. Of course now active development for the next release can continue on Dev, while we Beta test this new Archived version on Beta. When beta testing is complete, we commit any fixes and switch Live to the new version (/repo/archive/2011.01).
Now, you've probably figured out that the database merging is a bit trickier. We use MySQL and, to my knowledge, there's no suitable versioning system for MySQL. So with each migration (VM to Dev, Dev to Beta, Beta to Live) I'll first make a backup of the current database, then make the necessary changes. Personally I use a commercial version of SQLYog which allows me to synchronize the database schema (super handy).
I've always deployed my web applications via FTP (sometimes even xcopy), and then manually run database scripts myself.
I started deploying this way in the 90's, but lately, I've seen a few web apps with installers. I'm starting to question, if I'm locked into an out dated process. I'm a consultant, my apps are usually internal, so I don't worry about distributing and having others installing them.
But I'm curious; does anybody create installers to deploy internal asp.net web applications?
If so, why? (Voluntarily, mandated, or part of an automation process)
And have you had any problems doing it this way?
absolutely. We use it to do all of our apps. That way we create the installer and run it on the qa and uat environments to test and we know exactly what is going to happen in production. There are no guesses as to what order someone might do something in, or if they miss a step. It makes things a lot easier.
Ooh I forgot about the automated process too. We have systems in place (Ant Hill Pro) which automatically deploy it to the proper environments. The qa people don't have to wait for something to be done, because it's all done at 2 am. If they need to rerun the build with updates, the devs check the code in and we push a button, and it's automatically deployed. No waiting for the build engineer, because he's in a meeting or sick or whatever.
You always want to have an automated way to build and deploy - it greatly reduces the chances of a one-off error if you forget a certain step. Also, it allows you to offload the deploy to someone else easily without having to teach them 100 customized steps. Whether the project is internal or not, all applications should follow best practices.
Personally I'm a bit like the OP; generally I just deploy using FTP, but in saying that typically my applications are internal, or in the case of other projects, 100% managed by me.
I've also been thinking about this lately however, and have started to think about how using proper deployment may improve the process - having to document a detailed install process can be a real pain.
I use Powershell and found really easy to automate lots of tasks. You will probably find a bit different at the very begining but at the end you will see that it's all about the power of the .NET libraries !!!
I have use the "Web Setup Project" to create an MSI that installed the output of a "Web Deployment Project" for an internal app. Our server admin wasn't up to the task to doing a 50 step manual install. For my current app, my server admin doesn't like the 'black box' feel of MSI installers and prefers getting a pile of files and a 50 step deployment manual. (See a pattern here? Ask your server admin what he wants.)
The Web Setup Project doesn't make it immediately obvious how to install to anything other than the "Default Website", other than that, it made the installation process repeatable and created a built in way to rollback (by just running the installer from 1 version ago).
This of course assumes that your virtual directory doesn't hold any user modified content-- I wouldn't trust an MSI to properly merge user created and new files.
We use the "XCopy" deploy model here, since the Ops folks have their own method of setting up security on a new web application on the server.
However, we did need to use an installer when we had to install a web application that was using a newer version of Crystal Reports since it had to do something special with a key and we didn't have a full blown version of CR on the server itself. So keep that in mind when working with third party apps, they may need to do some kind of merge module that the MSI handles easily.
Yep...we have an app that needs a lot of pre-requisites set up....web service, windows service, user accounts, security, folder creation, GAC bits etc....I rolled it all up into a nice MSI with custom actions that can install and uninstall cleanly. Saved about one hours worth of work to deploy on a new box.
A lot of the other smaller apps are just deployed by doing Publish Website to a local folder then ftp'ing the contents to the target.
It greatly depends upon the scale of your project, your enviornment and your internal user base. I rarely deploy with an msi because we are too small an operation to have multiple environments (except for SharePoint, that's different all together) . We develop and use VS to deploy web apps to a development box, assuming they are approved then we use VS again to deploy to the live box.
The only proviso is that we have multiple copies of the web.config (appended with test, dev and live) and we then delete the suffix off the relevant file depending upon where its been deployed.
It's probably not the best methodology (I know it's not), but it works and it aids rapid deployment of small to medium sized solutions in a small-scale user environment.
F5ToDebug...
Your saying its OK to take short cuts if you dont have time to do it properly?
"who's going to test the code on the test environment?" You said it yourself that you have config files for _test - why would that not be a suitable test?