Here is what I am trying to do: I have my code sitting on Bitbucket (it is a ASP.net web application). I want to put together a build machine that has a web-based dashboard. I go to the dashboard and I say: Ok show me all the branches and stuff on my Bitbucket project. Now please get the latest code from this branch for me and build it. Then please deploy it to this location for me or maybe this other location. I want this dashboard give me a history of all this builds and deployments too. I am very new to this concept I have been reading about CC.net and MSBuild and other stuff but I can not find the right answer. Please put me in the right direction.
The point of a build server is that it automatically runs a build each time you commit something to your repository.
In order for the build server to know exactly what to do, you normally put a build script (with MSBuild or NAnt) into your solution which does everything you want - building your solution, maybe create a setup package and so on.
The build server basically knows where the repository is and where in the repository your build script is.
You need to configure this once in the build server, and then it will always run after you commit (but you can also start a build manually, if you want).
If you want a solution with web-based dashboard, try TeamCity.
It's a commercial product, but it's free for up to 20 users.
You can do everything in the web interface - configuration, running the builds AND browsing the build history.
EDIT:
Houda, concerning your question about deployment:
I don't think that TeamCity has a "deployment mode" in that sense. What you could do is include the deployment stuff in your build script that is run by TeamCity.
So, after the build itself is finished, copy the generated assemblies and files on your web server(s).
If you do it this way, you HAVE to make sure in the build script that the deployment will only happen if the build didn't fail (and if you have unit tests, if the unit tests didn't fail as well).
This is very important for a live application, because if you don't take care of this well enough, your app will go immediately offline every time someone commits "bad" code to your repository (and it will stay offline until the next "good" commit)!!
EDIT 2:
See Lasse V. Karlsen's comment below: it's more comfortable with the new TeamCity version 6.
Related
I am a front-end developer. I do a lot of PHPs/CSS/JS and HTMLs. Currently, how we do our deployment to staging environment is to push our codes to GIT servers. Go to our staging servers and do a pull to some directory. And then manually move the files from the directory to the correct directories in our apache web server.
Will it be overkill if I use TeamCity to do this? I intend to write an ANT script that does the copying which means to say Runner type will be ANT. So every time there is a push to the GIT repo, Teamcity will pull and then run the ANT script to copy the affected codes to the correct directories.
If not, I will gladly love to listen to any suggestions.
Thanks
Teamcity may be overkill right now, as you would just be using it as a fancy trigger for your build.
But consider adding custom build parameters, which it can pass to your script. You can then start automating builds to different environments through a friendly UI.
You then have a platform to base a correct deployment process around further down the road.
Come the time when you need PHP compilation, JS minifcation, unit testing, its all just another step in your TC configuration.
I would recommend it.
Is there any way in ASP.NET website project, that allow to publish it from command prompt and I can continue working on project, or if it is not easy to use , at least speed up my publish task?
I know about the auto publishing tools like TFS or CruiseControl, so please don't tell me these ways.
I am thinking to create a .bat file , that I'll run everytime I have to publish. but it should not take changes made by me during its running process.
asp.net single file publish
I really like the Answer given by Ludwo, providing more information on that would be very helpful.
You can use MsBuild to publish your websites in parallel. Start with this article. It is about publishing one website using MsBuild. Define your projects inside ItemGroup and use MSBuild task this way:
<MSBuild Projects="#(YourProjectsToBuildInParallel)" BuildInParallel="true" ...
The final step is to enable parallel processing for MSBuild task.
Open another Visual Studio to continue :P. Publishing mechanism can detect updated and can send only changes. So dont upload full site everytime, if its really disturbs you.
Use source control and a build server mechanism. The build server should be able to pull from source control when you commit a change, build the project, do any unit tests you may/should have, and then deploy to a test site.
Depending on which build server platform you use you may or may have to do varying amounts of work. In the past I have used Bamboo by Atlassian. Fantastic product but you have to configure the deployment using MSBuild - it's fine but it can take some time to get it perfect. I am sure there are some good examples out there for it.
How it will work for you:
When you are finished working on a file/issue you can commit your changes. The build server will then detect these changes and wait a varying amount of time (waiting for you to commit more) e.g. 3 minutes, check out your changes, and deploy. You can set up notifications when the deployment is done to goto your testing team - with a link in the email saying where the site is, and what the change that occurred (based on your SVN commit log).
So your net effort is to check a file in with a correct comment - and you are finished.
At work we currently use the following deployment strategy:
Run a batch script to clear out all Temporary ASP.NET files
Run a batch script that compiles every ASPX file into its own DLL (ASP.NET Web Site, not Web Application)
Copy each individually changed file (ASPX and DLL) to the appropriate folder on the live server.
Open up the Deployment Scripts folder, run each SQL script (table modifications, stored procs, etc.) manually on the production database.
Say a prayer before going to sleep (joking on this one, maybe)
Test first thing the next morning and hope for the best - fix bugs as they come up.
We have been bitten a few times in the past because someone will forget to run a script, or think they ran something but didn't, or overwrote a sproc related to some module because there are two files (one in a Sprocs folder and one in a [ModuleName]Related folder) or copied the wrong DLL (since they can have the same names with like a random alphanumeric number generated by .NET).
This seems highly inefficient to me - a lot of manual things and very error prone. It can sometimes take 2-3 or more hours for a developer to perform a deployment (we do them late at night, like around midnight) due to all the manual steps and remembering what files need to be copied, where they need to be copied, what scripts need to be run, making sure scripts are run in the right order, etc.
There has got to be an easier way than taking two hours to copy and paste individual ASPX pages, DLLs, images, stylesheets and the like and run some 30+ SQL scripts manually. We use SVN as our source control system (mainly just for update/commit though, we don't do branching) but have no unit tests or testing strategy. Is there some kind of tool that I can look into to help us make our deployments smoother?
I did not go through all of it, but the You're deploying it wrong series from Troy Hunt might be a good place to look at.
Points discussed in the series :
Config transforms
Build Automation
Continuous Integration
We have four stages before it can be deployed.
Development
QA
UAT
Production
We have build scripts (inside bamboo build server) running against QA and against UAT. Our DBA is the only person who can run create scripts against QA, UAT, and PROD. Anything that goes from QA -> UAT is like a test run deployment. UAT gets reverted by copying the production systems down again.
When we release into Production we just create a whole new site and point it at the UAT database and test that environmentally it is working fine. Then when this is working good we flick the 'switch' and point the production IIS record at the next site, and change the DB connection to point at Prod DB.
Because we are using a completely diff folder structure all of our files get copied up so there is no chance of missing one. Because we have had test runs of deployment into UAT we know we haven't missed a DB script (DB Scripts are combined into one generally). Because we have tested a shadow copy of the IIS website we know that environmentally it should work. We can then do all this set up during the day - and then do the final switch flicking at midnight or whenever - reducing the impact on devs.
tl;dr; Automated build and deploy; UAT system for test running deployment; Deployment during work hours; Flick switch/run DB update at midnight.
I am a developer for BuildMaster, a tool which can very easily automate the steps you have outlined above, and we have a limited version free for a team of 5 developers.
Most of your pain points will disappear the moment you set up the deployment automation - mainly the batch script execution and the file-by-file copying. Once you're fully automated, you can even schedule the deployment for night time and only have to worry about it if there's an error in the process (you can set up a notifier for a failed build).
On the database side, you can integrate your database with BuildMaster as well and if you upload the scripts into the tool it will keep track of which ones were run against which database.
To see how to set up a simple web application deployment plan, you can run one of the example applications included. You can also check out: http://inedo.com/support/tutorials/lunchmaster/part-1 to see how to create one yourself - it's slightly outdated since we've made it even easier to get started out-of-the-box but the main concepts are the same.
Please see this blog post and associated talk by Scott Hanselman titled "Web Deployment Made Awesome"
Blog
Video
As for SQL Deployment, you might want to consider one of the following:
RikMigrations
Migrator.NET
FluentMigrator
Mantee Introduction & Source
Have a User Acceptance Testing (UAT) environment which is completely isolated from your development environment, and only accessible to the UAT manager.
Setup a UAT build which you can manually trigger upon each release, when triggered this should send all your deployment files as well as a deployment checklist to the UAT manager, who will redeploy all files to the UAT environment, and run any database upgrade scripts.
Once the applications users and testers have signed off the UAT release, the UAT manager can be authorised to deploy to the PRODUCTION environment using the exact same procedure and checklists as the UAT release. This will guarantee that you never miss any deployment steps, and test the deployment process prior to moving it into production.
Caveats- I'm in an environment where we can't use MSI, batch, etc for the final deployment
Things that helped:
A build server that does the full compilation on a build server and runs all unit tests and integration tests. Why find out you have something in an aspx page that doesn't compile on deployment night? (I admit your Q doesn't make it clear if compilation is happening on deployment night)
I have a page that administrators can reach that exercises environment and deployment failure points, e.g. connect to db, connect to reporting services, send an email, read and write to the temp folder.
Also, put all the things that the administrator needs to change into a file external from web.config. The connection string and app settings sections natively support a way to do this (i.e. don't reinvent the web.config system just to create a separate file)
Here is an article on how to do better integration tests: http://suburbandestiny.com/Tech/?p=601 There is a ton of good literature how to do unit tests, but often if you app already exists, you will have to refactor until unit testing becomes possible. If that isn't an option, then don't be a purist and put together some integration tests that are fast and repeatable as possible.
Keep your dependencies in bin instead of GAC, since it's easier to tell an administrator to copy files than it is to teach them to administer the GAC.
I'm fairly new to Subversion. Most of my work so far has been with Visual Source Safe. I'm looking to improve my deployment process with SVN and TeamCity. This is my plan:
There would be three branches:
Development (/trunk) - Entire ASP.NET solution, including a Web Deployment Project.
Staging (/branches/staging) - Web Deployment Project output (files needed for execution only - bin, .aspx, images, etc)
Deployment (/branches/deployment) - same as Staging
The CI process:
Commit source changes in trunk.
TeamCity detects the change, builds the solution and runs unit tests.
If all tests pass, TeamCity commits Web Deployment Project output to branches/staging and exports it to wwwroot on the staging web server.
Then when I'm ready to deploy to production, I'll do the following manually:
Merge branches/staging with branches/production
Update production web server's working copy of branches/production.
Does this make sense? Is there anything that a VSS user like myself might be missing/misunderstanding in this process?
I've written a pretty long post on how to do this with ASP.Net and web deployment projects - sounds right up your alley (don't know if I'm allowed to post this - mods?):
http://www.diaryofaninja.com/blog/2010/05/09/automated-site-deployments-with-teamcity-deployment-projects-amp-svn
Late answer, but can be helpful to the readers of this thread:
I have done a little online research and found a step-by-step tutorial that may help with your problem.
This tutorial covers the basics of the continuous integration (CI) and the means to build a new database whenever a new change is detected on the source control repository, run specified unit tests against the database, and synchronize tested database to QA environment.
The necessary prerequisite to implement the continuous integration (CI) in your database development process is to have a database under source control.
This might work for you, but usually the stage is the place where customers accept changes.
If you deploy on each build they dont get a consistent behaviour.
We dont keep the build result in SVN. For us it was ok to just have it in Teamcity under artifacts. I am not shure if we use best practice at this point.
You will be so much happier with SVN and Teamcity...good luck!
Seems like there are so many different ways of automating one's build/deployment that it becomes difficult to parse through all the different scenarios that people support in tutorials on the web. So I wanted to present the question to the stackoverflow crowd ... what would be the best way to set up an automated build and deployment system using the following configuration:
Visual Studio 2008
Web Application Project
CruiseControl.NET
One of the first things I tried was to have CCnet automatically zip the output and copy it to the server, but then that requires manual work to unzip at the destination. However, if we try to copy all the files individually, then it could potentially take a long time if it's a large application (build server lives outside of the datacenter in our office ... I know).
Also of particular interest is how we would support multiple environments as we have dev, qa, uat, and then of course prod.
MSDeploy seems really interesting, but unless I'm interpreting the literature incorrectly, doesn't help in the scenario of deploying from the output of a build server. If anything, it seems like it'll be useful in deploying one build across a build farm ... but even for deploying from one environment to another, one would have to manually change config settings and web service URLs, etc.
I recently spent a few days working on automating deployments at my company.
We use a combination of CruiseControl, NAnt, MSBuild to generate a release version of the app. Then a separate script uses MSDeploy and XCopy to backup the live site and transfer the new files over.
Our solution is briefly described in an answer to this question Automate Deployment for Web Applications?
You might be interested in MSDeploy. Here's a Scott Hanselman post on this. It's only available as a technical preview at the moment (September 2008) but is worth evaluating against your requirements.
There is another new build tool (a very intelligent wrapper) called NUBuild. Its lightweight, open source and extremely easy to setup and provides almost no-touch maintenance. I really like this new tool and we have made it standard tool for our continuous build and integration process of our projects (we have about 400 projects across 75 developers). Try it out.
http://nubuild.codeplex.com/
Easy to use command line interface
Ability to target all .Net framework
version i.e. 1.1, 2.0, 3.0 and 3.5
Supports XML based configuration
Supports both project and file
references
Automatically generates the “complete
ordered build list” for a given
project – No touch maintenance.
Ability to detect and display
circular dependencies
Perform parallel build -
automatically decides which of the
projects in the generated build list
can be built independently.
Ability to handle proxy assemblies
Provides visual clue to the build
process e.g. showing “% completed”,
“current status” etc.
Generates detailed execution log both
in XML and text format
Easily integrated with
Cruise-Control.Net continuous
integration system
Can use custom logger like XMLLogger
when targeting 2.0 + version
Ability to parse error logs
Ability to deploy built assemblies to
user specified location
Ability to synchronize source code
with source-control system
Version management capability
Do you have the ability to run commands remotely? The PsExec utility from Systinternals would let run a command line unzip program on the remote machine. If you have a script that copies the build as a .zip file to the remote site, you would just need one more line for the PsExec call to unzip the files.
I had a related question about getting a deployable set of files from an automated build. I found Web Deployment Projects (links and all in the old question) did what I needed - they're a VS and MSBuild add-on.
This is a common problem (and I wish I had read it sooner) for all development, not just ASP.NET. Being one of its developers, my team naturally uses BuildMaster internally for the entire release process, and for most scenarios it's free. Within the tool, we are able to perform all the standard CI builds to create artifacts and then set up an automation process to deploy these artifacts to any one of the 40+ servers we have internally or externally hosted, depending on the specific application or environment.
Since you specifically mentioned deployment to different testing environments, this is a fundamental aspect of the tool. The idea is to model the environment workflow (e.g. Integration -> QA -> Production) you already have in place and essentially promote a build all the way from source control to production. Most times, it's as simple as adding a deployment action that deploys an artifact to the environment, other times it can be much more complex.
You also casually mentioned configuration file changes are part of deployment, which is another built-in component to BuildMaster. The idea we had was to use the tool itself as the central hub for all configuration files and deployments, thus ensuring the latest changes are applied automatically with a simple "deploy configuration files" action in your deployment plan.
One thing you didn't mention with regard to this process is the database deployment aspect. Most ASP.NET applications require an associated database, otherwise they could just be static HTML files. It is crucial that the database schema gets updated to the appropriate database version with every deployment. There is, not surprisingly, a module within BuildMaster that handles this for you as well. The idea is to store DDL-DML scripts within the tool itself, and by executing scripts only once per environment, it ensures that all of your databases across each environment are up-to-date as your builds are deployed through them. Other scripts (e.g. stored procedures, views, triggers, etc.) are essentially code files and therefore belong in source control. These DROP-CREATE-CONFIGURE type scripts can be run each and every time in most cases with a simple deployment action.
Another piece of the deployment puzzle that most developers do not think about is process automation. Many developers need to perform sign-offs or fill out change request forms in order to manually perform these processes. Again, this is all available as part of the automated workflow setup within BuildMaster. You can setup blockers that do not allow promotion to say the QA environment unless all unit tests have passed, or block promotion to the Staging environment unless someone from the QA team approves the build and all issues in your issue tracking tool are resolved/closed for that particular release.
While I realize I left out CC.NET from the answer, our applications are all built and deployed through BuildMaster so we no longer need it, though we could however just as easily pickup the artifacts from a drop location and deploy them in later environments.
I see that many people use CC for their .NET projects, but why not use Jenkins, Sonarqube? They got all you need. I setup all this in 3 days. I have a Win 2008 server R2, MSSQL, Jenkins, VIsual SVN and Sonarqube.
It all works great and u get all metrics on your project. Sonarqube uses Gallio, Gendarme, FXcop, Stylecop, NDepths and PartCover to get your metrics and all this is pretty straight forward since SonarQube do this automatically without much configuration.
i post som pictures for u too get a feeling for it. Here is Jenkins witch builds and get Sonar metrics and a another job for deploying automatically to IIS
And Sonarqube, all metrics for my project. This is a simple MVC4 app, but it works great!:
If you want more information i can be more specific but i think you should at least consider jenkins. If CC suites you better, at least you looked at good alternative before you chose.
This whole setup uses MSBuild, too build and deploy the apps.