How should I debug a Codeship build that runs successfully with jet steps locally? I make sure to run it on a fresh checkout, so it starts in the same state as Codeship. My process is:
Make fresh clone of repo
Call jet steps --push --tag master --no-cache (steps
will only push for master)
Running in Codeship, the same master continues to fail
Build Error: No source files were specified
2017-08-07T13:25:59.136Z my-image build/pull had an error
The docs say to use jet steps for debugging, but what should one do when this isn't helpful?
The answer is that I used false assumptions.
The truth of the matter is that running local with jets and remotely on Codeship are very different processes. Success locally does not portend success on Codeship.
jet will be successful, regardless of what volume configs you pass
jet does not run individual steps in isolated containers, Codeship does
jet will share state between steps and services, Codeship won't
Further, if you need state between steps and services, very carefully read the footnotes on the Codeship volumes docs.
Related
I am in the process of converting our legacy custom database deployment process with custom built tools into a full fledged SSDT project. So far everything has gone very well. I have composite projects that can deploy a base database as well as projects that deploy sample and test data.
The problem I am having now is finding a solution for running some sort of code that can call a web service to get an activation code and add it to the database as the final step of the process. Can anyone point me to a hook that I might be able to use?
UPDATE: To be clearer I am doing this to make it easier to maintain and deploy our sample and test data to a local machine. We can easily use Jenkins to activate the sites when they are deployed nightly to our official testing environments. I'm just hoping to be able to do this in a single step to replace the homegrown database deploy tool that we use now.
In my deployment scenario I wrapped database deployment process in some powershell scripts which do necessary prerequisites. For example:
powershell script is started and then it stops some services
next it run sqlpackage.exe or preproduced sql deployment scripts
finally powershell script starts services.
You can pass some parameters from powershell to sql scripts or sqlpackage.exe as sqlcmd variables. So you can call webservice first, then pass activation code as sqlcmd variable and use the variable in postdeployment script.
Particularly if it's the final step, I'd be tempted to do this separately, using whatever tool you're using to do the deployment: Powershell scripts, msbuild, TFS, Jenkins, whatever. Presumably there's also a front-end of some description that gets provisioned this way?
SSDT isn't an eierlegende Wollmilchsau, it's a set of tools for managing database changes.
I suspect if the final step were "provision a Google App Engine Instance and deploy a Python script", for example, it wouldn't appear to be a natural candidate for inclusion in an SSDT post-deploy script, and I reckon this falls into the same category.
As we all know, we can run a meteor app by just typing meteor in a terminal.
By default it will start a server and use port 3000.
So why do I need to deploy it using MUP etc.
I can configure it to use port 80 or use nginx to route to port 80 for the app. So the port is not the point.
Edit:
Assume meteor is running on a VPS or cloud server with public IP address, not a personal computer.
MUP does a few extra things you can do yourself:
it 'bundles' the code into a single file, using meteor build bundle
the javascript is one file, and css another; it's minified, and obfuscated so it's smaller and faster to load, and less easy to decipher on the client.
some packages are also meant to be removed when running in production. For example meteorToys, the utility toolset to look up collections and much more, is not bundled into the production bundle, as per the instructions in its package. This insures you don't deploy code with security vulnerabilities (Meteor toys basically opens up client side delete / updates etc... if you're not careful)
So, in short, it installs a minimal version of your site, making sure that what's meant for development only doesn't get push to a production environment.
EDIT: On other reason to do this, is that you don't need all the Meteor build tools on your production server; that can add up to a lot of stuff, especially if you keep caches going for a while...
I believe it also takes care of hooking up to a remote MongoDB Instance (at least it used to be the case on the free meteor site) which is more scalable and fault tolerant than running on the same instance as the web server, as well as provision storage etc... if needed.
basically, to deploy a Meteor app yourself manually, you need to:
on your dev box:
meteor build bundle your app to a tar file (using the architecture flag corresponding to the OS you will use)
on the server:
install node v0.10 (or whatever is the current version of node required by Meteor)
you might have to install Fiber#1.0.5 (but I believe this is now part of meteor install already)
untar the bundle, get into bundle/programs/server/ and run npm install
run the server with node main.js in the bundle folder.
The purpose of deploying an application is that you are situating your project on hardware outside of your local machine. For example if you deploy an application on Heroku app you create a repository on heroku's systems and that code based is used to serve your application off of their servers.
If you just start an application on your personal system, you will suffer a lack of network and resource availability as well as under use of computer time at non-peak hours as your system will need to remain attentive for additional users without having alternative tasks. Hosting providers provide resources as needed, and their diverse client base allows their systems to work around the clock on a global scale.
Standard procedure to start with
meteor app -p 3000&
This works, except when I close the ssh connection, application is no more running? I have no clue why this is happening.
Awakening Edit:
I use PM2 for traditional node apps, but if I have to setup process monitors, logs, database all by myself. I could just go back to reactjs and socketio and rock it with node.
Consider use mup package from Arunoda to easily deploy and run your app in production.
You could launch meteor with nohup (no hang-up) which serves this purpose.
nohup meteor --production &
But it's not a good idea to run a site in production with meteor anyway.
What should I do to run meteor forever ?
You can use forever, a Node.js tool designed to run node apps as services.
I also want to point that forever is getting old and I've heard of better recent alternatives but it seems to still be a pretty common tool. You could also use systemd which integrates better with the UNIX service ecosystem but that's anoter story.
But first, you'll have to "demeteorize" your meteor application like this :
cd my-project-meteor
meteor bundle --directory ../my-project-node
this is going to take some time
cd ../my-project-node/programs/server
npm install
this is going to take some time too
So now you have a plain node app, that you can run with node main.js
Let me mention that it might be a good idea to use the node version used by meteor which is 0.10.29 as of meteor 0.9.1 You can install it traditionally or you could use the node version that is shipped with the meteor tool.
sudo ln -s ~/.meteor/packages/meteor-tool/1.0.27/meteor-tool-os.linux.x86_64/dev_bundle/bin/node /usr/bin/node
sudo ln -s ~/.meteor/packages/meteor-tool/1.0.27/meteor-tool-os.linux.x86_64/dev_bundle/bin/npm /usr/bin/npm
Note that this way of "installing" node + npm on your system is problematic because :
it assumes you're doing only meteor related stuff.
it is dependant on the release process of the meteor tool (you'll need to rerun these commends if the meteor tool is updated).
You can install the forever tool using npm :
-g means globally : give access to forever to all users on the system
sudo npm install -g forever
To launch your node app as a service, you can use the following command, which sets correctly some environment variables and run the app using forever :
sudo export PORT=80 MONGO_URL=mongodb://localhost/my-project-mongodb ROOT_URL=http://localhost forever start my-project-node/main.js
You can monitor it using forever stop my-project-node/main.js
Also, what's the point of using 3rd party database service like https://mongolab.com/?
When using the meteor tool, it launches a mongod process automatically for you, and the underlying node process executed by meteor representing your app connects to this mongo instance.
When we want to launch our meteor app as a node app, we have to handle the mongo stuff ourself, which kinda answer the question : why not using another service to handle it for us, they know better, right ?
Doesn't it slow down the website, because now application has to connect to their database instead of local database ?
Of course, relying on a 3rd party database service has its inconvenients, and this is one of them. Network communications will always be slower than interprocess communications taking place on localhost (this is especially true on these SSD backed cheap VPS you can find nowadays).
And how exactly do I connect to mongolab for example ?
By setting an appropriate value to the environment variable MONGO_URL, the database service provider will give you an url that corresponds to your online mongodb, this is what you need to pass to the node process in command line if you want meteor to connect to your distant database and work as usual.
If you want to launch a dedicated local mongod instance to let your application connect to it, well this is another topic but you'll have to follow these steps :
first install mongodb correctly on your server, using the reference documentation for the OS version. By correctly I mean choose the same version as meteor is using currently (2.4.9) and let it run as a service so that it will actually restart when your server reboots.
test that mongod is running by launching a client with the mongo command.
pass the correct MONGO_URL when launching your app with forever (something like mongodb://localhost/my-project-mongodb)
Understand now why meteor deploy is amazing :D
Answer copy from here
At work we currently use the following deployment strategy:
Run a batch script to clear out all Temporary ASP.NET files
Run a batch script that compiles every ASPX file into its own DLL (ASP.NET Web Site, not Web Application)
Copy each individually changed file (ASPX and DLL) to the appropriate folder on the live server.
Open up the Deployment Scripts folder, run each SQL script (table modifications, stored procs, etc.) manually on the production database.
Say a prayer before going to sleep (joking on this one, maybe)
Test first thing the next morning and hope for the best - fix bugs as they come up.
We have been bitten a few times in the past because someone will forget to run a script, or think they ran something but didn't, or overwrote a sproc related to some module because there are two files (one in a Sprocs folder and one in a [ModuleName]Related folder) or copied the wrong DLL (since they can have the same names with like a random alphanumeric number generated by .NET).
This seems highly inefficient to me - a lot of manual things and very error prone. It can sometimes take 2-3 or more hours for a developer to perform a deployment (we do them late at night, like around midnight) due to all the manual steps and remembering what files need to be copied, where they need to be copied, what scripts need to be run, making sure scripts are run in the right order, etc.
There has got to be an easier way than taking two hours to copy and paste individual ASPX pages, DLLs, images, stylesheets and the like and run some 30+ SQL scripts manually. We use SVN as our source control system (mainly just for update/commit though, we don't do branching) but have no unit tests or testing strategy. Is there some kind of tool that I can look into to help us make our deployments smoother?
I did not go through all of it, but the You're deploying it wrong series from Troy Hunt might be a good place to look at.
Points discussed in the series :
Config transforms
Build Automation
Continuous Integration
We have four stages before it can be deployed.
Development
QA
UAT
Production
We have build scripts (inside bamboo build server) running against QA and against UAT. Our DBA is the only person who can run create scripts against QA, UAT, and PROD. Anything that goes from QA -> UAT is like a test run deployment. UAT gets reverted by copying the production systems down again.
When we release into Production we just create a whole new site and point it at the UAT database and test that environmentally it is working fine. Then when this is working good we flick the 'switch' and point the production IIS record at the next site, and change the DB connection to point at Prod DB.
Because we are using a completely diff folder structure all of our files get copied up so there is no chance of missing one. Because we have had test runs of deployment into UAT we know we haven't missed a DB script (DB Scripts are combined into one generally). Because we have tested a shadow copy of the IIS website we know that environmentally it should work. We can then do all this set up during the day - and then do the final switch flicking at midnight or whenever - reducing the impact on devs.
tl;dr; Automated build and deploy; UAT system for test running deployment; Deployment during work hours; Flick switch/run DB update at midnight.
I am a developer for BuildMaster, a tool which can very easily automate the steps you have outlined above, and we have a limited version free for a team of 5 developers.
Most of your pain points will disappear the moment you set up the deployment automation - mainly the batch script execution and the file-by-file copying. Once you're fully automated, you can even schedule the deployment for night time and only have to worry about it if there's an error in the process (you can set up a notifier for a failed build).
On the database side, you can integrate your database with BuildMaster as well and if you upload the scripts into the tool it will keep track of which ones were run against which database.
To see how to set up a simple web application deployment plan, you can run one of the example applications included. You can also check out: http://inedo.com/support/tutorials/lunchmaster/part-1 to see how to create one yourself - it's slightly outdated since we've made it even easier to get started out-of-the-box but the main concepts are the same.
Please see this blog post and associated talk by Scott Hanselman titled "Web Deployment Made Awesome"
Blog
Video
As for SQL Deployment, you might want to consider one of the following:
RikMigrations
Migrator.NET
FluentMigrator
Mantee Introduction & Source
Have a User Acceptance Testing (UAT) environment which is completely isolated from your development environment, and only accessible to the UAT manager.
Setup a UAT build which you can manually trigger upon each release, when triggered this should send all your deployment files as well as a deployment checklist to the UAT manager, who will redeploy all files to the UAT environment, and run any database upgrade scripts.
Once the applications users and testers have signed off the UAT release, the UAT manager can be authorised to deploy to the PRODUCTION environment using the exact same procedure and checklists as the UAT release. This will guarantee that you never miss any deployment steps, and test the deployment process prior to moving it into production.
Caveats- I'm in an environment where we can't use MSI, batch, etc for the final deployment
Things that helped:
A build server that does the full compilation on a build server and runs all unit tests and integration tests. Why find out you have something in an aspx page that doesn't compile on deployment night? (I admit your Q doesn't make it clear if compilation is happening on deployment night)
I have a page that administrators can reach that exercises environment and deployment failure points, e.g. connect to db, connect to reporting services, send an email, read and write to the temp folder.
Also, put all the things that the administrator needs to change into a file external from web.config. The connection string and app settings sections natively support a way to do this (i.e. don't reinvent the web.config system just to create a separate file)
Here is an article on how to do better integration tests: http://suburbandestiny.com/Tech/?p=601 There is a ton of good literature how to do unit tests, but often if you app already exists, you will have to refactor until unit testing becomes possible. If that isn't an option, then don't be a purist and put together some integration tests that are fast and repeatable as possible.
Keep your dependencies in bin instead of GAC, since it's easier to tell an administrator to copy files than it is to teach them to administer the GAC.
I am attempting to setup automated tests for our applications using a virtual machine environment.
What I would like to have is something like the following scenario:
Build server is automatically triggered to start an automated test for the application
A "build" script is then run which consist of:
Copy application files and a test script to a location accessible by the VM
Start the VM
In the VM, a special application looks in the shared folder and start the test script
The tests script do its job, results are output to shared folder
Test script ends
The special application then delete the test script
The special application somehow have the VM manager close the VM and revert to the previous snapshot
When the VM has exited, process the result and send to build server.
I am using TeamCity if that matters.
For virtual machines, we use VirtualBox but we are open to any other if needed.
Is there any applications/suite that would manage this scenario?
If there are none then I would then code it myself, should be easy but the only part I am not sure is the handling of the virtual machine.
What I need to be able to do is to have the VM close itself after the test and revert to a previous snapshot since I want it to be in a known state for the next test.
Any pointers?
I have a similar setup running and I chose to use Vagrant as its the same thing our developers where using for normalizing the development environment.
The initial state of the virtualmachine was scripted using puppet, but we didn't run the deployment scripts from scratch on each test, only once a day.
You could use puppet/chef for everything, but for all other operations on the VM, we would use Fabric scripts, as they were used for the real deployment too, and somehow fitted how we worked better. In sum the script would look something like the following:
vagrant up # fire up the vm, and run the puppet provisioning tool
fab vm run_test # run tests on vm
fab local process_result # process results on local shared folder
vagrant destroy # destroy the vm
The advantage is that your developers can also use vagrant to mimic your production environment without having to take care of that themselves (i.e. changes to your database settings get synced to all your developers vm's wherever they are) and the same scripts can be used in production too.
VirtualBox does have a COM API. I have no experience with it, but it may be possible to use that. One option would be to have TeamCity fire off a script to do this. I'd suggest starting with NAnt (supported natively by TeamCity) and possibly executing PowerShell if necessary.
Though I don't have any experience with either, I happen to have heard of a couple applications in this space recently:
http://www.infoq.com/news/2011/05/virtual_machine_test_harness
http://www.automatedqa.com/techpapers/testcomplete/automated-testing-in-virtual-labs/