I'm not a Git novice but also not a guru either and I have a question. We want to create remote repos that appear as folders within our network. The folders will actually contain a large legacy ASP app running in a production manner.
Then we want to be able to make local changes and be able to push commits to these networked repos and thus dynamically update the production application.
We already have the repo on Github and our developers fork that and work locally (we use SmartGit for most day to day stuff).
However (because the app is huge and legacy) we have (prior to using Git) always had a process for copying changed files to the target systems (production, QA etc).
But it dawned on me that we may be able to treat the operational system "as" a repo that's checked out to master. Then (when suitably tested) we want to simply use SmartGit to do a "push to" the operational system and have the changes delivered that way.
I'm at the edge of my Git knowledge though and unsure if this is easy to do or risky.
We don't want to install Git on the operational machine (its restricted to running Windows 2003 - yes I know...) so want to simply treat the remote system just like it was a local folder - with Git installed on our local machines.
Any tips or suggestions?
My tip: don't bother.
You can only push to bare repositories. These are such that they only contain the files normally residing in .git, with no working directory at all. So you cannot "run" those on the server. You would need to push to a bare repos on the server, and then clone/checkout that bare repos into a non-bare local repos on the server itself (which can be done in a post-receive hook inside git). But as you said, you cannot even install git on the server. So git push does nothing for you.
Second option would be to mount the servers filesystem on whatever staging/deployment machine you have, presumably one which you can install git on. Then you can git push into a bare repos on that deployment machine, run git hooks, and copy newly pushed stuff into your non-git server filesystem.
Third option would be to package everything up locally, make a tarball (or, I guess, zip-ball...) and just unpack that on the server.
So. Automated, continuous deployment => great idea. Using git => great idea. Directly using git push, not so much, mainly due to your constraints.
Related
Total newbie question but what is the best practice when it comes to using SSH with Git? I'm working on a WordPress project. In the root I have gulp and other dev files/folders like SASS and Scripts that I don't need on the server and in the same project I have my WordPress folder that contains a theme and a few custom plugins. As you can imagine when the theme or any of the plugins are ready to be deployed I don't want to pull everything in my repository on the server. So far as a newbie I've always just pull and pushed the entire repository and used FTP to upload what I need to the server, so how is this done with SSH and Git and is there a better way to have my setup?
EDIT: To make my question a little bit more clear let me give you an example of what I think my issue is. In my main project folder, I have a SASS folder next to my WordPress folder. All I really need to deploy to the server is the WordPress folder. My build process that happens on my dev machine combines all of the SASS files into a single CSS that is then placed into the WordPress folder. I need the SASS folder to be tracked by Git so that any other developer can pull them and continue developing so I can't have git ignore it. However none of those SASS files need to be on the server for WordPress to work either. I just simply need to deploy the WordPress folder and everything that's in it.
I understand the idea of creating a bare repository on the server and using post-receive hook to point the git folder sitting outside your web root to point to where the web root is. But that's basically how GIT and SSH work and that's not answering my concern.
Not with Git
Git is not designed to pull specific files or directories only. It's a directed acyclic graph with binary blobs as objects and sometimes multiple objects get compacted into a single larger object.
Due to Git design, your specific request is not possible.
Alternatives
post-receive hook
If your website only contains simple static files then it's okay to push to a git repository over SSH. In reality, it's unlikely your repository will be large as long as you don't have non-text files.
Take for example the following setup.
/var/lib/www - apache web dir which is the cloned copy of www.git
/var/lib/www.git - a bare git repository.
/var/lib/www.git/hooks/post-recieve - A server side git hook. It can be a shell script that pulls the www repository when this repository is updated.
Sample post-recieve hook script:
#!/bin/bash
cd /home/sam/sandbox/git-hooks/www
unset GIT_DIR
git fetch origin master
git reset --hard origin/master
Zip up build in a tar.gz
At the end of your build you can zip up your files in a tar.gz. This file should be hosted somewhere (perhaps GitHub releases if you're using GitHub). Some enterprises use on premise artifact hosting like Nexus or Artifactory.
The idea being: you have a tested artifact that has a specific sha256sum. The artifact you test is the exact same artifact which eventually goes to production.
Diving into more detail such as continuous integration, continuous delivery, and the software development life cycle might be out of scope for your question.
No best practice.
Git is for source control, not for deployment. There is no best practice for using git this way because git is not a deployment tool. You also don't need git history on your server. In fact, you don't need git at all unless you insist on using it for deployment. You are welcome to use it this way but it's not ideal because of exactly the kind of problem you're asking about.
What is the best practice?
There are a number of tools you could use to handle your deployments. Most of the tools generally let you set up a series of steps that let you deploy the code you want into the environment you want. You could go with simple tools such as Phing or Deployer in the PHP world, or something more sophisticated like Puppet or Chef if you have more complex needs. You could just write your own bash scripts if what you need is really very simple. I recommend Phing or Deployer given the info you've provided. https://deployer.org/ https://www.phing.info/
You'll just configure whichever tool you want to ssh into your target box and copy over only the files you want into the directory you want on the server, in whatever way you would like to do that. Usually, you have the script copy files into a temp dir, tarball them up, ssh them over and untar them. After that, you'll usually do some additional work on the server to move files around, change symlinks, whatever else you might need to do.
What about compiled SASS, ES6 js files, or modern static stuff?
All you need to do is add steps to the handle the static files and where you want them to go. Include the generated static files in your tarball when you push stuff up, and put them in the right directories in the server once you untar it.
When you configured your SASS compiler, and whatever other pre-compiled static code you may have - you configured it to create a destination file. That is, the file(s) of actual CSS and JS that they generate. That's all you need to bring along - and if you have the destination directory set to be inside your wordpress theme, you may not even have to pay all that much special attention to it's handling. You may need to move them somewhere else once they are on the server but that all depends on the specific setup in your server, which I think is outside the scope of this question.
Additional Notes:
You didn't ask about this but I thought it was worth mentioning, that you shouldn't be sending the entire wordpress repository every time you update. Just like you don't need the uncompiled SASS code, you also don't need to be repackaging core WordPress. You don't even need to be commiting core WordPress, its a dependency and you don't need to be changing it.
All that should be getting committed by you is your theme and plugin code, and the uncompiled static files. Compiled static files and external dependencies like the WordPress core don't belong in your git history. For deployment purposes, WordPress should already be installed. The stuff in your tarballs should just be plugins and themes, and additional static files if they aren't already in there for some reason.
TLDR;
Don't use git for this. Use a tool like Phing or Deployer. Build your static files into your theme, and create phing/deployer scripts that tarball up only the code you want, SSH's it over to your server, and untars it into the directories you want. If you have some special location on the server for your static files, just make sure to add steps in your script for that.
So, based on your question and comment, there are three computers involved. There is a web server (when you say "server", I take it as a Web server in this scenario, or the server computer that runs a Web server program). There is another server where your git repo is hosted. And, there is your dev workstation. Is this correct?
It seems like, you have a cloned git repo on your Web server. Your current practice/workflow appears to be (1) (based on your expression "SSH'ed into my server") you log into the web server via SSH (just like Telnet) from your workstation (SSH is just a protocol, which can be used for different purposes). (2) you pull from your repo on hosted service (e.g., github), and (3) deploy it to your "www" directory on the same server. Is this correct?
(I can think of an alternative scenario based on your use of the word "FTP", etc., but let's focus on the above scenario, for now.)
Now, your question is, whenever you "pull" (on your Web server), you feel like you are pulling everything from your repo on your hosted service. And, is there a better way? Am I understanding your question correctly?
If so, as another commenter suggested, git (and, any version control system, in general) is very good at fetching "deltas" only. If you are worried about "fetching everything" every time you pull (the step (2) above), then your worry is unfounded.
Now, the question is, why do you have a git repo on your Web server, if that is indeed the case? This is a pretty legit setup and I've done this before (e.g., on EC2). But, as a best practice, people generally don't do that on "production" servers. It's because you have to "build" your web app, and you really don't want to do that on production servers.
The next question is, what do you exactly do in Step (3)? The build process (whatever process you use) typically generates an "output" which can be directly deployed to the web server. (The convention is the output is generally a single folder, "public", "www", "dist", or whatever, or a single file (e.g., tar.gz, zip, jar, war), etc.) Regardless of whether you build the deployable output on your dev workstation (or, a build machine) or on your Web server, you don't generally do "deltas" in this context. Even if you've only changed a single file (say, a CSS file), you generally build the whole output again (instead of, say, just replacing the changed CSS file only). When you use FTP to upload files, etc., you can selectively upload certain files and/or directories, etc., but as a general practice, we don't do that. We always build the complete output from scratch and deploy it to the Web server. (This is mainly to reduce the potential deployment errors and increase the reliability.)
So, to answer your question, (A) If you are pulling git repo on your Web server, you should really change that practice, and move the build process to your dev computer or a dedicated build machine. (BTW, services like github, gitlab, TFS, ... provide the build service for you.) (B) If you are currently selectively FTP'ing your web app files to your Web server, then you should really consider adopting some kind of formal build, and deployment, process moving forward.
After your SASS build process is done use scp or rsync to move the files to the prod server:
scp -r /[local wordpress dir]/wp-content/themes/your-theme/ username#your.prod.server.com:/path/to/dir/wordpress/wp-content/themes/
scp -r /[local wordpress dir]/wp-content/plugins/* username#your.prod.server.com:/path/to/dir/wordpress/wp-content/plugins/
I am working in a project and using git ssh with bitbucket following is the process i am using it may work for you also if not please correct me :
Step 1 ->I have setup git and create repo in bit-bucket.
Step 2 ->And setup project with my local and linked with my repo.
Step 3 ->connect my server using ssh.
Step 4 ->Work in my local and commit and push all changes in my git repo.
Step 5 ->Run git pull on ssh so all changes deployed in my server.
I am using above process and i love this process.i have used .gitignore file that is not required for push on my repo.
Thanks
I want to create a master repository on our server, from which I can clone a local version onto my computer.
I am using R Studio v0.98.994.
So far, this is what I have tried doing:
Create a folder for the master repository to live in. I do this using 'new project' in R studio, and tell it to make a git repository.
I can then open up another new project, located on my C drive, and use R studio to clone, by telling it to open an existing project and setting the URL as the location of the master project.
However, then when I make changes and commit to my local repository (which works fine) I cannot push to the master repository, I get an error exactly as described in this question: git push fails: `refusing to update checked out branch: refs/heads/master`
So it appears that R Studio creates non-bare repositories?
Now I thought, well okay, I will use git bash to initialise the repository and then connect to that within R studio.
I do so, but cannot then find a way to use that repository in R Studio.
I am very new to Git, so it is entirely probable that this is one of those 'read the instructions' questions, in which case I am very sorry - and could someone possibly point me towards some guidance for this situation? I have spent the better half of a day googling around this error and haven't yet managed to pull together the pieces :( I also apologise; this doesn't feel like a very reproducible question.
It sounds like you are using Windows Git, with a setup on a local Windows machine (C: drive) and a server of some kind, mounted as the S: drive. There's a few things you should be aware of when doing this.
Shared Repositories
If you are intending for multiple people to share the same repository, you want to initiate a shared repository. See the --shared option in git-init for more details. Note that I'm not sure how having your repository on a Windows machine affects the sharing options. If you are just trying to keep your repository in two places, that makes things a lot easier.
Bare Repositories
Separate from the discussion of sharing is the discussion of bare repositories. If you don't intend to ever work with files in the server (i.e. it's just going to be a place to push changes so they are safely stored), you could initialize a bare repository. A bare repository contains the database structure of Git, but does not have the actual files in the directory.
A standard Git repository is a directory with a hidden folder in it named .git. This .git folder contains all the various data structures that Git uses to track changes. A bare repository is essentially a folder containing only the contents of .git.
The good thing about a bare repository is that no one can work in the repository itself (since there is no working directory, just the database). This means that no one could log into S: and edit the repository themselves. Instead, they would have to clone the repository, then push their changes back to the origin. The GitGuys have a good article about why this is ideal.
Note that shared repos and bare repos are not dependent or mutually exclusive. As a general practice, if you are having a "server repo" from which you pull and to which you push, you should have it be bare, regardless of whether the project is shared.
A Non-Shared Workflow
Since it's not clear if you are sharing or not sharing and you're on a Windows environment, which I don't know about from a sharing standpoint, I'm going to give you a simple example. Using git-bash, you should be able to change directories to wherever on S: you have your repositories. Then, use git init with the bare options as described by the link above to initialize a bare repository. Navigate to where you want your repository to live on C:, and then do git clone to get a working copy.
Add a README file or something else so you can do your initial commit, and then commit and do git push origin master to push your changes to the S: repository. Once all that is done, THEN initialize the RStudio Git project. RStudio should defer to your existing configuration, and things should hopefully work.
So I have a lot of websites, 150+. Starting with the bigger sites I am beginning to set up git repositories for tracking the changes to these sites. I can create a localserver version of a site and set up the repository and everything is running fine.
I have set up a .gitignore file to ignore all the core files and plugin folders etc. Again this is fine, the files are still on my local machine and have been deleted from my repository.
What I want to do is set up this repository on multiple computers (my colleagues who do less development work but will still need access to the repository). I imagine cloning won't work as all the core files are no longer in repository. How do I get around this?
Thanks all!
EDIT:
I should have mentioned we're using BitBucket to act as a central repository if that makes any difference.
There are few ways you can do that.
You can set local environment in one location, and keep git repository in other location.
After cloning or pulling the repository you can then run script which will copy the files from repository to the local environment.
You can add all files to the repository ignoring only var/, .htaccess, app/etc/local.xml and .gitignore. Bare in mind that you can break a website by changing files which should not have been changed. Debugging then becomes a nightmare. Having all in git, you know instantly what went wrong.
We've managed to set up great workflow using beanstalk.com. They've got option to share repositories (like github) and then deploying them on different server through SSH. Works like a charm - highly recommended.
I have Aptana Studio 3 set up using the Deployment Wizard to sync changes in both directions from local/remote for my projects. This works fine for syncing local changes to the remote site; my changes get FTP'd up automatically.
It doesn't seem to be working out when a file has changed remotely - there have been several situations where I've opened a file that has been modified by someone else since I last downloaded it, and the remote changes have not been downloaded to my local copy.
I've been manually synchronising directories before starting to work on them but this is a bit of an annoying workaround. Is it possible for Aptana to automatically check for remote changes, and sync my local copy?
I found a bug report which pretty much answers my own question: https://jira.appcelerator.org/browse/APSTUD-2811
Unfortunately it doesn't look like this is going to be fixed any time soon. It's a pretty essential feature IMO, especially for those working in teams.
You should consider using Git for moving files around (plus the whole version saving feature).
If you make a change on your local development system, simply
$ git add .
$ git commit -m "made a change"
$ git push origin
And then pull it down on your deployment server.
$ git pull
All done.
If you make a change on your deployment server, simply commit that change on the deployment server, and subsequently pull that change down to your development machine. There isn't really that much of a need for FTP anymore, although it does come in handy for moving untracked binary files.
http://try.github.com/levels/1/challenges/1
For quite a while now, I have been using Dropbox to sync a Git repository on several virtual machines (one Windows, one Mac, one Linux). I would commit my changes on one of the machines and Dropbox would take care of syncing the changes of the files and the repo onto the other machines.
This works very seamless. I code on OSX, test the code on Windows and Linux, maybe make some changes there, then commit from one of the three.
However, it has three major drawbacks:
It requires an internet connection. I frequently have to rely on my cellphone for internet connectivity, which is unreliable if I'm on a train and only good for a few hundred Mb per month.
Dropbox syncs EVERYTHING including object files, Visual Studio debug databases and just a whole lot of unnecessary stuff that does not need to be synced.
It always goes through Dropbox servers, which is fine for some minor project or some open source stuff, but I don't want to push my work files to an untrusted server.
So, how do you manage an environment like this?
Edit:
Actually, all the three virtual machines live on the very same laptop, so network connections between them are not a problem. Also, I frequently code on one OS and compile on another--and go back and forth until I have found all errors. I don't want to spam the company repo with hundreds of incremental commits.
Edit 2:
To give you an idea for what I am looking for, here is a partial solution I came up with: On each machine, I created a git repository of the files I want to work with. Typically, I will start working on a bug/feature one machine, then commit my work. On the next machine, I will call git reset origin to load the changes from the first machine, then continue working on the commit using git commit --amend. This will go back and forth a few times. Once I am done, I will finally commit the changes for real (no more amending) and start working on the next feature/bug.
However, this workflow feels cumbersome and inelegant. What I am looking for is something that results in the same output--one commit on the repo--but was created fluently between the three machines.
You could consider setting up your own software versioning server.
Most clients for these servers have implementations on varying OS's and platforms.
But if you want to communicate between machines that are not in a LAN, you're going to need an internet connection.
The versioning servers network communication can be exposed over NAT through a gateway to the internet. You could implement security by setting up a tunnel mechanism. Any client would then tunnel up to a gateway server and then communicate with the versioning server.
As for control over which files are actually versioned: I have some experience with SVN, with which you can select on file level which files to add to versioning. the SVN client will then simply ignore the rest of the files and directories.
Edit:
Reading the edit of the original author's question:
Maybe setup a 4th virutal machine, containing the Versioning server. SVN isn't (by any stretch of the imagination) hard to manage. (RTM). Have the three virtual machines connect to the server on the 4th. (This is ofcourse, if it's possible to run the machines in parallel on the same machine.)
If you can share a disk between the three, put the master repo on that. (Make sure you make backups! Especially for removable media.)
Edit: In more detail, you can have your master repo on a USB stick or a shared partition on your hard drive (as you indicate you are using multiple virtual machines on the same hardware).
To set up a private git repo, simply create an empty directory and run git init.
Assuming you are on your Ubuntu box and have an USB stick with a file system which you can read and write in all your operating systems, mounted in /media/usbgit, run this:
vnix$ mkdir /media/usbgit/mycode
vnix$ cd /media/usbgit/mycode
vnix$ git init
Initialized empty Git repository in /media/usbgit/mycode/.git/
(Given that you already have a git repo, you probably just want to clone it to the USB stick instead:
vnix$ cd /media/usbgit
vnix$ git clone /home/you/work/wherever/mycode
Initialized empty Git repository in /media/usbgit/mycode/.git/
This will now contain all the commits from the repo you pulled from.)
Now you have an empty repository which you can clone and pull from on all the boxes. Once you have the USB stick mounted, you can clone from it.
vnix$ cd
vnix$ mount | fgrep usbgit
/dev/whatever on /media/usbgit type NTFS (rw)
vnix$ git clone /media/usbgit/mycode
Initialized empty Git repository in /home/you/mycode/.git/
warning: You appear to have cloned an empty repository.
All of this is doable with SVN too (use svnadmin create to initialize a repository, and svn checkout file:///media/usbgit/mycode to check it out), but you will lose the benefits of a distributed VCS, which seem useful in your scenario.
In particular, with a distributed VCS, you can have multiple private repositories (each working directory is a repository on its own) and you can sync with and pull from your private master and a public repository e.g. on Github -- just make sure you know what you have where.