When uploading changed files to a live webserver, how do I enforce at any given moment, my entire file set is either in the old state, or new, but never something in between?
With 'in between' I mean some files old and some new, or some particular file being overwritten halfway, et cetera.
Suppose I have a directory called foo/ on my live webserver containing a framework or bunch of script files including eachother, and I need to replace a whole bunch of files at the same time. I'd call this an 'atomic update'.
The closest I can get is uploading my new foo/ directory (containing the new files) to a different name, e.g. bar/, and then on a shell on the webserver I do:
mv foo foo-old; mv bar foo;
But this way there is still a tiny fraction of a second where foo/ doesn't exist, when the old dir has just been renamed and the new dir is about to.
Is there a 100% correct way of doing this? I guess I need some sort of 'atomic swap', to rename or swap two directory names as a single, atomic action on file system level.
In case it's OS dependent: I'm using a webserver running CentOS and got SSH access.
Near-atomic
The simplest way to achieve a near-atomic change of directories is to use a symbolic link as your web root, which you can re-point to a different location when upgrading.
$ mkdir old
$ mkdir new
$ ln -s old live
$ ls -l
live -> old
new
old
...
$ ln -snf new live
live -> new
new
old
Changing the target of a symbolic link is actually a 2 step operation internally, unlink followed by symlink.
Atomic
An atomic change of directories can be achieved by creating a secondary symbolic link which points to a new directory, and then renaming the new symbolic link into the old symbolic link.
$ mkdir old new
$ ln -s old live
$ ln -s new live_new
$ mv -fT live_new live
mv command will use a single atomic operation (rename) to overwrite the old symbolic link with a new one.
Network layer atomic
Use 2 separate hosts (physical or virtual) and route every new user from a defined moment in time to a new host which contains the upgraded website.
Related
I have a directory that contains media that I am trying to setup a basic symbolic link to - the directory is a mounted storage on a digital ocean droplet in the following directory /mnt/storage/media/all
this contains directories as shown below:
0118
0119
0218 and so on.......
I am trying to make a symlink from my unix terminal as follows :
$ root#server1:/var/www/abcd/public ln -s /mnt/storage/media/all
So if I cd into the public directory above I would expect to see the directories 0118, 0119, 0218 and so on... however when I cd into this directory I see the directory all and within this directories are the 0118, 0119, 0218 subdirectories.
How do I change the symbolic link so I see the directories 0118, 0119, 0218 etc.. and not the all directory (which contains those same sub directories)
Try giving a second argument for the function. Let that argument be the desired destination for the link (but it shouldn't exist prior to this).
E.g. ln -s /mnt/storage/media/all /var/www/abcd/public
In case the folder public already exists, the symlink will be created inside it.
I am having trouble symlinking dotfiles. I have a folder in my home directory ~/dotfiles which I have synced to a github repo. I am trying to take my .vimrc file in ~/dotfiles/.vimrc and create a symbolic link to put it at ~/.vimrc. To do this I type in
ln -s ~/dotfiles/.vimrc ~/.vimrc
But when I run that it says
ln: /Users/me/.vimrc: File exists
What am I doing wrong?
That error message means that you already have a file at ~/.vimrc, which ln is refusing to overwrite. Either delete the ~/.vimrc and run ln again or let ln delete it for you by passing the -f option:
ln -s -f ~/dotfiles/.vimrc ~/.vimrc
There is a better solution for managing dotfiles without using symlinks or any other tool, just a git repo initialized with --bare.
A bare repository is special in a way that they omit working directory, so you can create your repo anywhere and set the --work-tree=$HOME then you don't need to do any work to maintain it.
Approach
first thing to do is, create a bare repo
git init --bare $HOME/.dotfiles
To use this bare repo, you need to specify --git-dir=$HOME/.dotfiles/ and --work-tree=$HOME, better is to create an alias
alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME
At this point, all your configuration files are being tracked, and you can easily use the newly registered dotfiles command to manage the repository, ex :-
# to check the status of the tracked and untracked files
dotfiles status
# to add a file
dotfiles commit .tmux.conf -m ".tmux.conf added"
# push new files or changes to the github
dotfiles push origin main
I also use this way to sync and store my dotfiles, see my dotfiles repository and can read at Storing dotfiles with Git where I wrote about managing for multiple devices.
How to symlink all dotfiles in a directory recursively
Have a dotfiles directory that is structured as to how they should be structured at $HOME
dotfiles_home=~/dotfiles/home # for example
cp -rsf "$dotfiles_home"/. ~
-r: Recursive, create the necessary directory for each file
-s: Create symlinks instead of copying
-f: Overwrite existing files (previously created symlinks, default .bashrc, etc)
/.: Make sure cp "copy" the contents of home instead of the home directory itself.
Tips
Just like ln, if you want no headache or drama, use an absolute path for the first argument like the example above.
Note
This only works with GNU cp (preinstalled in Ubuntu), not POSIX cp. Check your man cp, you can install GNU coreutils if needed.
Thanks
To this and this.
So this doesn't seem like a terribly complicated question I have, but it's one I can't find the answer to. I'm confused about what the -p option does in Unix. I used it for a lab assignment while creating a subdirectory and then another subdirectory within that one. It looked like this:
mkdir -p cmps012m/lab1
This is in a private directory with normal rights (rlidwka). Oh, and would someone mind giving a little explanation of what rlidwka means? I'm not a total noob to Unix, but I'm not really familiar with what this means. Hopefully that's not too vague of a question.
The man pages is the best source of information you can find... and is at your fingertips: man mkdir yields this about -p switch:
-p, --parents
no error if existing, make parent directories as needed
Use case example: Assume I want to create directories hello/goodbye but none exist:
$mkdir hello/goodbye
mkdir:cannot create directory 'hello/goodbye': No such file or directory
$mkdir -p hello/goodbye
$
-p created both, hello and goodbye
This means that the command will create all the directories necessaries to fulfill your request, not returning any error in case that directory exists.
About rlidwka, Google has a very good memory for acronyms :). My search returned this for example: http://www.cs.cmu.edu/~help/afs/afs_acls.html
Directory permissions
l (lookup)
Allows one to list the contents of a directory. It does not allow the reading of files.
i (insert)
Allows one to create new files in a directory or copy new files to a directory.
d (delete)
Allows one to remove files and sub-directories from a directory.
a (administer)
Allows one to change a directory's ACL. The owner of a directory can always change the ACL of a directory that s/he owns, along with the ACLs of any subdirectories in that directory.
File permissions
r (read)
Allows one to read the contents of file in the directory.
w (write)
Allows one to modify the contents of files in a directory and use chmod on them.
k (lock)
Allows programs to lock files in a directory.
Hence rlidwka means: All permissions on.
It's worth mentioning, as #KeithThompson pointed out in the comments, that not all Unix systems support ACL. So probably the rlidwka concept doesn't apply here.
-p|--parent will be used if you are trying to create a directory with top-down approach. That will create the parent directory then child and so on iff none exists.
-p, --parents
no error if existing, make parent directories as needed
About rlidwka it means giving full or administrative access. Found it here https://itservices.stanford.edu/service/afs/intro/permissions/unix.
mkdir [-switch] foldername
-p is a switch, which is optional. It will create a subfolder and a parent folder as well, even if parent folder doesn't exist.
From the man page:
-p, --parents no error if existing, make parent directories as needed
Example:
mkdir -p storage/framework/{sessions,views,cache}
This will create subfolder sessions,views,cache inside framework folder irrespective of whether 'framework' was available earlier or not.
PATH: Answered long ago, however, it maybe more helpful to think of -p as "Path" (easier to remember), as in this causes mkdir to create every part of the path that isn't already there.
mkdir -p /usr/bin/comm/diff/er/fence
if /usr/bin/comm already exists, it acts like:
mkdir /usr/bin/comm/diff
mkdir /usr/bin/comm/diff/er
mkdir /usr/bin/comm/diff/er/fence
As you can see, it saves you a bit of typing, and thinking, since you don't have to figure out what's already there and what isn't.
Note that -p is an argument to the mkdir command specifically, not the whole of Unix. Every command can have whatever arguments it needs.
In this case it means "parents", meaning mkdir will create a directory and any parents that don't already exist.
I tried to create a symbolic link to a non existing file
ln -s non_existing_file.txt $HOME/dir1/dir2/my_symbolic_link
Then I tried to write something in the non existing file using the symbolic link
vi $HOME/dir1/dir2/my_symbolic_link
Now after saving and exiting non_existing_file.txt is created under dir2.
Can someone explain why?
ln -s target linkpath
creates a symlink at linkpath which holds the name target. Operations on the symlink interpret the name target relative to the directory where the symlink resides, not the present working directory.
So, if you have a symlink holding, say, ../usr in /tmp/link-to-usr, then ls /tmp/link-to-usr will list the contents of /usr (which is /tmp/../usr) regardless of where the ls command is executed.
I would like to know the code implementation of built-in pwd and /bin/pwd especially when the directory path is a symbolic link.
Example:
hita#hita-laptop:/home$ ls -l
lrwxrwxrwx 1 root root 31 2010-06-13 15:35 my_shell -> /home/hita/shell_new/hita_shell
hita#hita-laptop:/home$ cd my_shell
hita#hita-laptop:/home/my_shell$ pwd <SHELL BUILT-IN PWD>
/home/my_shell
hita#hita-laptop:/home/my_shell$ /bin/pwd
/home/hita/shell_new/hita_shell
The output is different in both the cases. Any clue?
thanks
The kernel maintains a current directory (by inode) and when you need the current directory, it determines its name by walking up the directory tree (using ..) to find the names of all the path components. This is the 'real' or sometimes called 'physical' working directory. There is a library function getcwd(3) which does this for you; on more-recent Linux systems this is actually a system call, which helps with getting a consistent view should the parent directories be in the process of being renamed.
Some shells, notably bash, maintain a environment variable PWD to keep track of where you are, and if you changed directory through a symbolic link, this environment variable will show that. They call this the 'logical' path.
/bin/pwd shows the result of getcwd(3), ie the real path; if you give it -L it will tell you the value of PWD (unless it's rubbish, then you get the real path). (Gnu's version of /bin/pwd does more work than this to deal with complexities of parent directories without read permission and very long path names.)
Bash's built-in pwd shows you the 'logical' path with whatever symlinks you used to get there; even if it's now rubbish (ie deleted or renamed since you used it). The default of the built-in pwd can be changed with set -o physical (on) or set +o physical (off is plus!) The default prompt (containing the current directory) follows the option too.
# make a directory with a symlink alias
cd /tmp
mkdir real
ln -s real sym
cd sym
pwd # will say sym
pwd -L # will say sym
pwd -P # will say real
/bin/pwd # will say real
/bin/pwd -L # will say sym
/bin/pwd -P # will say real
rm /tmp/sym
pwd # says sym, though link no longer exists
/bin/pwd -L # will say real!
rmdir /tmp/real
pwd # says sym, though no directory exists
/bin/pwd # says error, as there isn't one
For what it's worth, my opinion is that all the 'logical' business is just adding to the confusion; the old way was the better way. It's true that symbolic links can be confusing, but this makes it more confusing, because any file operations which open .. don't do the same thing as any directory changes which use .. for example in this rather nasty example:
mkdir -p /tmp/dir/subdir
ln -s /tmp/dir/subdir /tmp/a
cd /tmp/a
ls .. # shows contents of /tmp/dir
(cd .. ; ls) # shows contents of /tmp
To avoid all this, you can put the following in your ~/.bashrc
set -o physical
Hope that helps!
Kind regards,
J.
PS The above is pretty specific to Linux and Gnu bash; other shells and systems are similar but different.
The shell's builtin pwd has the advantage of being able to remember how you accessed the symlinked directory, so it shows you that information. The standalone utility just knows what your actual working directory is, not how you changed to that directory, so it reports the real path.
Personally, I dislike shells that do what you're describing because it shows a reality different than that which standalone tools will see. For example, how a builtin tool parses a relative path will differ from how a standalone tool parses a relative path.
The shell keeps track in its own memory what your currenct directory is by concatenating it with whatever you cd to (and eliminating . and .. entries). It does this so that symbolic links don't mess up cd ... The /bin/pwd implementation walks the directory tree upwards trying to find inodes with the right names.
The built-in pwd shows symbolic links by default, but won't do if you give it the -P option.
In contrast, the pwd command doesn't show symbolic links by default, but will do if given the -L option.