Transferring a new folder to remote using rsync? - rsync

If I create a completely new folder locally, I want to be able to rsync it remotely to an SFTP server, how can I achieve this?
I have tried:
rsync Documents/SomeFolder username#host:/home/Documents/RemoteFolder
Meaning SomeFolder must go into RemoteFolder, but this doesn't work, instead it creates a file called SomeFolder
Would appreciate some help on this

If you use the -r (recurse into directories) option that should make it work. Also -d (transfer directories without recursing) will work. You should use -r if sometimes the folder will not be empty and you want to copy its contents. Use either as shown here:
rsync -r Documents/SomeFolder username#host:/home/Documents/RemoteFolder

Related

rsync not copying my file from remote server

I have two VM's : dev and prod.
I want to use rsync to copy dump file from prod and then restore it on dev. I'm using this command to copy:
rsync -rave user#ip:/home/user/dumps /home/anotheruser/workspace/someapp/dumps
The same thing successfully copies static files (.html, .css) from another directory, but in this case only the folder itself is created but without the file:
/home/anotheruser/workspace/someapp/dumps
but I'm expecting:
/home/anotheruser/workspace/someapp/dumps/dumpfile
What is going wrong? dumpfile exists there user#ip:/home/user/dumps/dumpfile.
The command you want is probably this:
rsync -av user#ip:/home/user/dumps/ /home/anotheruser/workspace/someapp/dumps/
I've
removed the r because it's implied by the a anyway.
removed the e because that was probably your problem; it requires a parameter that you haven't given.
added the / at the end of the pathnames to make sure they're treated as directories.

rsync skip subdirectory path but not contents

I want to use rsync to copy some files from a folder structure and in the new location have the structure modified slightly. Below is what I currently have and what I'm trying to achive
Folders:
Parent/A/1/a,b,c
Parent/A/2/j,k,l
Parent/A/3/x,y,z
Parent/B/1/a1,b1,c1
Parent/B/2/j1,k1,l1
Parent/B/3/x1,y1,z1
In the new location what I want is
Parent/A/x,y,z
Parent/B/x1,y1,z1
what I have is
PathToParent/A/3/x,y,z
PathToParent/B/3/x1,y1,z1
after using the following command sequence
rsync -avzP --exclude=*/1 --exclude=*/2 ../Parent/ remote:../ParentPath/
I can easily work around this issue but I was hoping that rsync had an option to allow me to run this is as a single command.
Thanks in advance!
No, it can't do that transformation.
You can put multiple rsync invocations in a script, however ...
rsync -a Parent/A/3/ remote:../ParentPath/A/
rsync -a Parent/B/3/ remote:../ParentPath/B/

Backup using rsync apparently does not remove files and folders that does not exist anymore on source

I'm using very handy rsync command which allows me to have a backup of particular folders on a specific volumes.
I call rsync with following parameters:
rsync -avzP
To be explicit, when I want to do a backup of all pictures and Lightroom catalogs I call:
rsync -avzP /Volumes/SLICK-2TB/Pictures /Volumes/SLICK-PICTURES-BACKUP
So SLICK-2TB is my source drive and SLICK-PICTURES-BACKUP is my destination drive.
My problem is, whenever I delete / remove a file on source, the change is not reflected on destination. In other words, all new stuff will be always archived on backup volume, but things that don't exist on the source, will be left intact on destination.
Is there a particular attribute that I could add to -avzP that will help me achieve / solve the problem?
Thanks.
You need to use the --delete argument to delete the files on the destination.
Also consider using --dry-run first to make sure you aren't deleting the wrong files. I guess the reason --delete is not a default argument (or part of -a) is that it can wipe out the destination if it's not set correctly.

mkdir's "-p" option

So this doesn't seem like a terribly complicated question I have, but it's one I can't find the answer to. I'm confused about what the -p option does in Unix. I used it for a lab assignment while creating a subdirectory and then another subdirectory within that one. It looked like this:
mkdir -p cmps012m/lab1
This is in a private directory with normal rights (rlidwka). Oh, and would someone mind giving a little explanation of what rlidwka means? I'm not a total noob to Unix, but I'm not really familiar with what this means. Hopefully that's not too vague of a question.
The man pages is the best source of information you can find... and is at your fingertips: man mkdir yields this about -p switch:
-p, --parents
no error if existing, make parent directories as needed
Use case example: Assume I want to create directories hello/goodbye but none exist:
$mkdir hello/goodbye
mkdir:cannot create directory 'hello/goodbye': No such file or directory
$mkdir -p hello/goodbye
$
-p created both, hello and goodbye
This means that the command will create all the directories necessaries to fulfill your request, not returning any error in case that directory exists.
About rlidwka, Google has a very good memory for acronyms :). My search returned this for example: http://www.cs.cmu.edu/~help/afs/afs_acls.html
Directory permissions
l (lookup)
Allows one to list the contents of a directory. It does not allow the reading of files.
i (insert)
Allows one to create new files in a directory or copy new files to a directory.
d (delete)
Allows one to remove files and sub-directories from a directory.
a (administer)
Allows one to change a directory's ACL. The owner of a directory can always change the ACL of a directory that s/he owns, along with the ACLs of any subdirectories in that directory.
File permissions
r (read)
Allows one to read the contents of file in the directory.
w (write)
Allows one to modify the contents of files in a directory and use chmod on them.
k (lock)
Allows programs to lock files in a directory.
Hence rlidwka means: All permissions on.
It's worth mentioning, as #KeithThompson pointed out in the comments, that not all Unix systems support ACL. So probably the rlidwka concept doesn't apply here.
-p|--parent will be used if you are trying to create a directory with top-down approach. That will create the parent directory then child and so on iff none exists.
-p, --parents
no error if existing, make parent directories as needed
About rlidwka it means giving full or administrative access. Found it here https://itservices.stanford.edu/service/afs/intro/permissions/unix.
mkdir [-switch] foldername
-p is a switch, which is optional. It will create a subfolder and a parent folder as well, even if parent folder doesn't exist.
From the man page:
-p, --parents no error if existing, make parent directories as needed
Example:
mkdir -p storage/framework/{sessions,views,cache}
This will create subfolder sessions,views,cache inside framework folder irrespective of whether 'framework' was available earlier or not.
PATH: Answered long ago, however, it maybe more helpful to think of -p as "Path" (easier to remember), as in this causes mkdir to create every part of the path that isn't already there.
mkdir -p /usr/bin/comm/diff/er/fence
if /usr/bin/comm already exists, it acts like:
mkdir /usr/bin/comm/diff
mkdir /usr/bin/comm/diff/er
mkdir /usr/bin/comm/diff/er/fence
As you can see, it saves you a bit of typing, and thinking, since you don't have to figure out what's already there and what isn't.
Note that -p is an argument to the mkdir command specifically, not the whole of Unix. Every command can have whatever arguments it needs.
In this case it means "parents", meaning mkdir will create a directory and any parents that don't already exist.

synchronise local directories over ssh

The following command works great for me for a single file:
scp your_username#remotehost.edu:foobar.txt /some/local/directory
What I want to do is do it recursive (i.e. for all subdirectories / subfiles of a given path on server), merge folders and overwrite files that already exist locally, and finally downland only those files on server that are smaller than a certain value (e.g. 10 mb).
How could I do that?
Use rsync.
Your command is likely to look like this:
rsync -az --max-size=10m your_username#remotehost.edu:foobar.txt /some/local/directory
-a (archive mode - the sync is recursive, transfers ownership, attributes, symlinks among other things)
-z (compresses transfer)
--max-size (only copies files up to a certain size)
There are many more flags which may be suitable. Checkout the docs for more details - http://linux.die.net/man/1/rsync
First option: use rsync.
Second option, and it's not going to be a one liner, but can be done in three or four lines:
Create a tar archive on the remote system using ssh.
Copy the tar from remote system with scp.
Untar the archive locally.
If the creation of the archive gets a bit complicated and involves using find and/or tar with several options it is quite practical to create a script which would do that locally, upload it on the server with scp, and only then execute remotely with ssh.

Resources