In /etc/exports, I configured like that
/var/vols/tzhong *(rw,sync,anonuid=1999,anongid=1999,root_squash)
If I mount the nfs folder /var/vols/tzhong to my local folder /mnt/tzhong. I use root user to create a file under /mnt/tzhong, the file ownership will be 1999:1999. Is that possible to change the uid and gid when doing the mount command? Something like below.
mount -t nfs4 -o uid=2000 -o gid=2000 192.168.1.123:/var/vols/tzhong /mnt/tzhong
So the ownership of folder /mnt/tzhong will be 2000:2000. Also, all the files created under /mnt/tzhong will be 2000:2000.
But, actually, the command is failed with mount.nfs4: an incorrect mount option was specified.
Im using codeship for deployment and it provides a way to access the build machine with ssh:
ssh rof#1.2.3.4 -p 65503
This works fine and I get into the machine. Now I want to copy a file from the remote machine to my local machine. Im trying:
sudo scp -p 65503 -v -i ~/.ssh/id_rsa rof#1.2.3.4:~/home/rof/cache/app.js /
And I get a whole host of errors:
cp: 65503: No such file or directory
cp: -v: No such file or directory
cp: -i: No such file or directory
rof#23.20.112.101: Permission denied (publickey).
I dont know why it's saying No such file or directory for each argument.
id_rsa exists and is in ~/.ssh/ directory.
The Permission Denied error appears to be a separate issue.
Any ideas?
The first problem I see from looking at the documentation:
man scp:
-P port
Specifies the port to connect to on the remote host. Note that
this option is written with a capital ‘P’, because -p is
already reserved for preserving the times and modes of the
file.
-p Preserves modification times, access times, and modes from the
original file.
So scp -p is taken to mean "copy while preserving timestamps" and 65503 is the name of (one of the) source file(s).
Try scp -P 65503 instead.
I have multiple AWS EC2 instances which are updated from a Git repository via CodeDeploy. However, since keeping the wp-content/uploads folder in Git is messy and hard to maintain, I'm instead trying to move all uploads to a directory which I have mounted as an EFS filesystem. That way I should be able to share the uploads between multiple EC2 instances.
However, now I'm running into a new problem; there's no way for me to set the WP uploads folder to be outside of the WP root.
WordPress is located at /opt/bitnami/apps/wordpress/htdocs, which is also where our domain points. The EFS system is mounted to /home/bitnami/efs. Since the EFS directory is located outside of the WP root there's no way for me to link to it.
I have gotten this to work using a symlink directing the default wp-content/uploads folder to my desired path; however, this doesn't really solve my issue since I can't rely on the symlink to not be overwritten during CodeDeploy deployments.
So my questions are as follows:
Is it possible to have this setup work without using a symlink, so I can make deployments without worrying about by uploads being affected?
If a symlink is the only/best way to make this work, is there any way to add the symlink to the git repository, or any other way to make absolutely sure it persists during CodeDeploy deployments?
You can directly mount your EFS to /opt/bitnami/apps/wordpress/htdocs/wp-content/uploads to avoid symlinks.
In your appspec.yml add two hooks:
hooks:
BeforeInstall:
- location: /deploy/BeforeInstall.sh
timeout: 3000
runas: root
AfterInstall:
- location: /deploy/AfterInstall.sh
timeout: 3000
runas: root
Then create directory deploy and two files in it BeforeInstall.sh and AfterInstall.sh
BeforeInstall.sh unmount the EFS if its mounted
#!/bin/bash
if mount | grep /opt/bitnami/apps/wordpress/htdocs/wp-content/uploads > /dev/null; then
sudo umount /opt/bitnami/apps/wordpress/htdocs/wp-content/uploads
fi
Then mount EFS again via AfterInstall.sh after deployment
#!/bin/bash
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-fxxxxxx.efs.us-west-2.amazonaws.com:/ /opt/bitnami/apps/wordpress/htdocs/wp-content/uploads
Note: You can also mount entire wp-contents directory, not just uploads folder, so that update in themes and plugins also get reflected in other EC2s automatically.
Also, symlink is a better solution, so you don't have to ever worry about accidentally removing the EFS mount during deployment if umount ever fails. You can just recreate symlink again after deployment using AfterInstall hook and add code in file AfterInstall.sh
#!/bin/bash
ln -s /home/bitnami/efs /opt/bitnami/apps/wordpress/htdocs/wp-content/uploads
I tried to create a directory /data as a mount point but that failed for other reasons, now I've got a weird file/data thing in my root directory that just won't die:
user#PC:~$ sudo ls /data
ls: cannot access /data: Transport endpoint is not connected
user#PC:~$ sudo rm -rf /data
rm: cannot remove ‘/data’: Is a directory
I initially created it using mkdir but if I browse to the file in ubuntu it looks like a file in the file browser. How do I get rid of this thing?
First try unmounting then you should be able to delete it.
Then if that fails, and this is going to sound like a Windows solution but it might work, try rebooting and doing unmount/remount delete again.
I would like to rsync from local computer to server. On a directory that does not exist, and I want rsync to create that directory on the server first.
How can I do that?
If you have more than the last leaf directory to be created, you can either run a separate ssh ... mkdir -p first, or use the --rsync-path trick as explained here :
rsync -a --rsync-path="mkdir -p /tmp/x/y/z/ && rsync" $source user#remote:/tmp/x/y/z/
Or use the --relative option as suggested by Tony. In that case, you only specify the root of the destination, which must exist, and not the directory structure of the source, which will be created:
rsync -a --relative /new/x/y/z/ user#remote:/pre_existing/dir/
This way, you will end up with /pre_existing/dir/new/x/y/z/
And if you want to have "y/z/" created, but not inside "new/x/", you can add ./ where you want --relativeto begin:
rsync -a --relative /new/x/./y/z/ user#remote:/pre_existing/dir/
would create /pre_existing/dir/y/z/.
From the rsync manual page (man rsync):
--mkpath create the destination's path component
--mkpath was added in rsync 3.2.3 (6 Aug 2020).
Assuming you are using ssh to connect rsync, what about to send a ssh command before:
ssh user#server mkdir -p existingdir/newdir
if it already exists, nothing happens
The -R, --relative option will do this.
For example: if you want to backup /var/named/chroot and create the same directory structure on the remote server then -R will do just that.
this worked for me:
rsync /dev/null node:existing-dir/new-dir/
I do get this message :
skipping non-regular file "null"
but I don't have to worry about having an empty directory hanging around.
I don't think you can do it with one rsync command, but you can 'pre-create' the extra directory first like this:
rsync --recursive emptydir/ destination/newdir
where 'emptydir' is a local empty directory (which you might have to create as a temporary directory first).
It's a bit of a hack, but it works for me.
cheers
Chris
This answer uses bits of other answers, but hopefully it'll be a bit clearer as to the circumstances. You never specified what you were rsyncing - a single directory entry or multiple files.
So let's assume you are moving a source directory entry across, and not just moving the files contained in it.
Let's say you have a directory locally called data/myappdata/ and you have a load of subdirectories underneath this.
You have data/ on your target machine but no data/myappdata/ - this is easy enough:
rsync -rvv /path/to/data/myappdata/ user#host:/remote/path/to/data/myappdata
You can even use a different name for the remote directory:
rsync -rvv --recursive /path/to/data/myappdata user#host:/remote/path/to/data/newdirname
If you're just moving some files and not moving the directory entry that contains them then you would do:
rsync -rvv /path/to/data/myappdata/*.txt user#host:/remote/path/to/data/myappdata/
and it will create the myappdata directory for you on the remote machine to place your files in. Again, the data/ directory must exist on the remote machine.
Incidentally, my use of -rvv flag is to get doubly verbose output so it is clear about what it does, as well as the necessary recursive behaviour.
Just to show you what I get when using rsync (3.0.9 on Ubuntu 12.04)
$ rsync -rvv *.txt user#remote.machine:/tmp/newdir/
opening connection using: ssh -l user remote.machine rsync --server -vvre.iLsf . /tmp/newdir/
user#remote.machine's password:
sending incremental file list
created directory /tmp/newdir
delta-transmission enabled
bar.txt
foo.txt
total: matches=0 hash_hits=0 false_alarms=0 data=0
Hope this clears this up a little bit.
eg:
from: /xxx/a/b/c/d/e/1.html
to: user#remote:/pre_existing/dir/b/c/d/e/1.html
rsync:
cd /xxx/a/ && rsync -auvR b/c/d/e/ user#remote:/pre_existing/dir/
rsync source.pdf user1#192.168.56.100:~/not-created/target.pdf
If the target file is fully specified, the directory ~/not-created is not created.
rsync source.pdf user1#192.168.56.100:~/will-be-created/
But the target is specified with only a directory, the directory ~/will-be-created is created. / must be followed to let rsync know will-be-created is a directory.
use rsync twice~
1: tranfer a temp file, make sure remote relative directories has been created.
tempfile=/Users/temp/Dir0/Dir1/Dir2/temp.txt
# Dir0/Dir1/Dir2/ is directory that wanted.
rsync -aq /Users/temp/ rsync://remote
2: then you can specify the remote directory for transfer files/directory
tempfile|dir=/Users/XX/data|/Users/XX/data/
rsync -avc /Users/XX/data rsync://remote/Dir0/Dir1/Dir2
# Tips: [SRC] with/without '/' is different
This creates the dir tree /usr/local/bin in the destination and then syncs all containing files and folders recursively:
rsync --archive --include="/usr" --include="/usr/local" --include="/usr/local/bin" --include="/usr/local/bin/**" --exclude="*" user#remote:/ /home/user
Compared to mkdir -p, the dir tree even has the same perms as the source.
If you are using a version or rsync that doesn't have 'mkpath', then --files-from can help. Suppose you need to create 'mysubdir' in the target directory
Create 'filelist.txt' to contain
mysubdir/dummy
mkdir -p source_dir/mysubdir/
touch source_dir/mysubdir/dummy
rsync --files-from='filelist.txt' source_dir target_dir
rsync will copy mysubdir/dummy to target_dir, creating mysubdir in the process. Tested with rsync 3.1.3 on Raspberry Pi OS (debian).