Is it possible to use gsutil's rsync command to target a remote server?
I want to do:
gsutil -m rsync -d -r gs://some-bucket username#some.remote.server.net:destination
If not, what would you recommend?
Related
I can scp files to a remote host, but if I try to scp -r a directory it does not work
scp -r mydir me#remote.host:/where/the/dir/has/to/go/
scp: realpath /whatever/mydir: No such file
scp: upload "/whatever/mydir": path canonicalization failed
scp: failed to upload directory cf2t to /where/the/dir/has/to/go/
I thought it was some issue with globbing, so I quoted different parts of the damn command to no avail.
on remote.host: mkdir -p /where/the/dir/has/to/go/
on local host: scp -r mydir me#remote.host:/where/the/dir/has/to/go/
I am trying to leverage the power of elastic beanstalk with a fresh wordpress install. To keep it stateless I am trying to use EFS to persist wp-content files between ec2 instances, but for some reason I can't get my EFS setup to persist my wp-content folder.
The following is my efs.config file.
packages:
yum:
nfs-utils: []
jq: []
files:
"/tmp/mount-efs.sh" :
mode: "000755"
content: |
#!/usr/bin/env bash
mkdir -p /mnt/efs
EFS_NAME=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.EFS_NAME')
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $EFS_NAME:/ /mnt/efs || true
mkdir -p /mnt/efs/wp-content
chown webapp:webapp /mnt/efs/wp-content
mkdir -p /mnt/efs/wp-content/themes
chown webapp:webapp /mnt/efs/wp-content/themes
mkdir -p /mnt/efs/wp-content/plugins
chown webapp:webapp /mnt/efs/wp-content/plugins
mkdir -p /mnt/efs/wp-content/uploads
chown webapp:webapp /mnt/efs/wp-content/uploads
commands:
01_mount:
command: "/tmp/mount-efs.sh"
container_commands:
01-rm-wp-content-uploads:
command: rm -rf /var/app/ondeck/wp-content
02-symlink-uploads:
command: ln -snf /mnt/efs/wp-content /var/app/ondeck/wp-content
It seems like it's mounting and deleting the files? Each time when a new instance is auto created I was able to ssh into the instance and see the new instance mount a wp-content folder but without my existing files?
Thanks in Advance!
Also, would I be able to see what files are in EFS directly in the AWS console?
Thanks.
You need setup to mount your efs folder automatically when your instance reboots.
Follow the steps which mention in this article.
http://docs.aws.amazon.com/efs/latest/ug/mount-fs-auto-mount-onreboot.html
It's unclear to me how to get my build files from the Gitlab CI (hosted on https://ci.gitlab.com) over to my personal server using rsync.
I have setup 1 test and 1 deploy job.
Under the deploy tab I have inputed the bash commands to:
Install rsync
Update packages
Finally, the rsync command to
transfer files over SSH to my personal server.
When I enter the SSH credentials (with verbose flag on) for my private personal server, it would appear that the SSH key is the issue. In Gitlab, I have already established the deploy key (for hooks - tested this and it works).
Where do I locate the public SSH key for the Gitlab deploy instance so that I can install that key on my server?
Below is the exact script entered in Gitlab CI deploy job script pane:
# Run as root
(
set -e
set -u
set -x
apt-get update -y
apt-get -y install rsync
)
git clone https://github.com/bla/deployments.git $HOME/deploy/deployments
SVR_WEB1_WEBSERVER="000.11.22.333"
USER1="franklin"
GROUP1="team1"
FROM_DIR="/gitlab-ci-runner/tmp/builds/myrepo-1/"
DEST1="subdomains/gitlab/myrepo"
EXCLUSIONS_LIST="${HOME}/deploy/deployments/exclusions/exclusions.txt"
ssh -v "$USER1#$SVR_WEB1_WEBSERVER"
/usr/bin/rsync -avzh --progress --delete -e ssh --group=$GROUP1 -p --exclude-from "$EXCLUSIONS_LIST" "$FROM_DIR" "$USER1#$SVR_WEB1_WEBSERVER:$DEST1"
Providing your private ssh key is dangerous unless you use your own gitlab-ci runners for deployment. That's why it is better to use rsync modules.
I'm looking for some assistance please to create a proper command-line for syncing from a local machine to a remote server over ssh.
Here is a draft that is not working.
/usr/bin/rsync --dry-run --delete -arzh /Users/HOME/.0.data/ "--rsh=/Users/HOME/.0.data/.0.emacs/elpa/bin/sshpass -p 'alpine' ssh -p '2222' -l root localhost" -t "cd /var/mobile/Applications/F30B1574-5979-4764-8742-7F9DB2863094/Documents/.0.data && bash --login"
The following command-line successfully logs in to my iphone over ssh via usb. Id like to incorporate that working command-line into something that can be used with rsync, but I need some assistance in that regard.
/Users/HOME/.0.data/.0.emacs/elpa/bin/sshpass -p 'alpine' ssh -p '2222' -l root localhost -t "cd /var/mobile/Applications/F30B1574-5979-4764-8742-7F9DB2863094/Documents/.0.data && bash --login"
For anyone who is interested in leaning how to ssh into an iphone over usb, here is a link that discusses the method: http://iphonedevwiki.net/index.php/SSH_Over_USB
rsync must be installed on both locations. Cydia has an rsync binary that installs on the iPhone. The method of connection with rsync is the same as any regular ssh sever.
Here is a bash script solution (includes --dry-run):
#!/bin/bash
HOST="localhost";
PORT="2222";
USER="root";
PWD="alpine";
SOURCE="/Users/HOME/Desktop/test/";
TARGET="/private/var/mobile/Applications/F30B1574-5979-4764-8742-7F9DB2863094/Documents/test";
SSHPASS="/Users/HOME/.0.data/.0.emacs/elpa/bin/sshpass";
RSYNC="/Users/HOME/.0.data/.0.emacs/elpa/bin/rsync";
$RSYNC --dry-run --progress --delete -arvzh --rsh="$SSHPASS -p $PWD ssh -p $PORT -l $USER" $SOURCE $HOST:$TARGET
For an example of how to use rsync in conjunction with Emacs, see the following thread: https://emacs.stackexchange.com/a/5844/2287
How do you use an identity file with rsync?
This is the syntax I think I should be using with rsync to use an identity file to connect:
rsync -avz -e 'ssh -p1234 -i ~/.ssh/1234-identity' \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
But it's giving me an error:
Warning: Identity file ~/.ssh/1234-identity not accessible: No such file or directory.
The file is fine, permissions are set correctly, it works when doing ssh - just not with rsync - at least in my syntax. What am I doing wrong? Is it trying to look for the identity file on the remote machine? If so, how do I specify that I want to use an identity file on my local machine?
Use either $HOME
rsync -avz -e "ssh -p1234 -i \"$HOME/.ssh/1234-identity\"" dir remoteUser#server:
or full path to the key:
rsync -avz -e "ssh -p1234 -i /home/username/.ssh/1234-identity" dir user#server:
Tested with rsync 3.0.9 on Ubuntu
You may want to use ssh-agent and ssh-add to load the key into memory. ssh will try identities from ssh-agent automatically if it can find them. Commands would be
eval $(ssh-agent) # Create agent and environment variables
ssh-add ~/.ssh/1234-identity
ssh-agent is a user daemon which holds unencrypted ssh keys in memory. ssh finds it based on environment variables which ssh-agent outputs when run. Using eval to evaluate this output creates the environment variables. ssh-add is the command which manages the keys memory. The agent can be locked using ssh-add. A default lifetime for a key can be specified when ssh-agent is started, and or specified for a key when it is added.
You might also want to setup a ~/.ssh/config file to supply the port and key definition. (See `man ssh_config for more options.)
host 22.33.44.55
IdentityFile ~/.ssh/1234-identity
Port 1234
Single quoting the ssh command will prevent shell expansion which is needed for ~ or $HOME. You could use the full or relative path to the key in single quotes.
You have to specify the absolute path to your identity key file. This probably some sort of quirck in rsync. (it can't be perfect after all)
I ran into this issue just a few days ago :-)
This works for me
rsync -avz --rsh="ssh -p1234 -i ~/.ssh/1234-identity" \
"/local/dir/" remoteUser#22.33.44.55:"/remote/dir/"
use key file with rsync:
rsync -rave "ssh -i /home/test/pkey_new.pem" /var/www/test/ ubuntu#231.210.24.48:/var/www/test
Are you executing the command in bash or sh? This might make a difference. Try replacing ~ with $HOME. Try double-quoting the string for the -e option.