Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I have rsync executed continuously between 2 systems which has the tcp connection getting interrupted due to known reasons.
In a rare case, it so happens that the entire rsync destination directory is deleted and the data gets synced to alternative location.
The rsync option used is "-rpt -iP --stats --inplace" I read about --inplace being inconsistent with interrupted connection.
-rpt -iP --stats --inplace >> FAILS
Need help to come up with safest approach to avoid inconsistencies in rsync in an environment with frequent connection disruption
If you need a consistent way to syncing whole directory use:
rsync -avz \
--partial \
--partial-dir=.rsync-partial/ \
--delay-updates \
--delete \
--stats \
...
Linux man says about --inplace:
This has several effects: (1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash), (2) the file's data will be in an inconsistent state during the transfer, (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails
So, --inplace can not be used for consistent syncing. Instead use delay-updates algorithm that guarantees that destination-dir changes will be implied only after successfully completed transfer.
Also you may decide to use -a instead of -rpt. -a param is equivalent to -rlptgoD and this is complete params set for consistent syncing. -vz params useful for verbose output and compression while transfer (reducing traffic).
I have a 200GB encrypted file as aes256.
I believe the password is one of 10 however it takes 5+ hours to decrypt currently before it gives an archive failed message in terminal.
my command is:
openssl enc -in ~/path/file.tgz.aes256 -aes-256-cbc -d -k password | tar -zxv -C ~/Desktop/location/
Is there a command I can run to quickly test archive with different passwords to ascertain the right password to use (brute force, essentially, but with a library of 10 passwords).
Decrypt the first part of the file data specifying no padding, it will need to be a multiple of the block size (16-bytes for AES). Then check if the decryption succeeded by looking at the decrypted data. It will either be what looks like random bytes or the correct data.
But your decryption is very slow, I can decrypt 200MB/s on an iPhone so 200GB would only take about 15 minutes plus the time to read the 200GB from disk. You may need to find another implementation that takes advantage of the Intel AES instructions (AES-NI). Software only decryption can be up to 1000 times slower.
On a typical Unix system, if I try passing a raw password into SSH through a pipe I will get an error like
$ echo password | ssh user#host
Pseudo-terminal will not be allocated because stdin is not a terminal.
The exact same thing will work with keyboard input which as I understand is provided over stdin in exactly the same manner.
I'm not interested in passing a raw password to SSH (this would be terrible for too many reasons to list)
I want to understand what's different in this case between the stdin from a keyboard and the stdin from a Unix pipe.
EDIT: I am aware of the existence of SSH keys and how to use them. I am also aware that passing a plaintext password to ssh is a bad idea. This question is only about understanding what is happening differently between stdin from a pipe and the stdin from a keyboard.
As mentioned in the other answer, security is the reason.
From technical point of view, by doing echo password | ssh user#host, you push the string to pipe and the characters are waiting on the ssh side to be read. But most of password prompts truncate this input before showing prompt (and modifying it to not-show the characters -- described later).
To answer the final question about difference, both are some kind of pipes, but when you have terminal on the other side, you can communicate with this side using control characters and you can reading/setting specific properties of the terminal, which both obviously fails on normal pipe from echo.
The error message is caused by the fact, that when you read the password from terminal, you modify terminal properties, so it will not show characters written (so called raw mode). And if the ssh program can't communicate with the terminal it fails like this.
I have a 3 Disk RAIDz1 Pool, encrypted with AES128 in GEOM_ELI, that I have been using in FreeNAS since version 8.
There have been many zpool upgrades, and over all I am very happy with ZFS.
Lately however I have been growing frustrated with FreeNAS. Largely many bugs that haven't been fixed over the years. But overall its the INSISTING on me using a Flash drive for their os, even though most of it is read only.
It's still a Single point of failure and has always extended boot times by several minutes. Bottom line, I just want to use Vanilla FreeBSD with this pool.
I am looking for more flexibility and a I wish to educate myself with this awesome Operating System.
Doing some more extended research I have found many tutorials on installing FreeBSD naively to a ZFS volume and mounting it as /
It wasn't till I did more research and found an article on mounting a zfs encrypted volume as root. Later I found that FreeBSD 10 does this during installation, which is awesome to say the least.
Tutorial I used
I made a VM With VMWare workstation, with three 2TB Drives, passed through as Physical Disks, and followed every step to a T and everything worked out very well. Now that I had a better grasp on the commands I was doing and why I was doing them, I wanted to do this to an already existing pool, that has a lot of data already on it.
By Default, FreeNAS Creates a 2GB SWAP partition at the front of every data disk. I removed the swap space and made it 1.5GB partition on each drive with 512MB remaining for Swap. I followed through every step, changing things as needed. (I have 3 disks, tutorial speaks of 4, My pool name is foxhole, the tutorial is zroot.) I was successful in decrypting my volume with geom_eli and mounted it successfully.
I did not skip any steps provided. I even copied every command I was given and altered them in a text file so they would suit my case.
Here is my problem now.
After finally restarting to test everything, The kernel begins starting, then I am spat at a mountroot terminal. It seems that geom_eli didn't make an attempt to decrypt my root volume. I have a suspicion why. Correct me if I am wrong.
At the start of the tutorial, I am given commands to create new geoms for the encrypted volume:
geli init -b -B /boot/zfs/bootdir/da0p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p4
geli init -b -B /boot/zfs/bootdir/da1p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p4
geli init -b -B /boot/zfs/bootdir/da2p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p4
Since my volume already exists, I cant perform those commands that would have created "/boot/zfs/bootdir/daXp4.eli" files.
I am really just guessing at this being the cause.
I noticed this when i attempted to perform:
mv bootdir/*.eli bootdir/boot/
Gave me "No Match."
I assumed those would have been created when the pool was decrypted.
I apologize for this post. I am trying to give as much info as I can without giving too much. I have been working on this for the last 18 hours. I would really love someone with a clear head to take a peek at this.
If I missed any useful information, let me know.
Turns out I was correct. The daXp4.eli files are necessary as it's the metadata of each disk. A reference point if you will.
By performing:
geli backup /dev/daXp4 /boot/daXp4.eli
It create the meta files required for geom to attempt a decryption at boot time.
I hope this helps someone else interested in this stuff.
I now have a NAS with 23 Disks. 3 ZFS Volumes, all encrypted with geom_eli
I created a program that iterates over a bunch of files and invokes for some of them:
scp <file> user#host:<remotefile>
However, in my case, there may be thousands of small files that need to transferred, and scp is opening a new ssh connection for each of them, which has quite some overhead.
I was wondering if there is no solution where I keep one process running that maintains the connection and I can send it "requests" to copy over single files.
Ideally, I'm looking for a combination of some sender and receiver program, such that I can start a single process (1) at the beginning:
ssh user#host receiverprogram
And for each file, I invoke a command (2):
senderprogram <file> <remotefile>
and pipe the output of (2) to the input of (1), and this would cause the file to be transferred. In the end, I can just send process (1) some signal to terminate.
Preferably the sender and receiver programs are open source C programs for Unix. They may communicate using a socket instead of a pipe, or any other creative solution.
However, it is an important constraint that each file gets transferred at the moment I iterate over it: it is not acceptable to collect a list of files and then invoke one instance of scp to transfer all the files at once at the end. Also, I have only simple shell access to the receiving host.
Update: I found a solution for the problem of the connection overhead using the multiplexing features of ssh, see my own answer below. Yet, I'm starting a bounty because I'm curious to find if there exists a sender/receiver program as I describe here. It seems there should exist something that can be used, e.g. xmodem/ymodem/zmodem?
I found a solution from another angle. Since version 3.9, OpenSSH supports session multiplexing: a single connection can carry multiple login or file transfer sessions. This avoids the set-up cost per connection.
For the case of the question, I can first open a connection with sets up a control master (-M) with a socket (-S) in a specific location. I don't need a session (-N).
ssh user#host -M -S /tmp/%r#%h:%p -N
Next, I can invoke scp for each file and instruct it to use the same socket:
scp -o 'ControlPath /tmp/%r#%h:%p' <file> user#host:<remotefile>
This command starts copying almost instantaneously!
You can also use the control socket for normal ssh connections, which will then open immediately:
ssh user#host -S /tmp/%r#%h:%p
If the control socket is no longer available (e.g. because you killed the master), this falls back to a normal connection. More information is available in this article.
This way would work, and for other things, this general approach is more or less right.
(
iterate over file list
for each matching file
echo filename
) | cpio -H newc -o | ssh remotehost cd location \&\& | cpio -H newc -imud
It might work to use sftp instead of scp, and to place it into batch mode. Make the batch command file a pipe or UNIX domain socket and feed commands to it as you want them executed.
Security on this might be a little tricky at the client end.
Have you tried sshfs?
You could:
sshfs remote_user#remote_host:/remote_dir /mnt/local_dir
Where
/remote_dir was the directory you want to send files to on the system you are sshing into
/mnt/local_dir was the local mount location
With this setup you can just cp a file into the local_dir and it would be sent over sftp to remote_host in its remote_dir
Note that there is a single connection, so there is little in the way of overhead
You may need to use the flag -o ServerAliveInterval=15 to maintain an indefinite connection
You will need to have fuse installed locally and an SSH server supporting (and configured for) sftp
May be you are looking for this:
ZSSH
zssh (Zmodem SSH) is a program for interactively transferring files to a remote machine while using the secure shell (ssh). It is intended to be a convenient alternative to scp , allowing to transfer files without having to open another session and re-authenticate oneself.
Use rsync over ssh if you can collect all the files to send in a single directory (or hierarchy of directories).
If you don't have all the files in a single place, please give some more informations as to what you want to achieve and why you can't pack all the files into an archive and send that over. Why is it so vital that each file is sent immediately? Would it be OK if the file was sent with a short delay (like when 4K worth of data has accumulated)?
It's a nice little problem. I'm not aware of a prepackaged solution, but you could do a lot with simple shell scripts. I'd try this at the receiver:
#!/bin/ksh
# this is receiverprogram
while true
do
typeset -i length
read filename # read filename sent by sender below
read size # read size of file sent
read -N $size contents # read all the bytes of the file
print -n "$contents" > "$filename"
done
At the sender side I would create a named pipe and read from the pipe, e.g.,
mkfifo $HOME/my-connection
ssh remotehost receiver-script < $HOME/my-connection
Then to send a file I'd try this script
#!/bin/ksh
# this is senderprogram
FIFO=$HOME/my-connection
localname="$1"
remotename="$2"
print "$remotename" > $FIFO
size=$(stat -c %s "$localname")
print "$size" > $FIFO
cat "$localname" > $FIFO
If the file size is large you probably don't want to read it at one go, so something on the order of
BUFSIZ=8192
rm -f "$filename"
while ((size >= BUFSIZ)); do
read -N $BUFSIZE buffer
print -n "$buffer" >> "$filename"
size=$((size - BUFSIZ))
done
read -N $size buffer
print -n "$contents" >> "$filename"
Eventually you'll want to extend the script so you can pass through chmod and chgrp commands. Since you trust the sending code, it's probably easiest to structure the thing so that the receiver simply calls shell eval on each line, then send stuff like
print filename='"'"$remotename"'"' > $FIFO
print "read_and_copy_bytes " '$filename' "$size" > $FIFO
and then define a local function read_and_copy_bytes. Getting the quoting right is a bear, but otherwise it should be straightforward.
Of course, none of this has been tested! But I hope it gives you some useful ideas.
Seems like a job for tar? Pipe its output to ssh, and on the other side pipe the ssh output back to tar.
I think that the GNOME desktop uses a single SSH connection when accessing a share through SFTP (SSH). I'm guessing that this is what's happening because I see a single SSH process when I access a remote share this way. So if this is true you should be able to use the same program for this purpose.
The new version of GNOME used GVFS through GIO in order to perform all kind of I/O through different backends. The Ubuntu package gvfs-bin provides various command line utilities that let you manipulate the backends from the command line.
First you will need to mount your SSH folder:
gvfs-mount sftp://user#host/
And then you can use the gvfs-copy to copy your files. I think that all file transfers will be performed through a single SSH process. You can even use ps to see which process is being used.
If you feel more adventurous you can even write your own program in C or in some other high level language that provides an API to GIO.
One option is Conch is a SSH client and server implementation written in Python using the Twsited framework. You could use it to write a tool which accepts requests via some other protocol (HTTP or Unix domain sockets, FTP, SSH or whatever) and triggers file transfers over a long running SSH connection. In fact, I have several programs in production which use this technique to avoid multiple SSH connection setups.
There was a very similar question here a couple of weeks ago. The accepted answer proposed to open a tunnel when ssh'ing to the remote machine and to use that tunnel for scp transfers.
Perhapse CurlFTPFS might be a valid solution for you.
It looks like it just mounts an external computer's folder to your computer via SFTP. Once that's done, you should be able to use your regular cp commands and everything will be done securely.
Unfortunately I was not able to test it out myself, but let me know if it works for ya!
Edit 1: I have been able to download and test it. As I feared it does require that the client have a FTP server. However, I have found another program which does has exactly the same concept as what you are looking for. sshfs allows you to connect to your client computer without needing any special server. Once you have mounted one of their folders, you can use your normal cp commands to move whatever files you need to more. Once you are done, it should then be a smile matter of umount /path/to/mounted/folder. Let me know how this works out!
rsync -avlzp user#remotemachine:/path/to/files /path/to/this/folder
This will use SSH to transfer files, in a non-slow way
Keep it simple, write a little wrapper script that does something like this.
tar the files
send the tar-file
untar on the other side
Something like this:
tar -cvzf test.tgz files ....
scp test.tgz user#other.site.com:.
ssh user#other.site.com tar -xzvf test.tgz
/Johan