I have a 3 Disk RAIDz1 Pool, encrypted with AES128 in GEOM_ELI, that I have been using in FreeNAS since version 8.
There have been many zpool upgrades, and over all I am very happy with ZFS.
Lately however I have been growing frustrated with FreeNAS. Largely many bugs that haven't been fixed over the years. But overall its the INSISTING on me using a Flash drive for their os, even though most of it is read only.
It's still a Single point of failure and has always extended boot times by several minutes. Bottom line, I just want to use Vanilla FreeBSD with this pool.
I am looking for more flexibility and a I wish to educate myself with this awesome Operating System.
Doing some more extended research I have found many tutorials on installing FreeBSD naively to a ZFS volume and mounting it as /
It wasn't till I did more research and found an article on mounting a zfs encrypted volume as root. Later I found that FreeBSD 10 does this during installation, which is awesome to say the least.
Tutorial I used
I made a VM With VMWare workstation, with three 2TB Drives, passed through as Physical Disks, and followed every step to a T and everything worked out very well. Now that I had a better grasp on the commands I was doing and why I was doing them, I wanted to do this to an already existing pool, that has a lot of data already on it.
By Default, FreeNAS Creates a 2GB SWAP partition at the front of every data disk. I removed the swap space and made it 1.5GB partition on each drive with 512MB remaining for Swap. I followed through every step, changing things as needed. (I have 3 disks, tutorial speaks of 4, My pool name is foxhole, the tutorial is zroot.) I was successful in decrypting my volume with geom_eli and mounted it successfully.
I did not skip any steps provided. I even copied every command I was given and altered them in a text file so they would suit my case.
Here is my problem now.
After finally restarting to test everything, The kernel begins starting, then I am spat at a mountroot terminal. It seems that geom_eli didn't make an attempt to decrypt my root volume. I have a suspicion why. Correct me if I am wrong.
At the start of the tutorial, I am given commands to create new geoms for the encrypted volume:
geli init -b -B /boot/zfs/bootdir/da0p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da0p4
geli init -b -B /boot/zfs/bootdir/da1p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da1p4
geli init -b -B /boot/zfs/bootdir/da2p4.eli -e AES-XTS -K /boot/zfs/bootdir/encryption.key -l 256 -s 4096 /dev/da2p4
Since my volume already exists, I cant perform those commands that would have created "/boot/zfs/bootdir/daXp4.eli" files.
I am really just guessing at this being the cause.
I noticed this when i attempted to perform:
mv bootdir/*.eli bootdir/boot/
Gave me "No Match."
I assumed those would have been created when the pool was decrypted.
I apologize for this post. I am trying to give as much info as I can without giving too much. I have been working on this for the last 18 hours. I would really love someone with a clear head to take a peek at this.
If I missed any useful information, let me know.
Turns out I was correct. The daXp4.eli files are necessary as it's the metadata of each disk. A reference point if you will.
By performing:
geli backup /dev/daXp4 /boot/daXp4.eli
It create the meta files required for geom to attempt a decryption at boot time.
I hope this helps someone else interested in this stuff.
I now have a NAS with 23 Disks. 3 ZFS Volumes, all encrypted with geom_eli
Related
I get huge .cap files by iptrace on AIX. The file is about 800MB. I'm on MacOS, and tshark has been running for a whole day parsing it.
CPU of my host keeps 99% occupied. I really need to speed it up.
I've already added -n flag of tshark.
I'm thinking about adding frame number range to the filter, which should narrow down the amount of packets for analysis. But I don't know the total amount of frames, therefore can't really add that parameter.
Can I browse some general info about the .cap file before fully open it?
Is there anything else to do to remarkably speed up tshark performance?
Thanks.
Perhaps TShark is stuck in an infinite loop, in which case the problem isn't "the file is too big" (I have a capture that's 776MB, and it takes only a few minutes to run it through tshark -V, albeit on a 2.8 GHz Core i7 MBP with 16GB of main memory), the problem is "TShark/Wireshark have a bug".
File a bug on the Wireshark Bugzilla, specifying the TShark command that's been running for a day (all command-line arguments). You may have to either provide the capture file for testing by the Wireshark developers or run test versions of TShark.
What's the UNIX command to see the processes table, remember that table contains:
process status
pointers
process size
user ids
process ids
event descriptors
priority
etc
The "process table" as such lives in the kernel's memory. Some systems (such as AIX, Solaris and Linux--which is not "unix") have a /proc filesystem which makes those tables visible to ordinary programs. Without that, programs such as ps (on very old systems such as SunOS 4) required elevated privileges to read the /dev/kmem (kernel memory) special device, as well as having detailed knowledge about the kernel memory layout.
Your question is open ended, and an answer to a specific question you may have had can be looked up in any man page as #Alfasin suggests in his answer. A lot depends on what you are trying to do.
As #ThomasDickey points out in his response, in UNIX and most of its' derivatives, the command for viewing processes being run in the background or foreground is in fact the ps command.
ps stands for 'process status', answering your first bullet item. But the command uses over 30 options and depending on what information you seek, and permissions granted to you by the systems administrator, you can get various types of information from the command.
For example, for the second bullet item on your list above, depending on what you are looking for, you can get information on 3 different types of pointers - the session pointer (with option 'sess'), the terminal session pointer (tsess), and the process pointer (uprocp).
The rest of your items that you have listed are mostly available as standard output of the command.
Some UNIX variants implement a view of the system process table inside of the file system to support the running of programs such as ps. This is normally mounted on /proc (see #ThomasDickey response above)
Typical reasons for understanding the working of the command include system-administration responsibilities such as tracking the origin of the initiated processes, killing runaway or orphaned processes, examining the file size of the process and setting limits where necessary, etc. UNIX developers can also use it in conjunction with ipc features, etc. An understanding of the process table and status will help with associated UNIX features such as the kvm interface to examine crash dump, etc. or to get or set the kernal state.
Hope this helps
MPI require I deploy mpi program to each machine. Currently, I put the mpi program in nfs, but this method has 2 issues, one is nfs has latency issue and the other is nfs not suitable for large cluster. I know that I could use some linux shell commands to sync up my program to each node, but looks like not so convenient. especially, when I change the program frequently. Is there any easy method to to that ?
There's nothing wrong with NFS or any other network filing system in large clusters. It just means your file server isn't sized for the job. If you replace NFS with anything like ssh, ftp, scripts, or whatever and change nothing else, I don't think that'll make any significant difference. Also, if the loading time is a significant and bothersome component of the overall runtime then why use MPI in the first place?
OK, enough of playing devils advocate. One thing you can do is to have nodes load your program onto other nodes in a binary tree type arrangement. You'll need a script that will copy the executable to two other nodes along with a copy of the script, start that script running asynchronously on those nodes and then runs the executable locally. The result would be a chain reaction of copying and running spreading across the network. The only difficult bit is in choosing which nodes to copy to so that each one is visited just once. It will be a lot faster.
Depending on the nature of the application and the nature of the NFS network, using a shared file system for both the MPI implementation and the application "should" be able to scale with reasonable performance, to a point. Keep in mind that there is some NFS caching at the node level, so multiple ranks on the same node will not each have to traverse the network to reach the files.
In general terms, I tend to advise that NFS be discontinued at about 128 nodes or 1024 ranks in favor of local installations. That advice changes if the NFS is delivered with 10GigE, IPoIB, or if a high performance file system like SFS or GPFS is used.
If you are committed to local installations, then tools like rsync, or scp are good candidates to distribute the bits. Script the final result. You can even do a tar to shared, and remote command (e.g. ssh, clush) un-tar to local disc. The "solution" only needs to be robust, not polished or elegant.
I'll also chime in to say the NFS should be just fine in this use-case, unless you have a cluster of over 100-200 nodes.
If you just want a lightweight tool for doing many-node parallel operations, I'd suggest pdsh. pdsh is a very common tool on HPC clusters. It includes a command called pdcp for doing parallel node copies, i.e.
pdcp -w node[00-99] myfile /path/to/destination/myfile
Where the nodenames are node00, node01, ... node99.
Similarly, you use the pdsh command to run a command in parallel across all the nodes. I.e.,
pdsh -w node[00-99] /path/to/my/executable
Alternatively, if you're looking for something a little less ad-hoc for doing these operations, I can recommend Ansible as an easy and lightweight configuration management and deployment tool. It's not as simple to get started as pdsh, but might be more manageable in the long run...
For example, a simple Ansible playbook to copy a tarball to all nodes, extract it, and then execute a binary might look like:
---
- hosts: computenodes
user: myname
vars:
num_procs: 32
tasks:
- name: copy and extract tarball to deployment location
action: unarchive src=myapp.tar.gz dest=/path/to/deploy/
- name: execute app
action: command mpirun -np {{num_procs}} /path/to/deploy/myapp.exe
An article about setting up Ghost blogging says to use scp to copy from my local machine to a remote server:
scp -r ghost-0.3 root#*your-server-ip*:~/
However, Railscast 339: Chef Solo Basics uses scp to copy in the opposite direction (from the remote server to the local machine):
scp -r root#178.xxx.xxx.xxx:/var/chef .
In the same Railscast, when the author wants to copy files to the remote server (same direction as the first example), he uses rsync:
rsync -r . root#178.xxx.xxx.xxx:/var/chef
Why use the rsync command if scp will copy in both directions? How does scp differ from rsync?
The major difference between these tools is how they copy files.
scp basically reads the source file and writes it to the destination. It performs a plain linear copy, locally, or over a network.
rsync also copies files locally or over a network. But it employs a special delta transfer algorithm and a few optimizations to make the operation a lot faster. Consider the call.
rsync A host:B
rsync will check files sizes and modification timestamps of both A and B, and skip any further processing if they match.
If the destination file B already exists, the delta transfer algorithm will make sure only differences between A and B are sent over the wire.
rsync will write data to a temporary file T, and then replace the destination file B with T to make the update look "atomic" to processes that might be using B.
Another difference between them concerns invocation. rsync has a plethora of command line options, allowing the user to fine tune its behavior. It supports complex filter rules, runs in batch mode, daemon mode, etc. scp has only a few switches.
In summary, use scp for your day to day tasks. Commands that you type once in a while on your interactive shell. It's simpler to use, and in those cases rsync optimizations won't help much.
For recurring tasks, like cron jobs, use rsync. As mentioned, on multiple invocations it will take advantage of data already transferred, performing very quickly and saving on resources. It is an excellent tool to keep two directories synchronized over a network.
Also, when dealing with large files, use rsync with the -P option. If the transfer is interrupted, you can resume it where it stopped by reissuing the command. See Sid Kshatriya's answer.
Finally, note that rsync:// the protocol is similar to plain HTTP: unencrypted and no integrity checks. Be sure to always use rsync via SSH (as in the examples from the question above), not via the rsync protocol, unless you really know what you're doing. scp will always use SSH as underlying transport mechanism which has both integrity and confidentiality guarantees, so that is another difference between the two utilities.
rysnc can be useful to run on slow and unreliable connections. So if your download aborts in the middle of a large file rysnc will be able to continue from where it left off when invoked again.
Use rsync -vP username#host:/path/to/file .
The -P option preserves partially downloaded files and also shows progress.
As usual check man rsync
Difference b/w scp and rsync on different parameter
1. Performance over latency
scp : scp is relatively less optimise and speed
rsync : rsync is comparatively more optimise and speed
https://www.disk91.com/2014/technology/networks/compare-performance-of-different-file-transfer-protocol-over-latency/
2. Interruption handling
scp : scp command line tool cannot resume aborted downloads from lost network connections
rsync : If the above rsync session itself gets interrupted, you can resume it as many time as you want by typing the same command. rsync will automatically restart the transfer where it left off.
http://ask.xmodulo.com/resume-large-scp-file-transfer-linux.html
3. Command Example
scp
$ scp source_file_path destination_file_path
rsync
$ cd /path/to/directory/of/partially_downloaded_file
$ rsync -P --rsh=ssh userid#remotehost.com:bigdata.tgz ./bigdata.tgz
The -P option is the same as --partial --progress, allowing rsync to work with partially downloaded files. The --rsh=ssh option tells rsync to use ssh as a remote shell.
4. Security :
scp is more secure. You have to use rsync --rsh=ssh to make it as secure as scp.
man document to know more :
scp : http://www.manpagez.com/man/1/scp/
rsync : http://www.manpagez.com/man/1/rsync/
One major feature of rsync over scp (beside the delta algorithm and encryption if used w/ ssh) is that it automatically verifies if the transferred file has been transferred correctly. Scp will not do that, which occasionally might result in corruption when transferring larger files. So in general rsync is a copy with guarantee.
Centos manpages mention this the end of the --checksum option description:
Note that rsync always verifies that each transferred file was
correctly reconstructed on the receiving side by checking a whole-file
checksum that is generated as the file is transferred, but that
automatic after-the-transfer verification has nothing to do with this
option’s before-the-transfer “Does this file need to be updated?”
check.
There's a distinction to me that scp is always encrypted with ssh (secure shell), while rsync isn't necessarily encrypted. More specifically, rsync doesn't perform any encryption by itself; it's still capable of using other mechanisms (ssh for example) to perform encryption.
In addition to security, encryption also has a major impact on your transfer speed, as well as the CPU overhead. (My experience is that rsync can be significantly faster than scp.)
Check out this post for when rsync has encryption on.
scp is best for one file.
OR a combination of tar & compression for smaller data sets
like source code trees with small resources (ie: images, sqlite etc).
Yet, when you begin dealing with larger volumes say:
media folders (40 GB)
database backups (28 GB)
mp3 libraries (100 GB)
It becomes impractical to build a zip/tar.gz file to transfer with scp at this point do to the physical limits of the hosted server.
As an exercise, you can do some gymnastics like piping tar into ssh and redirecting the results into a remote file. (saving the need to build
a swap or temporary clone aka zip or tar.gz)
However,
rsync simplify's this process and allows you to transfer data without consuming any additional disc space.
Also,
Continuous (cron?) updates use minimal changes vs full cloned copies speed
up large data migrations over time.
tl;dr
scp == small scale (with room to build compressed files on the same drive)
rsync == large scale (with the necessity to backup large data and no room left)
it's better to think in a practical context. In our team, we use rsync -aP to replace a bad cassandra host in our cluster. We can't do this with scp (slow and no progress preservation).
I'm using NFS to allow two servers two communicate via simple text files, however sometimes it seems that the server reading the text files to get information is reading incomplete files, and then because of this crashes. Then I go to look at the "incomplete" file that made it crash, and the file is complete. Is it possible that the server reading these files is seeing them before they are complete written by NFS? I use linux's mv to move them from the local machine to NFS only when they are completely written, so there "should" never be an incomplete state on the NFS.
Could this problem have something to do with sync vs async? Right now I'm using async. From my understanding, async just means that you return from the write and your program can continue running, and this write will happen at a later time. Whereas sync means that your process will wait for that write to go through before it moves on. Would changing to sync fix this? Or is there a better way to handle this? I know two servers could communicate via a database, but I'm actually doing this to try to keep database usage down. Thanks!
mv across file-systems translates into cp+rm and is certainly not atomic, even without NFS involved. You should first copy the file to a temporary in the target file-system and then rename it to the correct name. For instance, instead of:
$ mv myfile.txt /mnt/targetfs/myfile.txt
do:
$ mv myfile.txt /mnt/targetfs/.myfile.txt.tmp
$ mv /mnt/targetfs/.myfile.txt.tmp /mnt/targetfs/myfile.txt
(This assumes that the process reading the file ignores it while it does not have the correct name.)