I am working on an encrypted filesystem that encrypts data just before writing it to disk and decrypts it right after reading from disk. Any file in disk is useless if not decrypted first. so far I changed standard read and write methods that filesystem reperesents.
the problem begins with mmap used for memory mapping files. for example in ext4 filesystem, as far as I know it does not use standard I/O so should be encrypted/decrypted just like read/write system calls. So how can I decrypt data when its being read from disk and encrypt it when kernel wants to update memory mapped files?
I want to stay in my filesystem specific module if it is possible.
UPDATE: read/write works in terminal perfectly. but:
I can not execute binary files in encrypted partition.
when I copy files using a GUI based filesystem (pcmanfm for example), result file is corrupted.
so should I edit any other system calls like I did with read/write?
Related
I have a requirement to perform checksum (for data integrity) for SFTP. I was hoping this could be done during the SFTP file transfer - I realize this could be product dependent (FYI: using CLEO VLTrader), but was wondering if this is customary?
I am also looking for alternative data integrity checking options that are as good (or better) than using a checksum algorithm. Thanks!
With the SFTP, running over an encrypted SSH session, there's negligible chance the file contents could get corrupted while transferring. The SSH itself does data integrity verification.
So unless the contents gets corrupted, when reading the local file or writing the remote file, you can be pretty sure that the file was uploaded correctly, if no error is reported. That implies that a risk of data corruption as about the same as if you were copying the files between two local drives.
If you would not consider it necessary to verify data integrity after copying the files from one local drive to another, then I do not think, you need to verify integrity after an SFTP transfer, and vice versa.
If you want to test explicitly anyway:
While there's the check-file extension to the SFTP protocol to calculate a remote file checksum, it's not widely supported. Particularly it's not supported by the most widespread SFTP server implementation, the OpenSSH. See What SFTP server implementations support check-file extension.
Not many clients/client libraries support it either. You didn't specify, what client/library you are using, so I cannot provide more details.
For details about some implementations, see:
Python Paramiko: How to check if Paramiko successfully uploaded a file to an SFTP server?
.NET WinSCP: Verify checksum of a remote file against a local file over SFTP/FTP protocol
What SFTP server implementations support check-file extension
Other than that, your only option is to download the file back (if uploading) and compare locally.
If you have a shell access to the server, you can of course try to run some shell checksum command (e.g. sha256sum) over a separate shell/SSH connection (or the "exec" channel) and parse the results. But that's not an SFTP solution anymore.
Examples:
Calculate hash of file with Renci SSH.NET in VB.NET
Comparing MD5 of downloaded files against files on an SFTP server in Python
I want to know what happens in the kernel when an open() system call is invoked? How does it return a file descriptor for a file?
The kernel creates internally a structure containing additional informations about the file you did just open. This structure holds informations such as the inode number, the name of the file on the file system, its size, its associated superblock, etc ...
In fact, within the kernel, it is the VFS (Virtual File System) that handles I/O operations on a file will it be local (on you hard disk) or remote (located on an FTP server for instance like ftpfs does).
Every file systems on GNU/Linux implements the same mechanisms of opening/reading/writing/closing files. This ensures every developers don't have to bother about what kind of file they are trying to access, no matter what kind of file you are interacting with, the same open(), read() ... APIs can be used. You can find additional informations on what the VFS is here and here (great article by IBM).
Finally, each file descriptor that is returned by let's say open is relative to your program, so the first file you might be opening will be associated to the file descriptor 3 and so on ... It is possible to find out what file descriptors are binded to each process on many GNU/Linux distributions via /proc/{pid_of_your_process}.
If you really want to dive deep, you can browse the source for many unix variants. For linux, check out http://lxr.linux.no/#linux+v3.9/fs/open.c -- search for SYSCALL_DEFINE3(open, to get to the actual "open" syscall.
The kernel:
looks for the file (hard drive, usb, named pipes, standard streams, ...)
if everything went well, saves itself a descriptor that you opened the file
returns you a descriptor
if you close() or the process exits, releases it's info about your open()
If I have a truecrypt file on a shared drive, if I mount it by using the shared path does my password data get sent in plain text across the network? Basically my question: is it safe to mount a truecrypt file across a network without copying the file to your local machine first.
Your password data is not sent across the network, because the cryptographic operations takes place on your computer, in the TrueCrypt driver. The password is used to derive a key that is used on your computer to decrypt the encrypted sectors sent across the network.
TrueCrypt FAQ has a section on this. I beleive item 2 is what you want to acheive. Their warning is that someone looking at the encrypted trafic could get some side-channel information, like the amount of data read and written, and the offset in the encrypted file.
Unless you want protection from your government or other well funded attacker, I beleive you should be ok, password wise. You might test what happens when a network failur occurs while writing a large file. It might corrupt the file system you mounted.
What I did:
mounted the TrueCrypt Drive and a TrueCrypt-Container with VeraCrypt (is newer)
created a windows (samba) and mac (afp) share of the drive and container with a password in the share settings (whatever software you use)
Mounting the container prevented it from being overwritten from some one else opening the container directly.
If I have to move a moderate amount of memory between two processes, I can do the following:
create a file for writing
ftruncate to desired size
mmap and unlink it
use as desired
When another process requires that data, it:
connects to the first process through a unix socket
the first process sends the fd of the file through a unix socket message
mmap the fd
use as desired
This allows us to move memory between processes without any copy - but the file created must be on a memory-mounted filesystem, otherwise we might get a disk hit, which would degrade performance. Is there a way to do something like that without using a filesystem? A malloc-like function that returned a fd along with a pointer would do it.
[Edit] Having a file descriptor provides also a reference count mechanism that is maintained by the kernel.
Is there anything wrong with System V or POSIX shared memory (which are somewhat different, but end up with the same result)? With any such system, you have to worry about coordination between the processes as they access the memory, but that is true with memory-mapped files too.
I have an executable program which outputs data to the harddisk e.g. C:\documents.
I need some means to intercept the data in Windows 7 before they get to the hard drive. Then I will encrypt the data and send it back to the harddisk. Unfortunately, the .exe file does not support redirection command i.e. > in command prompt. Do you know how I can achieve such a thing in any programming language (c, c++, JAVA, php).
The encryption can only be done before the plain data is sent to the disk not after.
Any ideas most welcome. Thanks
This is virtually impossible in general. Many programs write to disk using memory-mapped files. In such a scheme, a memory range is mapped to (part of) a file. In such a scheme, writes to file can't be distinguished from writes to memory. A statement like p[OFFSET_OF_FIELD_X] = 17; is a logically write to file. Furthermore, the OS will keep track of the synchronization of memory and disk. Not all logical writes to memory are directly translated into physical writes to disk. From time to time, at the whim of the OS, dirty memory pages are copied back to disk.
Even in the simpler case of CreateFile/WriteFile, there's little room to intercept the data on the fly. The closest you could achieve is the use of Microsoft Detours. I know of at least one snakeoil encyption program (WxVault, crapware shipped on Dells) that does that. It repeatedly crashed my application in the field, which is why my program unpatches any attempt to intercept data on the fly. So, not even such hacks are robust against programs that dislike interference.