I have some process that creates text log files, and some backup server.
I want to move text log files to backup server (with remove source file). And on next rsync start if there is log file with the same name - append/concatenate it with previously moved file.
rsync -av --append --remove-source-files user#server1:/source/ user#server2:/destination/
And everything works great except - if there is file with the same name on source and destination, and destination/receiver have longer file than on source server - it's skipped. How I may disable this --append's file length verification? I want simply append files with the same name with no restrictions.
The append option doesn't work like that.
From the man page:
This causes rsync to update a file by appending data onto the end of
the file, which presumes that the data that already exists on the
receiving side is identical with the start of the file on the sending
side.
So, if you "append" a 2KB file to a 1KB file, the end file with be 2KB, and the first 1KB of the larger file will be discarded. Conversely, if you "append" a file of equal or smaller size, then the whole file will end up getting discarded.
By passing the --append option you're basically promising rsync that you know the start of the file will never change.
But, if you delete the file then the start of the file has changed. Likewise, logrotate will make bad things happen.
Conclusion: --append does not play nice with --remove-source-files.
The --append-verify option will solve the logrotate problem, but won't solve your problem because it will still discard data (albeit different data).
Related
I have a little bit complicate situation here:
I need to download files from a SFTP daily. I connect to the SFTP with username and SSH key, the keys have a passphrase.
This SFTP has no actual files. All the files on the server is 0 bytes. The server will dynamicly generate the file if it get a "get" command.
So when I connect the SFTP with Winscp, everything went perfectly.
But I have to do it in Synapse.
I managed to connect it in Pipeline with copy activity, and I managed to download all the files, but with no data content inside.
Does anyone know how I can download the files with content?
If you actually have files with content in SFTP location, then they should also be automatically copied using the pipeline in your Synapse. In case if you just want to copy the files that are having the content and ignore empty files, then you will have to use a get metadata activity to check the size of the file (i.e., > 0 bytes) and then filter those files only to copy to your desired destination. Using the childItems you can get the fileName, Type and Size and use these properties in the subsequent copy activity to only copy filter files to your destination.
I've got dir per environment on receiver with links to some files.
This is because, some files are shared between environment.
So what I would like, when I'm doing a rsync from my remote host to my receiver, that the retrieve files are place following the links. But currently, my rsync replace my local links by the retrived files.
Is there a way to tell rsync to follow links on receiver hosts?
If those links are pointing to directories, you cas use -K option which works flawlessly.
Other thing is when you have on the receiver links pointing to files (not dirs).
I am afraid, currently there is no simple way how to preserve the links in the destination and amend the files they are pointing to with the contents of local links/files those point to.
You might be interested in the -L option if you are sending files links from source but want to copy contents they point to rather than links themselves. However, this would also remove the corresponding links in the receive destination and as mentioned earlier, just change the files they are pointing to.
Check out https://serverfault.com/questions/245774/perform-rsync-while-following-sym-links for more information.
I have created a media type that accepts XML files and saves them to a custom publicly accessible location on the server.
Ideally I would like the file to be overwritten when the exact same file is uploaded. This does not happen, instead it creates a new file and adds a number on the end. I have "Create new Revision" turned off.
To get around this issue I thought I could just delete the file via the CMS. The uploaded file has status of "Permanent" and is used 0 places. I know the cron job cleans up files for you, but when I run the cron the file in question is still there. I figure it's because the file is set to permanent, but I don't see a way to flip this to temporary.
Any help is much appeciated.
There is a setting nested away in the file system settings, which lets you configure it to remove (or not removed) orphaned files. If drush isn't removing them despite having no usages recorded, I'd check this option isn't ticked.
The temporary and permanent status are used for storing temporary files during the upload/save process, so I wouldn't tinker with those too much.
If you fancy making the form yourself using the form API, then you can save the file programmatically using the FILE_EXISTS_REPLACE parameter.
https://api.drupal.org/api/drupal/core%21modules%21file%21file.module/function/file_save_data/8.5.x
I'm using Vagrant to manage a VirtualBox instance of Debian Wheezy with PHP 5.5 and nginx 1.4.4. My local environment is Mac OS X 10.9 with PHP 5.5 and Apache 2.2.24. I have a synced directory that points to my document root for this particular project.
My local machine has a VirtualHost setup that also points to the document root for this project. Therefore, I actually have two different URLs that I can point my browser to, and the same code will be executed. One URL hits the virtual machine, while the other URL hits my local Apache installation.
I've been able to edit several PHP files with no problems. I make the changes locally and they immediately show up on both the virtual machine and my local web server. However, whenever I try to edit CSS, things get bizarre. I'm trying to add a single line of CSS to a static CSS file. Something along the lines of:
.body {margin-top:50px}
When I make this change, the virtual machine goes nuts. I'm not sure if it's sending a corrupted file, or if it's just appending a bunch of weird characters at the end of the CSS file, but I can't even paste them here. I did a Charles dump of the request for the CSS file and the response looks like this:
I've looked at the file in vim through the virtual machine. I've looked at the file using multiple text editors. For the life of me I can't find anything wrong with it. When I load the exact same file through my local Apache installation, it works just fine:
Notice that the body declaration appears between button and footer, and there are no weird characters at the end of the document.
The other thing that I noticed is that my the whitespace is being manipulated at some point as well. The CSS file in question uses four spaces for indentation, but in the response from the virtual machine, the lines only have two spaces of indentation. You can't see it from the pictures because they're formatted by Charles, but I looked at the raw data. It's truly bizarre.
Do you have any idea what is causing the manipulation of my CSS file when it is being served via the Vagrant/nginx combo vs my local/Apache combo?
EDIT
I pasted some of the characters into a hexadecimal converter and it converted all the characters to question marks. I tried a binary to decimal conversion and they all turned into �, which from what I can find is "used to replace a character whose value is unknown or unrepresentable in unicode". I ran file --mime on the file and it returned rental-application.js: text/plain; charset=us-ascii. So...nginx settings maybe?
I'm having the exact same issue with JavaScript files. If I vagrant destroy and vagrant up is the only way I've found to resolve the issue so far. It doesn't make for quick troubleshooting when I have to reboot my VM between file saves.
Every method that I use to look at the file works. The only issue is when it is served through nginx, which makes me think I have some type of encoding setting wrong. It's a mostly stock setup.
More bizarre
If I mv rental-application.js rental-application.html and load the page in my browser, it comes across perfectly. The same applies if I change the name to rental-application.php. However, as soon as I change the name to rental-application.js or rental-application.css, my changes disappear and the � characters re-appear at the end of the document.
Even MORE bizarre
I can create a js file on the VM and it will load fine via the VM nginx server. If I subsequently modify the file I just created and reload it, the changes don't appear, just the garbage characters.
If I mv that file to a different name that ends in .js, the garbage character problem still persists. However, if I then cat the contents of the renamed file to a new file with the original name, all is well in the universe.
Just so if people end up here for the same issue, they have the solution on how to resolve this: you need to turn off sendfile() in your web server' settings.
For Apache: EnableSendfile off
For Nginx: sendfile off
Quick explanation: https://coderwall.com/p/ztskha
You can find more online if you need more details.
Now you can get a beer and enjoy your static file being transferred properly, not more corrupted files :)
When I create or open a file in UNIX using O_CREAT flag, st_mtime,st_ctime and st_atime of the file changes. But when I create or open a file using O_TRUNC flag, only the st_mtime and st_ctime changes and not the st_atime.
From my understanding, st_atime changes when the file is accessed. When we open or create a file using O_TRUNC flag, are we not accessing the file?
This question is a bit old, but an answer for future generations at least...
From stat(2) man page (on a host with a linux 2.6.32 kernel):
The field st_atime is changed by file accesses, for example, by execve(2), mknod(2), pipe(2),
utime(2) and read(2) (of more than zero bytes). Other routines, like mmap(2), may or may not
update st_atime.
The field st_mtime is changed by file modifications, for example, by mknod(2), truncate(2),
utime(2) and write(2) (of more than zero bytes). Moreover, st_mtime of a directory is changed by
the creation or deletion of files in that directory. The st_mtime field is not changed for changes
in owner, group, hard link count, or mode.
The field st_ctime is changed by writing or by setting inode information (i.e., owner, group, link
count, mode, etc.).