I am wondering if a WebDAV server should store uploaded files in text-mode, if the mime-type is 'text/...'.
Unix, Windows and Mac OS use different line endings.
Opening a file in write+text mode may convert carriage return / newline according to the servers system convention (which may be different from the WebDAV client).
The obvious alternative would be to store all incoming files as binary blobs, without any conversion.
I see this pros for text-mode:
The text files can be opened on the server using text editors
The uploaded text files may also be easier to interpret by server software (i.e. xml parsers, script processors)
All clients get all text files with the same line ending convention (as defined by the server platform)
I think I have seen implementations doing this
and this cons of text mode
A client cannot expect to GET the same file content that was POSTed.
If a Windows WebDAV client stores a file to a Unix server, the file sizes are different.
I. e. the resources 'size' property is greater than the length of the data returned on GET.
Dangerous: if a file claims mime type 'text/foo', but the file is in fact binary (e.g. a zipped xml file), converting \n \r bytes will corrupt the file.
Text mode may be slower, since processing is required(?)
Am I missing something?
How do common WebDAV servers handle this? Is there a best practice?
I don't know how common WebDAV servers handle this, but I think it's a bad idea.
Just the risk of damaging binary files that are thought to be text, as you mentioned, makes it not worth doing. Here are some more disadvantages:
Breaks the model of the WebDAV server as a simple storage and retrieval device. As a user:
I'd view this as mangling my files.
I'd have to spend time figuring out how and why my files had changed.
Then I'd wonder what other things the server might be doing to my files on my behalf.
Changes line endings based on the server's OS, not the client's (customer's).
If I'm a Windows-only or Unix-only user, then all my line endings are right for me, and I don't want the server changing them. If I use both, then I already have tools that are either insensitive to the line endings or can convert between them.
My experience with text-processing client programs in recent years is that they're all insensitive to line endings. XML parsers and script interpreters, for example, can work with either style of line ending. So I don't see much benefit to offset the risk.
Related
I'm making an application which plays music from a remote server, and I would like to be able to sort by author/album/year/etc. AFAIK the only way to do this is by reading the metadata but I don't want to have to download the whole audio file just to read the metadata, is there any way to separate them and download only the metadata?
BTW. I am using webdav_client for flutter, which uses dio as a back-end so instructions for that specifically would be greatly appreciated. TY
Firstly, you can usually make a request for a certain byte range by using ranged requests. This is dependent on server behavior. Most servers support it, but many don't.
Next, you need to figure out the location of the ID3 tags you want. Some versions of ID3 are located at the front of the file. Some are at the back. Therefore, you should probably request the first 128 KB or so of the file and search for ID3 data, while also getting the Length response header. Then if you don't find your tag at the beginning, you can make a request for the last 128 KB or whatever of the file, and search there.
Most MP3 files aren't very big, and bandwidth is usually plentiful. Depending on the size and scope of this project, you might actually find it more efficient to just download the whole files.
I don't think that it is possible to read just the ID3-metadata (at the beginning or the end) of the audiofile without downloading the entire file first.
One idea would be to extract this information on the server side and provide it separately, in addition to the audio file itself. To do this, you would need one of the well-known extraction tools available for your platform. However, if you need to download hundreds or thousands of companion files, I am not sure about the reliability of such a system.
Let us assume we have a static file server (Nginx + Linux) that serves 10 files. The files are read almost as frequently as the server can process. However, some of the files need to be replaced with new versions, so that the filename and URL address remain unaltered. How to replace the files safely without a fear that some reads fail or become a mix of two versions?
I understand this is a rather basic operating system matter and has something to do with renames, symlinks, and file sizes. However, I failed to find a clear reference or a good discussion and I hope we can build one here.
Use rsync. Typically I choose rsync -av src dst, but YMMV.
What is terrific about rsync is that, in addition to having essentially zero cost when little or nothing changed, it uses atomic rename. So during file transfer, a ".fooNNNNN" temp file gets bigger and bigger. Once completed, rsync closes the file and renames it on top of "foo". So web clients either see all of the old, or all of the new file. Notice that range downloads (say from restart after error) are not atomic, exposing such clients to lossage, especially if bytes were inserted near beginning of file. SHA1 wouldn't validate for such a client, and he would have to restart his download from scratch. BTW, if these are "large" files, tell nginx to use zero-copy sendfile().
I am looking for a good solution by which we can prevent an exe file to be uploaded on server.
It will be best if we can discard the upload by just reading the file headers as soon as we receive them rather than waiting for entire file to upload.
I have already implemented the extension check, looking for a better solution.
There is a how and a when/where part. The how is fairly simple, as binary files do contain a header and the header is fairly easy to strip out and check. For windows files, you can check the article Executable-File Header Format. Similar formats are used for other binary types, so you can determine types you allow and those you do not.
NOTE: Linked article is for full querying of the file. There are cheap, down and dirty, shortcuts where you only examine a few bytes.
The when/where depends on how you are getting the files. If you are using a highly abstracted methodology (upload library), which is fairly normal, you may have to stream the entire file before you can start querying the bits. Whether it is streamed into memory or you have to save and delete depends on your coding and possibly even the library. If you control the streaming up, you have the ability to stream in the first bytes (header portion) and abort the process in mid stream.
The first point of access to uploaded data would be in a HttpModule.
Technically you can check before all the bytes are sent if you have an .exe on your hands and cancel the upload. It can get quite complicated depending on how far you want to take this.
I suggest you look at the HttpModule of Brettle's NeatUpload. Maybe it gives you a lead on how to deal with this on the level you want.
I think you can do that by a javascript by checking if the file end with .exe before submitting the data and also do the check server side.
In an AIR app you can use SQLite via the flash.data classes. It appears that by default the encoding of the database created is set to UTF-16le, which means that textual data is stored with two bytes per character, resulting in a nearly 100% overhead for ASCII-heavy database content.
The default for a SQLite database is UTF-8, assuming the shell program (sqlite3) is indicative. Presumably Adobe has decided to override this for some reason, but I'd prefer not to suffer the wasted storage space if possible.
A PRAGMA encoding = "UTF-8"; statement prior to writing anything to the database would normally resolve the issue, but it appears that's not allowed in AIR either.
My workaround for now is to use a "template.db" that I create ahead of time and bundled into the application. In this template.db I've set the encoding to UTF-8 manually. If the database file does not exist already when my program starts, I create it by copying that template to my database file, then proceed to open and use it normally. I've confirmed that TEXT data is then stored as UTF-8, as desired.
I haven't seen any ill effects yet, but this is hackish. Is there a better way to set the encoding to UTF-8? Or is it a Bad Idea for some reason?
With no other workarounds or answers found, I'm posting my workaround as the Answer. It worked fine in a PlayBook app for the last two years, so presumably has no unforeseen side-effects, at least in that environment:
My workaround for now is to use a "template.db" that I create ahead of time and bundled into the application. In this template.db I've set the encoding to UTF-8 manually. If the database file does not exist already when my program starts, I create it by copying that template to my database file, then proceed to open and use it normally. I've confirmed that TEXT data then now stored as UTF-8 as desired.
you may check the execution source file of the project like /bin/debug in C# Visual Studio projects.
The changed committed is not necessarily be executed in your db located in other folders.
Related to:
Flex SDK 3.5 - Check file mimetype
Is there a way to get a file's magic number in Flex SDK 3.5 in order to get the file type?
In my understanding the magic number is the first couple of bytes in a file. Also, in my opinion, this is a very loose term.
So assuming that the magic number is the first bytes in a file, you could open a byte stream and check the bytes read. I think this is the only reliable way (but also very error prone and might open security problems, think about IEs' content sniffing issues). The UNIX tool file actually does that and still, it is called a file type guesser.
The actual number depend on the file format you are trying to use. If you are lucky it's documented. If not, it might get painful.