The ChannelSftp class has versions of get() and put() methods not returning anything, or returning InputStream/OutputStream.
What's the use case for using the methods returning streams, and reading/writing the files byte by byte, versus using the easy get() and put() methods where you specify the source and destination paths, and let the program do everything for you?
My guess is if you are downloading and playing a video/audio file would be one case, but what if you just move files to/from one server to another? Any point in using the streams then?
Here is the documentation:
http://epaul.github.io/jsch-documentation/javadoc/com/jcraft/jsch/ChannelSftp.html#get(java.lang.String,%20java.lang.String)
As with any other I/O interface, the variants with streams are useful when you do not manipulate files, but in-memory data.
For example, you might have produced the content based on user input and you want to upload it. You do not need the local copy in a file. So you stream the in-memory data to SFTP.
Streams are also useful abstraction.
If you are uploading from a file or downloading to a file, use the overloads that take paths. Creating a file stream is unnecessary overhead in this case.
Related
My C++ application needs to support caching of files downloaded from the network. I started to write a native LRU implementation when someone suggested I look at using SQLite to store an ID, a file blob (typically audio files) and the the add/modify datetimes for each entry.
I have a proof of concept working well for the simple case where one client is accessing the local SQLite database file.
However, I also need to support multiple access by different processes in my application as well as support multiple instances of the application - all reading/writing to/from the same database.
I have found a bunch of posts to investigate but I wanted to ask the experts here too - is this a reasonable use case for SQLite and if so, what features/settings should I dig deeper into in order to support my multiple access case.
Thank you.
M.
Most filesystems are in effect databases too, and most store two or more timestamps for each file, i.e. related to the last modification and last access time allowing implementation of an LRU cache. Using the filesystem directly will make just as efficient use of storage as any DB, and perhaps more so. The filesystem is also already geared toward efficient and relatively safe access by multiple processes (assuming you follow the rules and algorithms for safe concurrent access in a filesystem).
The main advantage of SQLite may be a slightly simpler support for sorting the list of records, though at the cost of using a separate query API. Of course a DB also offers the future ability of storing additional descriptive attributes without having to encode those in the filename or in some additional file(s).
Each time a users visits my web site pages, several xml elements are written and removed from one predefined file.
Eventually there will be a situation, when on this same file a read and write operation will take place almost simultaniously. How do I handle concurrency in this case? Is there a special base class library class that can handle it for me?
XML files will not do this for you. You could set up a synchronizing object: the simplest would be to have a common function that accesses the file and use the lock key word. Use a Mutex for inter-process synchronization.
However, this will be a potential bottle neck.
It may be time to start using a database.
I need an SQLite implementation that allows me to have the db file encrypted on disk, for security reasons. I noticed that SQLite only works with regular files, and that there's no implementation that supports streams available (oddly enough, as many people seem to want one). If I had such an implementation, I could easily pass it a stream that encrypts/decrypts the file first.
After googling and reading about the matter, it seems like a custom VFS might solve the problem, implementing only the file methods to open, read, write etc. to a stream instead of a regular file (the other methods may keep the default behavior).
My question then is as follows:
1. Does that sound like the correct approach?
2. Is there really no such implementation available??
Thanks.
If you just need an encrypted sqlite database there is The SQLite Encryption Extension. If not- ignore my answer.
I have inherited some code which involves a scheduled task that writes data (obtained from an external source) to XML files, and a website that reads said XML files to get information to be presented to the visitor.
There is no synchronization in place, and needless to say, sometimes the scheduled task fails to write the file because it is currently open for reading.
The heart of the writer code is:
XmlWriter writer = XmlWriter.Create(fileName);
try
{
xmldata.WriteTo(writer);
}
finally
{
writer.Close();
}
And the heart of the reader code is:
XmlDocument theDocument = new XmlDocument();
theDocument.Load(filename);
(yep no exception handling at either end)
I'm not sure how to best approach trying to synchronize these. As far as I know neither XmlWriter.Create() nor XmlDocument.Load() take any parameters regarding file access modes. Should I manage the underlying FileStreams myself (with appropriate access modes) and use the .Create() and .Load() overloads that take Stream parameters?
Or should I just catch the IOExceptions and do some sort of "catch, wait a few seconds, retry" approach?
Provided that your web site does not need to write back to the XmlDocument that is loaded, I would load it via a FileStream that has FileShare.ReadWrite set. That should allow your XmlWriter in the other thread to write to the file.
If that does not work, you could also try reading the xml from the FileStream into a MemoryStream, and close the file as quickly as possible. I would still open the file with FileShare.ReadWrite, but this would minimize the amount of time your reader needs to access data in the file.
By using FileShare.ReadWrite (or FileShare.Write for that matter) as the sharing mode, you run the risk that the document is updated while you are still reading it. That could result in invalid XML content, preventing the XmlDocument.Load call from successfully parsing it. If you wish to avoid this, you could try synchronizing with a temporary "locking file". Rather than allowing file sharing, you prevent either thread from concurrently accessing, and when either of them is processing the file, write an empty, temporary file to disk that indicates this. When processing (reading or writing) is done, delete the temporary file. This prevents an exception from being thrown on either end, and allows you to synchronize access to the file.
There are a couple other options you could use as well. You could simply let both ends swallow any exception and wait a short time before trying again, although that isn't really the best design. If you understand the threading options of .NET well enough, you could also use a named system Mutex that both processes (your writing process and your web site process) know about. You could then use the Mutex to lock, and not have to bother with the locking file.
What should I use for writing a file to the response? There are two different options as I see it. Option one is to read the file into a stream and then write the bytes to the browser with
Response.BinaryWrite(new bytes[5])
Next option is to just write the file from the file system directly with Response.WriteFile. Any advantages/disadvantages with either approach?
Edit: Corrected typos
Response.TransmitFile is preferred if you have the file on disk and are using at least asp.net 2.0.
Response.WriteFile reads the whole file into memory then writes the file to response. TransmitFile "Writes the specified file directly to an HTTP response output stream without buffering it in memory."
The other consideration is whether this is a file which is written one time or frequently. If you are frequently writing this file, then you might want to cache it, thus Response.BinaryWrite makes the most sense.
If you have it in memory, I would not write it to the file system and use Response.WriteFile.