I'm writing a multipart downloader in Qt. Multiple QNetWorkRequest with http header "Range" are used to download a file. Now I write data received in each part(QNetworkReply) to file.part1, file.part2, etc.
Is it possible to write data to the same file simultaneously? Do I have to implement a lock for it and which is the best way to save data in my application?
Any suggestions?
Why not just merge the file parts when you are finished? You can write the parts to a QFile easily. Perhaps something about the approach or the data keeps you from doing this, but if you can, it's probably the approach I would take before dealing with treating a QFile as a shared resource.
If you want multiple concurrent replies to be able to write to and access the QFile, then yes, the QFile becomes a shared resource. As far as I know, you're going to need to lock it. At that point, you have several options. You can have the slot(s) handling the replies attempt to acquire a QSemaphore, you can use QMutex and QMutexLocker if you'd prefer to lock on a mutex. You could treat it as a multiple producer (the various QNetworkReplys) single consumer (whatever is writing to the file) problem (here's a Stack Overflow post that provides some useful links) if you want to go that route. In short, there are numerous approaches here, all of which I think are more of a hassle than simply merging the file.part's at the end if you're able to go that route.
In terms of merging to a single QFile concurrently, there may be an easier Qt way of doing it, but I've never found it. Perhaps someone else can chime in if such a method exists.
I'm not sure what you mean by "which is the best way to save data in my application?" Are you referring to saving application specific settings in a persistent manner? If so, look into QSettings. If you're referring to saving the data you're downloading, I'd probably write it to a QFile, just like you appear to be doing. Although it's hard to know for sure without knowing more about what you're downloading.
Related
I need to append some bytes to an existing object stored in Openstack Swift, say like a log file object and constantly append new logs to it. Is this possible?
Moreover, can I change (overwrite) some bytes (specify with offset and length) to an existing object?
I believe ZeroVM (zerovm.org) would be perfect for doing this.
Disclaimer: I work for Rackspace, who owns ZeroVM. Opinions are mine and mine alone.
tl;dr: There's no append support currently in Swift.
There's a blueprint for Swift append support: https://blueprints.launchpad.net/swift/+spec/object-append. It doesn't look very active.
user2195538 is correct. Using ZeroVM + Swift (using the ZeroCloud middleware for Swift) you could get a performance boost on large-ish objects by sending deltas to a ZeroVM app and process them in place. Of course you still have to read/update/write the file, but you can do it in place. You don't need to pipe the entire file over the network, which could/would be costly for large files.
Disclaimer: I also work for Rackspace, and I work on ZeroVM for my day job.
I am new to Qt, and I was learning on its Getting Started Page. I want to know what does the following statements mean and why are they required?
In Open function:
if (!file.open(QIODevice::ReadOnly)) {
QMessageBox::critical(this, tr("Error"), tr("Could not open file"));
return;
}
Also in Save function:
if (!file.open(QIODevice::WriteOnly)) {
// error message
}
I was unable to run these functions without these lines. I tried reading about Error Handling in the documentation but couldn't exactly find what do these statements mean.
You can open files for reading and for writing. Using QIODevice::WriteOnly or QIODevice::ReadOnly flags you are specifying mode in which you will open particular file.
But, why does it matter?
Suppose you have one file opened in several instances of different programs, and that there is no such thing as specifying file mode. Now, if every files are reading file - since they all have different pointer to the current position in the file - this is not a problem - since all programs will get the latest and correct information from file. But, if only one programs write something into file - your data will be inconsistent, so other programs will potentially read wrong data.
Intuitive approach would be to send a message to all programs that are attached on this file, so they could update themselves. But - what to do if the file is deleted? Or if there is no possibility to set the proper position in the new data? Also, every program now needs to have interface in order to be notified, and the whole message passing idea can be very slow (aside that it doesn't work).
So - there is simply consensus made - multiple programs can open file for reading - as they will all have the same and consistent data. But, if the only one program is signaling the operating system that it wants to gain write permissions - the file must not be opened in any program - nor for reading - nor for writing! Depending on the implementation, operating system may block the caller until all files are closed, or it can simply ignore the call and send the error information to caller - which is often a better idea, as the program (or the user) can block itself and try again later, or it can simply ask user to save into another destination, or it can send us creepy error message - but it will not be able to write into file.
Last paragraph is describing what is known as multiple readers-single writer technique, so you may want to look it up on the internet or concurrency classes textbooks.
I will be creating a structure more or less of the form:
type FileState struct {
LastModified int64
Hash string
Path string
}
I want to write these values to a file and read them in on subsequent calls. My initial plan is to read them into a map and lookup values (Hash and LastModified) using the key (Path). Is there a slick way of doing this in Go?
If not, what file format can you recommend? I have read about and experimented with with some key/value file stores in previous projects, but not using Go. Right now, my requirements are probably fairly simple so a big database server system would be overkill. I just want something I can write to and read from quickly, easily, and portably (Windows, Mac, Linux). Because I have to deploy on multiple platforms I am trying to keep my non-go dependencies to a minimum.
I've considered XML, CSV, JSON. I've briefly looked at the gob package in Go and noticed a BSON package on the Go package dashboard, but I'm not sure if those apply.
My primary goal here is to get up and running quickly, which means the least amount of code I need to write along with ease of deployment.
As long as your entiere data fits in memory, you should't have a problem. Using an in-memory map and writing snapshots to disk regularly (e.g. by using the gob package) is a good idea. The Practical Go Programming talk by Andrew Gerrand uses this technique.
If you need to access those files with different programs, using a popular encoding like json or csv is probably a good idea. If you just have to access those file from within Go, I would use the excellent gob package, which has a lot of nice features.
As soon as your data becomes bigger, it's not a good idea to always write the whole database to disk on every change. Also, your data might not fit into the RAM anymore. In that case, you might want to take a look at the leveldb key-value database package by Nigel Tao, another Go developer. It's currently under active development (but not yet usable), but it will also offer some advanced features like transactions and automatic compression. Also, the read/write throughput should be quite good because of the leveldb design.
There's an ordered, key-value persistence library for the go that I wrote called gkvlite -
https://github.com/steveyen/gkvlite
JSON is very simple but makes bigger files because of the repeated variable names. XML has no advantage. You should go with CSV, which is really simple too. Your program will make less than one page.
But it depends, in fact, upon your modifications. If you make a lot of modifications and must have them stored synchronously on disk, you may need something a little more complex that a single file. If your map is mainly read-only or if you can afford to dump it on file rarely (not every second) a single csv file along an in-memory map will keep things simple and efficient.
BTW, use the csv package of go to do this.
Once I think about new software projects and current-age data uses, I cannot stand raw files anymore. they seem unnatural now.
Basically a file should contain one or more data "streams", metadata/attributes, etc.
The file should be optimized for sequential, parallel read (like mkv I think) but have reasonable performance for direct ("random") read access, possiby write access, and direct insertions.
Maybe even explicit logical data structure (inside each stream) might be useful.
mkv seems a pretty generic container format, but it seems to not completely fit the need and iirc has some multimedia-specific features in the container format itself
I see sqlite suggested as an alternative, but has an all-file locking mechanism and I just don't know how stream data gets organized (if it stays multimedia-friendly enough or if it's more single-access optimized)
Do you have any hint for me? Are there any other open, cross-platform, generic container, generic-access optimized, insertable data formats? What could I study?
How about HDF5?
I have a flex application that repeatedly polls a remote XML file to detect changes, but I've found that once the file hits a certain size, the poll blocks the UI and makes the page unresponsive for a short time.
Is there any way to ensure that the call made to the server or the event from the flash.utils.Timer class runs asynchronously to the main UI thread?
It sounds like the blocking is caused by Flash parsing the XML rather than the actual loading.
If that is the case then you can keep loading the file and just check the size of the raw data you get back - if it's bigger, parse it and take the parsing hit. Otherwise toss the data and wait for the next request.
There's no explicit way to do threading with Flash at this time. Certain tasks happen async naturally (network and pixelbender comes to mind) but that's it.
Branden's right -- the code we write essentially always happens on the main thread; while the network call itself does happen on a background thread, the handling of that call happens on the main one.
One thing to keep in mind is that the WebService and HTTPService classes will likely attempt to serialize your responses automatically, which could mean taking that processing hit unnecesarily. Using URLLoader, on the other hand, could give you more direct access to the response data, allowing you to work more directly with it without the unnecessary overhead of that built-in processing.
In that light, if you find you do have to process the entire XML file, you might consider breaking it up into chunks somehow, and distributing the processing of those chunks into separate functions, rather than handling everything within the scope of a single function. Just doing that might allow the player to continue updating the UI while you're processing that big bunch of text (processing a bit, exiting the function, rendering the UI, entering the next function, rendering, and so on); Oliver Goldman, an engineer on the AIR team, did a presentation on this concept at last year's MAX conference.
Hope it helps!
Like mentioned, AS3 is single threaded. But there are a couple of ways to handle your situation. Here are ur possible choices:
First,make sure you really need to parse entire XML when loaded and cant just keep the loaded xml nodes in memory as the data model (XML being a native data type now). Sometimes I create value objects by passing them an XMLNode and thats kept in memory till a node value is needed at which point I read it. This allows you to keep some of the parsing for later.
If you are using an ArrayCollection or similar structure to hold data, try using a primitive Array ( see http://www.arpitonline.com/blog/?p=114 for an issue I ran into)
See if you can creatively use callLater()(http://livedocs.adobe.com/flex/2/langref/mx/core/UIComponent.html#callLater())
Can you pass the data to the client in a native format like SWX or using Remoting
Can you use data paging? ArrayCollections and I am pretty sure XMLCollection support it
Side Note:
While AS3 is single threaded, Pixel Bender and I think Alchemy (http://labs.adobe.com/technologies/alchemy/) run on a different thread. There have been a couple of experiments on blogs using Pixel Bender to do calculations which dont slow the UI of the app (Example:http://elromdesign.com/blog/2009/02/09/using-pixel-bender-to-do-heavy-lifting-calculations-makes-flash-player-multi-thread/).
Also please vote on this feature enhancement ticket with Adobe if you feel the need for this feature enough:
https://bugs.adobe.com/jira/browse/ASC-3222