How to read bigger than 600mb file in qt?
I am trying to read file using file.readAll(). It works small files. but it gives bad_alloc error in large files? what should I do?
Try adding
QMAKE_LFLAGS += -Wl,--large-address-aware
to your qt pro file, from what I understand it will allow a process to accumulate more memory than 2GB.
Don't do it.
It's rarely necessary to load a huge file into memory in one operation.
You can't be loading this much information for user navigation or manipulation, so if, as I suspect, you are simply acting as an intermediary between having the file on disc and sending the file somewhere else then use a mechanism which treats the Qfile as a QIODevice instead of loading it all completely as a QString or QByteArray.
If you (or your customers) are on Windows using a 32 bit system but are likely to have more than 2Gb of RAM at your disposal, you might want to be aware of the /LARGEADDRESSAWARE linker option which will allow you to support addresses larger than 2Gb and which may also improve your situation if you are truly unable to do without loading the file into memory.
Consider reading the file in chunks, instead of not all at once. Of course, your goal might be to display the entire file in a text editor, in which case loading it partially is more complicated. You're being very vague, so it's hard to be more specific.
Related
What is the max file size limit for .r extension file now?
I read that it has 5MB limit, is it still the same? How does that change, will it be different from OS to OS or R version to version.
Reference: RStudio maximum file size reached
I'm very new to R, can someone please help me?
Thanks
There is no documented limit for the maximum file size or R code files. In fact, R will be able to deal with anything that’s even remotely reasonable. But for what it’s worth a 5 MiB source code file is not reasonable. If you actually have such files, I strongly suggest removing the large data that’s declared inside it, and moving it to a proper data file instead: separate your code and data. Actual code will never be this big.
As for editing such a file, different code editors have different limits for the size of files they deal well with. Again, having such a big code file is plain unreasonable, so not many code editors bother catering to this use-case, and even though few editors have a hard-coded limit, interactively editing such large files may not work.
I am attempting to read data files stored as a .txt, some of which are very large (>1 GB). It seems that every time QFile attempts to use the .open() method on files larger than 600MB, it freezes and crashes. Is there a better way to open large files than QFile? None of the code below the if (_file.open(QIODevice::ReadOnly)) line shown below executes, so I believe the crash occurs where the open method is called.
I understand from answers to similar questions that reading in large text files is not a great way to handle huge amounts of data, but unfortunately these are log files that I have no control over. I need to be able to read these files OR elegantly handle/ignore an oversized file, but I can't find information on how to detect the maximum read size. I would rather not have to manually open and split these files in a text editor, as I have about a terabyte of these to process and manually splitting could lead to loss of important information. I am not overly concerned with the responsiveness of this program, and any method used to open files can sit and think for quite awhile, as this program will be used for data processing not any kind of user interaction.
Thanks for your help
Code:
void FileRead::openNewFile()
{
if(_listOfFiles.size()>0)
{
_file.setFileName(_listOfFiles.at(0));
if (_file.open(QIODevice::ReadOnly)) //file opened successfully
{
_file.reset();
emit fileOpened();
emit fileOpened(_file.fileName());
qDebug()<<"File Opened";
qDebug()<<_file.fileName();
}
else
{
qDebug()<<"Unable to open file";
qDebug()<<_listOfFiles;
_listOfFiles.removeAt(0);
emit fileSent();
}
}
else
{
qDebug()<<"All files processed";
}
}
I think you're re-using a QFile that's already open, and this might be problematic.
The call to reset() is pointless - you've just opened the file, it is reset by definition.
You have not provided a backtrace of where exactly does your code crash. I cannot reproduce your problem - I have a 16GB sparse file that I can open, read from, and close, successfully, on both Qt 4.8 and Qt 5.2, on both Windows 7 and OS X.
If you write a minimal test case for this (a stand-alone application that does nothing but opens the file, reads a few bytes from it, and closes it), you'll likely find that it doesn't crash - the problem is elsewhere in your code.
I wrote a tool which was originally thought for analyzing hard disc images. Now I'm trying to use this tool for live analyzis of computer systems, means my tool tries to access the physical drive.
I implemented my tool in QT accessing the images using the QFile class. Instead of images I hand over the physical drive, under windows it is \.\PHYSICALDRIVE0.
I was wondering first I didnt get any errors, I can open the device, I can seek, get the position, almost everything. The only thing I have problems with is retrieving the drive size with size().
Some code example:
QFile file( "\\.\PHYSICALDRIVE0" );
file.open( QIODevice::ReadOnly );
file.size(); //returns 0
I'm not too deep into QT, probably this is some easy thing. I would like to thank everybody who has an idea what is the reason.
thanks in advance!
QFileInfo may be able to help you out. It sounds like opening a read only file at that part of windows partition is allowed maybe even if it doesn't exist. There might be a chance that the call of GetLastError() may give more information why a file size of zero was returned.
With QFileInfo, you can check to make sure it exists before it opens.
You may end up needing some platform specific calls to be able to work with Physical Drives:
Volume to physical drive
It looks like there may be some example code of looking at partitions with PartMod on SourceForge.
As a side note of querying sizes of file folders, I thought it had to be cached somewhere by the operating system, or had to be calculated at the time of the query in many cases. I know it seems like that happens when looking at folder properties in Windows or Get Info on OSX.
Also, looking at the Volume to physical drive answers, there is a field there for the extent length. I think this is what you are looking for.
Hope that helps.
I'm writing a multipart downloader in Qt. Multiple QNetWorkRequest with http header "Range" are used to download a file. Now I write data received in each part(QNetworkReply) to file.part1, file.part2, etc.
Is it possible to write data to the same file simultaneously? Do I have to implement a lock for it and which is the best way to save data in my application?
Any suggestions?
Why not just merge the file parts when you are finished? You can write the parts to a QFile easily. Perhaps something about the approach or the data keeps you from doing this, but if you can, it's probably the approach I would take before dealing with treating a QFile as a shared resource.
If you want multiple concurrent replies to be able to write to and access the QFile, then yes, the QFile becomes a shared resource. As far as I know, you're going to need to lock it. At that point, you have several options. You can have the slot(s) handling the replies attempt to acquire a QSemaphore, you can use QMutex and QMutexLocker if you'd prefer to lock on a mutex. You could treat it as a multiple producer (the various QNetworkReplys) single consumer (whatever is writing to the file) problem (here's a Stack Overflow post that provides some useful links) if you want to go that route. In short, there are numerous approaches here, all of which I think are more of a hassle than simply merging the file.part's at the end if you're able to go that route.
In terms of merging to a single QFile concurrently, there may be an easier Qt way of doing it, but I've never found it. Perhaps someone else can chime in if such a method exists.
I'm not sure what you mean by "which is the best way to save data in my application?" Are you referring to saving application specific settings in a persistent manner? If so, look into QSettings. If you're referring to saving the data you're downloading, I'd probably write it to a QFile, just like you appear to be doing. Although it's hard to know for sure without knowing more about what you're downloading.
I need to parse a large trace file (up to 200-300 MB) in a Flex application. I started using JSON instead of XML hoping to avoid these problems, but it did not help much. When the file is bigger than 50MB, JSON decoder can't handle it (I am using the as3corelib).
I have doing some research and I found some options:
Try to split the file: I would really like to avoid this; I don't want to change the current format of the trace files and, in addition, it would be very uncomfortable to handle.
Use a database: I was thinking of writing the trace into a SQLite database and then reading from there, but that would force me to modify the program that creates the trace file.
From your experience, what do you think of these options? Are there better options?
The program that writes the trace file is in C++.
Using AMF will give you much smaller data sizes for transfer because it is a binary, not text format. That is the best option. But, you'll need some middleware to translate the C++ program's output into AMF data.
Check out James Ward's census application for more information about benchmarks when sharing data:
http://www.jamesward.com/census/
http://www.jamesward.com/2009/06/17/blazing-fast-data-transfer-in-flex/
Maybe you could parse the file into chunks, without splitting the file itself. That supposes some work on the as3 core lib Json parser, but it should be doable, I think.
I found this library which is a lot faster than the official one: https://github.com/mherkender/actionjson
I am using it now and works perfectly. It also has asynchronous decoder and encoder