I am trying to read a very large file in AS3 and am having problems with the runtime just crashing on me. I'm currently using a FileStream to open the file asynchronously. This does not work(crashes without an Exception) for files bigger than about 300MB.
_fileStream = new FileStream();
_fileStream.addEventListener(IOErrorEvent.IO_ERROR, loadError);
_fileStream.addEventListener(Event.COMPLETE, loadComplete);
_fileStream.openAsync(myFile, FileMode.READ);
In looking at the documentation, it sounds like the FileStream class still tries to read in the entire file to memory(which is bad for large files).
Is there a more suitable class to use for reading large files? I really would like something like a buffered FileStream class that only loads the bytes from the files that are going to be read next.
I'm expecting that I may need to write a class that does this for me, but then I would need to read only a piece of a file at a time. I'm assuming that I can do this by setting the position and readAhead properties of the FileStream to read a chunk out of a file at a time. I would love to save some time if there is a class like this that already exists.
Is there a good way to process large files in AS3, without loading entire contents into memory?
You can use the fileStream.readAhead and fileStream.position properties to set how much of the file data you want read, and where in the file you want it to be read from.
Lets say you only want to read megabyte 152 of a gigabyte file. Do this;
(A gigabyte file consists of 1073741824 bytes)
(Megabyte 152 starts at 158334976 bytes)
var _fileStream = new FileStream();
_fileStream.addEventListener(Event.COMPLETE, loadComplete);
_fileStream.addEventListener(ProgressEvent.PROGRESS, onBytesRead);
_fileStream.readAead = (1024 * 1024); // Read only 1 megabyte
_fileStream.openAsync(myFile, FileMode.READ);
_fileStream.position = 158334976; // Read at this position in file
var megabyte152:ByteArray = new ByteArray();
function onBytesRead(e:ProgressEvent)
{
e.currentTarget.readBytes(megabyte152);
if (megabyte152.length == (1024 * 1024))
{
chunkReady();
}
}
function chunkReady()
{
// 1 megabyte has been read successfully \\
// No more data from the hard drive file will be read unless _fileStream.position changes \\
}
Can't you create a stream, and read a chunk of bytes at a given offset, a chunk at a time... so:
function readPortionOfFile(starting:int, size:int):ByteArray
{
var bytes:ByteArray ...
var fileStream:FileStream ...
fileStream.open(myFile);
fileStream.readBytes(bytes, starting, size);
fileStream.close();
return bytes;
}
and then repeat as required. I don't know how this works, and haven't tested it, but I was under the impression that this works.
Related
I want to create a soundwave in my java programm from an mp3 file. I researched and found out, that for wav-files I need to use the AudioInputStream and calculate an byte array... From mp3-File I am using JavaFX media and media-player. Are the bytes from the Inputstream the same like from the Javafx media.getSource().getBytes(); ? An AudioInputStream cant read mp3...
Or how am I supposed to get the values for an mp3 file for soundwave?
Byte from AudioInputStream:
AudioInputStream audioInputStream;
try {
audioInputStream = AudioSystem.getAudioInputStream(next);
int frameLength = (int) audioInputStream.getFrameLength();
int frameSize = (int) audioInputStream.getFormat().getFrameSize();
byte[] bytes = new byte[frameLength * frameSize];
g2.setColor(Color.MAGENTA);
for(int p = 0; p < bytes.length; p++){
g2.fillRect(20 + (p * 3), 50, 2, bytes[p]);
}
} catch (UnsupportedAudioFileException | IOException e) {
e.printStackTrace();
}
And from JavaFX:
Media media;
MediaPlayer player;
media = new Media("blablafile");
player = new Mediaplayer(media);
byte[] bytes = media.getSource().getBytes();
The JavaFX Media API does not provide much low-level support as of Java 10. It seems to be designed with only the necessary features to play media, not manipulate it significantly.
That being said, you might want to look at AudioSpectrumListener. I can't promise it will give you what you want (I'm not familiar with computer-audio concepts) but it may allow you to create your sound-wave; at least a crude representation.
You use an AudioSpectrumListener with a MediaPlayer using the corresponding property.
If your calculations don't have to be in real time then you can do them ahead of time using:
byte[] bytes = URI.create(media.getSource()).toURL().openStream().readAllBytes();
Note that if the media is remote, however, that you will end up downloading the bytes twice; once to get the bytes for your sound-wave and again when actually playing the media with a MediaPlayer.
Also, you'll want to do the above on a background thread and not the JavaFX Application thread to avoid the possibility of freezing the UI.
I know Qt has a lot of good methods for reading file.But in my work , I have to open a file by using Qt and get a pointer pointing to the start address of the file.So how to do that?
Here is my codes:
char *buffer;
if(file.open(QIODevice::ReadOnly))
{
QByteArray dataArray=file.readAll();
buffer=dataArray.data();
}
char test[1024];
for(int i=0;i<1024;i++)
{
test[i]=*buffer;
buffer++;
}
I use QByteArray QIODevice::readAll()to read all available data from the device, and returns it as a byte array.
Then I use char * QByteArray::data() to set my pointer buffer.
But when I try to read the data by a for loop,my Qt debugger throw me an error:read access violation,so how would this happen?
BTW , the file to be read is very big so I can't use a buffer to read them all once. Instead , I have to do as what I did here. When I read a 7kb-sizes-file ,my codes runs well. When I read a 700kb-sizes-file , here comes the problem.
The reason you are getting access violation is that dataArray is declared inside if code block and goes out of scope. You want to declare the dataArray outside of that block, eg:
QByteArray dataArray;
if(file.open(QIODevice::ReadOnly))
{
dataArray = file.readAll();
}
else
{
// give error
}
char *buffer = dataArray.data();
char test[1024];
for(int i = 0; i < 1024; i++)
{
test[i] = *buffer;
buffer++;
}
There are other potential problems in your code though:
First, what if the size of data read from the file is less than 1024? Then you will be reading past the end of the buffer (more access violations).
Second, what if the file is really big? Reading all that data at once may cause swapping/paging.
I'm a newbie in QT and C++, I'm trying to create a QTcpserver using QThreadpools so it can handle multiple clients. Multiple clients are able to connect without any issues. But I'm trying to send an image from an android phone, with a footer "IMGPNG", indicating the end of image data. Now the issue when the readyRead signal is emitted I'm tring to read all the data available data and then perform some string operation later and reconstruct the image. I'm not sure how to receive the complete image for each client and then process it accordingly.
void VireClients::readyRead()//read ready
{
int nsize = socket->bytesAvailable();//trying to check the available bytes
qDebug()<< "Bytes Available" << nsize;
while(socket->bytesAvailable() < nsize){
QByteArray data = socket->readAll();//how to receive all the data and then process it
}
/*!These lines call the threadpool instance and reimplement run*/
imageAnalysis = new VireImageAnalysis(); //creating a new instance of the QRunnable
imageAnalysis->setAutoDelete(true);
connect(imageAnalysis,SIGNAL(ImageAnalysisResult(int)),this,SLOT(TaskResult(int)),Qt::QueuedConnection);
QThreadPool::globalInstance()->start(imageAnalysis);
}
Now i'm not sure how to get the data completely or save the received data in an image format. i want to know how to completely receive the image data. Please help.
A call to readAll() will not always read the complete image as it obviously cannot know the size of the image. It will only read all currently available bytes which might be less than your whole file, or more if the sender is really fast and you cannot catch up reading. The same way readyRead() only informs you that there are bytes available but not that a whole file has been received. It could be a single byte or hundreds of bytes.
Either you know the size of your image in the first place because it is always fixed or the sender has to tell the receiver the number of bytes he wants to sent.
Then you can either just ignore all readyRead() signals until bytesAvailable() matches your image size and call readAll() to read the whole image at once. Or you read whenever there are available bytes and fill up your buffer until the number of bytes read matches the bytes the receiver told you he will send.
Solved saving image issue by collecting, the string in temp variable and finally, used opencv imwrite to save the image, this solved this issue:
while(iBytesAvailable > 0 )
{
if(socket->isValid())
{
char* pzBuff = new char[iBytesAvailable];
int iReadBytes = socket->read(pzBuff, iBytesAvailable);
if( iReadBytes > 0 )
{
result1 += iReadBytes;
str += std::string(reinterpret_cast<char const *>(pzBuff), iReadBytes);
if(str.size() > 0){
search = str.find("IMGPNG");
if(search == result1-6){
finalID = QString::fromStdString(str);
Singleton_Global *strPtr = Singleton_Global::instance();
strPtr->setResult(finalID);
/*!Process the received image here*/
SaveImage= new VSaveImage();
SaveImage->setAutoDelete(false);
connect(SaveImage,SIGNAL(SaveImageResult(QString)),this,SLOT(TaskResult(QString)),Qt::QueuedConnection);
threadPool->start(SaveImage);
}
}
}
Finally did the image saving on the run method -->SaveImage, #DavidSchwartz you were a great help thanks. Thanks all for your help.
In Dart, I want to read BMP, so could be BIG file.
I do it like this :
var inputStream = imageFile.openInputStream();
inputStream.onData = () {
print(inputStream.available());
inputStream.read(18); // Some headers
int width = _readInt(inputStream.read(4));
int height = _readInt(inputStream.read(4));
// Another stuff ...
}
It works well with little image but when I a read a 3Mo file, the onData is executed many times. Indeed, the onData is trigged by 65536 bytes packets.
What the best practice ?
Should I write a automat with state like HEADER_STATE, COLORS_STATES, ... to set what is my reading state and consider by inputStream.read is a buffer ?
Or I miss a reader class ?
I fear to miss some bytes between 2 packets.
I'm a little disappointed about this, when I do it in java, I just write :
inputStream.read(numberOfBytes);
More easy to use.
Once you have your RandomAccessFile open, you can do something like this:
RandomAccessFile raf; // Initialized elsewhere
int bufferSize = 1024*1024; // 1 MB
int offsetIntoFile = 0;
Uint8List byteBuffer = new Uint8List(bufferSize); // 1 MB
Future<int> bytesReadFuture = raf.readList(byteBuffer, offsetIntoFile, bufferSize);
bytesReadFuture.then((bytesRead) {
Do something with byteBuffer here.
});
There is also a synchronous call readListSync.
John
Using .NET 4.0, IIS 7.5 (Windows Server 2008 R2). I would like to stream out a binary content of about 10 MB. The content is already in a MemoryStream. I wonder if IIS7 automatically chunks the output stream. From the client receiving the stream, is there any difference between these two approaches:
//#1: Output the entire stream in 1 single chunks
Response.OutputStream.Write(memoryStr.ToArray(), 0, (int) memoryStr.Length);
Response.Flush();
//#2: Output by 4K chunks
byte[] buffer = new byte[4096];
int byteReadCount;
while ((byteReadCount = memoryStr.Read(buffer, 0, buffer.Length)) > 0)
{
Response.OutputStream.Write(buffer, 0, byteReadCount);
Response.Flush();
}
Thanks in advance for any help.
I didn't try your 2nd suggestion passing the original data stream. The memory stream was indeed populated from a Response Stream of a Web Request. Here is the code,
HttpWebRequest webreq = (HttpWebRequest) WebRequest.Create(this._targetUri);
using (HttpWebResponse httpResponse = (HttpWebResponse)webreq.GetResponse())
{
using (Stream responseStream = httpResponse.GetResponseStream())
{
byte[] buffer = new byte[4096];
int byteReadCount = 0;
MemoryStream memoryStr = new MemoryStream(4096);
while ((byteReadCount = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
memoryStr.Write(buffer, 0, byteReadCount);
}
// ... etc ... //
}
}
Do you think it can safely pass the responseStream to Response.OutputStream.Write() ? If yes, can you suggest an economic way of doing so? How to send ByteArray + exact stream length to Response.OutputStream.Write()?
The second option is the best one as ToArray will in fact create a copy of the internal array stored in the MemoryStream.
But, you can also preferably use memoryStr.GetBuffer() that will return a reference to this internal array. In this case, you need to use the memoryStr.Length property because the buffer returned by GetBuffer() is in general bigger than the stored actual data (it's allocated chunk by chunk, not byte by byte).
Note that it would be best to pass the original data as a stream directly to the ASP.NET outputstream, instead of using an intermediary MemoryStream. It depends on how you get your binary data in the first place.
Another option, if you serve the exact same content often, is to save this MemoryStream to a physical file (using a FileStream), and use Response.TransmitFile on all subsequent requests. Response.TransmitFile is using low level Windows socket layers and there's nothing faster to send a file.