I've encountered a memory problem using FileReference.save(). My Flash application generates of a lot of data in real-time and needs to save this data to a local file. As I understand, Flash 10 (as opposed to AIR) does not support streaming to a file. But, what's even worse is that FileReference.save() duplicates all the data before saving it. I was looking for a workaround to this doubled memory usage and thought about the following approach:
What if I pass a custom subclass of ByteArray as an argument to FileReference.save(), where this ByteArray subclass would override all read*() methods. The overridden read*() methods would wait for a piece of data to be generated by my application, return this piece of data and immediately remove it from the memory. I know how much data will be generated, so I could also override length/bytesAvailable methods.
Would it be possible? Could you give me some hint how to do it? I've created a subclass of ByteArray, registered an alias for it, passed an instance of this subclass to FileReference.save(), but somehow FileReference.save() seems to treat it just as it was a ByteArray instance and doesn't call any of my overridden methods...
Thanks a lot for any help!
It's not something I've tried before, but can you try sending the data out to a php application that would handle saving the ByteArray to the server, much like saving an image to the server, so then you'd use URLLoader.data instead, using something like this:
http://www.zedia.net/2008/sending-bytearray-and-variables-to-server-side-script-at-the-same-time/
It's an interesting idea. Perhaps to start you should just add traces in your extended ByteArray to see how the FileReference#save() functions internally.
If it has some kind of
while( originalByteArray.bytesAvailable )
writeToSaveBuffer( originalByteArray.readByte() );
functionality the overrides could just truncate the original buffer on every read like you say, something like:
override function readByte() : uint {
var b : uint = super.readByte();
// Truncate the bytes (assuming bytesAvailable = length - removedBytes)
length = length - bytesAvailable;
return b;
}
On the other hand, if this now works I guess the original byte array would not be available afterwards in the application anymore.
(i havn't tested this myself, truncating might require more work than the example)
Related
I am using a an object of QScintilla and I am reading the file in QScintilla Object incrementally.
Header myEditor.h
class myScintilla: public QScintilla {
public readFile();
};
#include "myEditor.h"
void myEditor::readFile() {
if (FILE* fp = fopen(ofilename.toLatin1(), "r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
int bufferSize =(1024* 1024)/2;
do {
QString s = ts.read(bufferSize);
append(s);
} while(!ts.atEnd());
}
Even after this change there will be still performance issue while reading large files. It took around
1) 25 seconds to read a file of size 1.5 GB. (Machine cores 4 , 16 GB RAM)
2 10 seconds of file of size 512MB (on same machine)
Is there any way we can load the file in QScintilla object incrementally based on movement of scrollbar?
I found your question interesting so did a little bit of Googling on your behalf. It seems to me that while Scintilla exposes this functionality via the Loader interface, in fact the QScintilla class does not. To make this work, it seems that what you would have to do is use the QScintillaBase class to send the SCI_CREATELOADER message to the Scintilla control.
Edit: Also, you do not want to use append in a loop. That will cause all sorts of terrible things to happen. It will likely force rendering, some sort of indexing, etc. Before using my suggestion above, I would suggest that you instead build up a gigantic QString in memory and then set that at the end. Better to pre-allocate. That might be a little faster.
FINAL ANSWER
Edit #2: OK, it was bothering me that such an industrial strength editor component like Scintilla did not support this natively but it seems that the right way to do this is by using a combination of features:
You start with a document allocated using SCI_ALLOCATE where the number of bytes is the size of your file
You listen for the SCN_UPDATEUI event
Then, based on where the user is scrolling to, you load that data
It should be straightforward to map the above to QScintillaBase as a test.
I need to continue writing to the same file using a file output stream, However I'm implementing the program using a javafx GUI from different windows. but:
Since you can not access a global variable without intializing the stream as final I have to make it final however,
Since I have to wrap the the Stream in a try catch, the variable is not recognised as initialised.
I was previously intialising in method like this
eW = outputFactory .createXMLEventWriter(new FileOutputStream("File.txt"));
However, that obivously overwrites the same file when you write the same statement again.
So essentially my question is, how can you set a final variable that needs to be surrounded in a try catch?
final File file = new File("File.txt");
final FileOutputStream fileOS = new FileOutputStream(file);
You don't need to initialize a variable as final to access it from different threads or objects. There are other ways to do it.
You can use a helper class, or even a helper thread. The second option is better in (the most common) case where you have more than 1 thread in your project. You can then use a normal try-catch block that has a (almost infinite) while loop inside, that checks a write queue and writes something when needed.
If you only use 1 thread and only append to a file, you may want to do just that - open the file to append, rather than overwrite. There's a boolean flag in the constructor.
You can take a look at the FileOutputStream(File file, boolean append) constructor, which would allow you to append to the end of the file rather than overwrite each time.
I do not know how you implemented it, but if you are writing to the same file from multiple windows, it might be easier to have a helper class which exclusively handles writing to file. This would allow you to centralize your File output code to one place making it easier to debug and maintain.
everybody.
I have a doubt in this portion of my system: I want to encrypt some messages from users to the database and decrypt them when showing back to the users (that's just for privacy reasons). Since I couldn't find any native encrypt/decrypt library or even a better solution, then I am using the "crypto-js" (https://code.google.com/p/crypto-js/), and it's working well so far. The problem is: when the user write a message I encrypt it and save it at the database. But when I retrieve that message from the database using the "find" method in a Template Helper (using the reactive computation idea to approach the desired "Live HTML"), what I get is just a cursor that will be used to render the message in my HTML. As you can see, the message is shown without be decrypted.
I was thinking in "fetch" the data from the cursor, run the decrypt function in a "for loop" over all messages, but that's too inefficient.
I was wondering if someone know how may I manipulate the data from the cursor before it be render, then I'll be able to decrypt each message on the fly.
Really thanks for your attention and sorry any bother.
You could use a transform function. The transform is passed as the cursor is used so it only runs on each document as it is used:
YourCollection.find({}, {transform:function(doc) {
var encrypted = doc.field1;
doc.field1 = decrypt(encrypted)
return doc;
});
So now each field1 would be decrypted (on the web browser). Just before it was used. If you use .fetch() you would get all the decrypted data too.
You didn't posted the related code here, but I assume you have done something like this:
Template.yourTemplate.yourHelper = function(){
return yourCollection.find({});
}
The thing is your are returning the data to the helper, in form of cursor, which is perfectly acceptable in general. But as you are storing encrypted messages in collection, each message is rendered as it is without performing decryption.
So, try to fetch an array instead of cursor using find().fetch() which gives the array equivalent of your cursor. Something like :
Template.yourTemplate.yourHelper = function(){
var msg_arr = yourCollection.find({}).fetch(); // gives array instead of cursor.
msg_arr.forEach( function( msg ){
// traverse through each element of array and
// perform decryption.
});
}
I just had a talk at the #meteor IRC channel and some ideas came out. And I thought that this one is the best solution for my case:
{{#each messages}}
{{decrypt}}
{{/each}}
As helpers get the data from the context, the helper "decrypt" got the actual "message" in the loop as a "this" object. Then, I did the decryption and returned the message in plain text.
It works pretty well, further it is called reactively and make use of the cursor (that is update dynamically if data changes during the exhibition).
Thanks everybody whom helped me.
in my project, I let users pick pictures using the FileReference class. I then load these pictures into their .data properties, using the load() function. After this I perform some local manipulation and send them to the server.
What I would like to do, is to be able to iterate over the picked FileReferences again, load them into .data properties, perform different manipulation and send them to the server once again. I know that I should be able to do this from user-invoked event, that is not an issue here.
Problem is, once the FileReference is loaded for the first time, I can not unload it in any way, and I can not keep the data for all the pictures in the memory because these are huge.
So I guess there is only one thing I can do, which is performing a DeepCopy on the FileReference... Then I could load the first version, scrap it and use the copy for the second 'run'.
I tried to use ObjectUtil.copy, but when I access e.g. .name property of the copy, it fails with:
Error #2037: Functions called in incorrect sequence, or earlier call was unsuccessful.
at flash.net::FileReference/get name()
the relevant snippet:
registerClassAlias("FileReference",FileReference);
masterFileList.addItem(FileReference(ObjectUtil.copy(fr_load.fileList[i])));
trace(masterFileList[i].name)
Is it true that there are some protected properties of FileReference class that prevent it from being copied? If it is so, can I sidestep this somehow? Or is there any other solution to my overall problem?
I appreciate any hints/ideas!
I was trying to do almost exactly what you were doing, and I almost gave up after reading some of the answers, but I think I found a way to do it. I've found that if you have a FileReference object and call load() multiple times, it will work, but the main problem is that you're keeping the high-res bytes in memory after the first load. As you've mentioned, for people who don't know image processing, this is a big no-no.
The way to get around this is that after your first load(), you need to call the cancel() method on FileReference. From my testing so far, it looks like that will clear out the bytes in the FileReference, and load() will still work if you call it a second time later. Just a word of caution, this isn't explicitly-defined behavior in the API, so it is definitely subject to change, but it may help get you where you need to go in the mean time.
Hope that helps.
you cant use a ObjectUtil.copy. This method is designed for copying only data objects (VO classes).
you should create a new FileReference and copy the porperties, one by one. Create a function to do this..
Would copying it to a temporary file and then uploading the temporary file work? For example
var fileRef:FileReference = new FileReference();
fileRef.browse();
......................
var tmpFile:File = File.createTempFile();
try {
var tmpFileStream:FileStream = new FileStream();
tmpFileStream.open(tmpFile, FileMode.WRITE);
trace("Opened file: " + tmpFile.nativePath);
tmpFileStream.writeBytes(fileRef.data);
trace("copied file");
} catch ( error:Error ) {
trace("Unable to open file " + tmpFile.nativePath + "\n");
throw error;
}
I'm thinking that the operation is completely disallowed, for good reasons. If you can duplicate a new FileReference through ActionScript code, then you'd also be able to manufacture a FileReference object through ActionScript code. Of course, that'd be a pretty bad security hole if you could force the upload of an arbitrary file.
Keeping a copy of the data in memory really isn't that bad of a solution. After all, it's temporary. The typical client computer should be able to manage a few hundred extra MB of data with no problem. It's certainly a better option than having their browser do two separate uploads, which is what your attempted solution would end up doing.
A completely different potential solution to this problem is to avoid image manipulation by Flex altogether. Flex could post the uploaded file directly to the server, and the server could do the image manipulation itself. Of course, if the manipulation is driven through user interactions, then that wouldn't work at all.
I need to read xml tags and its datas from one file and then write it to another xml..how to do it?? please let me know immediately...?
See http://livedocs.adobe.com/flex/2/langref/XML.html . I find it hard to believe you googled this before asking.
You can use the FileReference.save() method to save XML data to a local file. It will prompt the user for a location to save the file first and then save the data.
Here's an example:
var xml:XML = <root><someXmlTag/></root>;
var fileReference:FileReference = new FileReference()
fileReference.save(xml, "myfile.xml");
As far as I knew, Flex wasn't able to write to files!
I use a HTTPService to load the XML file and a result even handler to access it.
<mx:HTTPService id="service" url="myXml.xml" result="ServiceResult (event)"/>
Do not specify a result format in the HTTPService tag.
This is the code for the result event handler.
private function ServiceResult (e : ResultEvent) : void {
e.result.XmlTag.AnotherXmlTag;
}
You can also use service.lastResult to access the last result returned by the HTTPService. The result is fully compatible with the dataProvider property, especially in arrays and chart series.
var series : LineSeries = new LineSeries ();
series.dataProvider = e.result.XmlTag.AnotherXmlTag;
This will take the value in all AnotherXmlTag tags within XmlTag. For series, though, you should also specify either a yField or and xField, but it digress :-)
If it doesn't work, you can also cast it using the as keyword, example:
series.dataProvider = e.result.XmlTag as ArrayCollection;
I haven't actually tried casting it in this scenario, but the bottom line is that XML tags are vary compatible with arrays and ArrayCollections.
In your case, you would just use e.result to get the complete XML file, assign it to a variable, and write it using Ben's method. You can also create an array from individual values using the way I explained above, and manually insert tags and such if you need. The advantage of this is that you have all the values ready in an array would you need them later on. If you loop through the indices, this won't require a lot of work, and it would be the way I'd do it.
Hope this helps!