I need to parse a large trace file (up to 200-300 MB) in a Flex application. I started using JSON instead of XML hoping to avoid these problems, but it did not help much. When the file is bigger than 50MB, JSON decoder can't handle it (I am using the as3corelib).
I have doing some research and I found some options:
Try to split the file: I would really like to avoid this; I don't want to change the current format of the trace files and, in addition, it would be very uncomfortable to handle.
Use a database: I was thinking of writing the trace into a SQLite database and then reading from there, but that would force me to modify the program that creates the trace file.
From your experience, what do you think of these options? Are there better options?
The program that writes the trace file is in C++.
Using AMF will give you much smaller data sizes for transfer because it is a binary, not text format. That is the best option. But, you'll need some middleware to translate the C++ program's output into AMF data.
Check out James Ward's census application for more information about benchmarks when sharing data:
http://www.jamesward.com/census/
http://www.jamesward.com/2009/06/17/blazing-fast-data-transfer-in-flex/
Maybe you could parse the file into chunks, without splitting the file itself. That supposes some work on the as3 core lib Json parser, but it should be doable, I think.
I found this library which is a lot faster than the official one: https://github.com/mherkender/actionjson
I am using it now and works perfectly. It also has asynchronous decoder and encoder
Related
I have binary asd i want to save as a myvideo.mp4, So my question is that if you have binary and you want to store your file,
So how it would accomplished any one idea i didn't work yet with binary and streaming i am new with these all things so if any one know please share a piece of example which solve my problem.
like response.outStream in this i am getting binary so can i save as a myvideo.mp4 what would be that process actually this binary is generating for mp4 conversion so any idea.
thanks.
You will need to create a FileStream to write to and then you will read the source stream in a loop, reading a block and then writing that block to the FileStream. It should be fairly easy to find an example of reading a stream in a loop using its Read method.
I will be creating a structure more or less of the form:
type FileState struct {
LastModified int64
Hash string
Path string
}
I want to write these values to a file and read them in on subsequent calls. My initial plan is to read them into a map and lookup values (Hash and LastModified) using the key (Path). Is there a slick way of doing this in Go?
If not, what file format can you recommend? I have read about and experimented with with some key/value file stores in previous projects, but not using Go. Right now, my requirements are probably fairly simple so a big database server system would be overkill. I just want something I can write to and read from quickly, easily, and portably (Windows, Mac, Linux). Because I have to deploy on multiple platforms I am trying to keep my non-go dependencies to a minimum.
I've considered XML, CSV, JSON. I've briefly looked at the gob package in Go and noticed a BSON package on the Go package dashboard, but I'm not sure if those apply.
My primary goal here is to get up and running quickly, which means the least amount of code I need to write along with ease of deployment.
As long as your entiere data fits in memory, you should't have a problem. Using an in-memory map and writing snapshots to disk regularly (e.g. by using the gob package) is a good idea. The Practical Go Programming talk by Andrew Gerrand uses this technique.
If you need to access those files with different programs, using a popular encoding like json or csv is probably a good idea. If you just have to access those file from within Go, I would use the excellent gob package, which has a lot of nice features.
As soon as your data becomes bigger, it's not a good idea to always write the whole database to disk on every change. Also, your data might not fit into the RAM anymore. In that case, you might want to take a look at the leveldb key-value database package by Nigel Tao, another Go developer. It's currently under active development (but not yet usable), but it will also offer some advanced features like transactions and automatic compression. Also, the read/write throughput should be quite good because of the leveldb design.
There's an ordered, key-value persistence library for the go that I wrote called gkvlite -
https://github.com/steveyen/gkvlite
JSON is very simple but makes bigger files because of the repeated variable names. XML has no advantage. You should go with CSV, which is really simple too. Your program will make less than one page.
But it depends, in fact, upon your modifications. If you make a lot of modifications and must have them stored synchronously on disk, you may need something a little more complex that a single file. If your map is mainly read-only or if you can afford to dump it on file rarely (not every second) a single csv file along an in-memory map will keep things simple and efficient.
BTW, use the csv package of go to do this.
I have an objective to transfer some files which can be pdf, jpg and xml from one location to another. I am creating a handler based API for that. What could be the best approach for doing it? Should i return a byte array so that the client can pick the byte array and convert it back to file? Also the max file size I need to handle is 18 MB, so i have to make sure that the sizes are properly handled. Should I do an asynchronous transfer for that? If there is some sample code available it would be great.
i got it done using this: http://blog.net-tutorials.com/2009/03/16/how-to-download-a-file-from-the-internet-with-c/ but it has another issue which i mentioned in another question here:
WebClient.DownloadFileAsync downloading the file on server
I've got a bunch of audio files (let's say ogg or mp3), with metadata.
I wish to read their metadata into R so to create a data.frame with:
file name
file location
file artist
file album
etc
Any way you know of for doing that ?
You take an existing mp3 or ogg client, look at what library it uses and then write a binding for said library to R, using the existing client as guide for that side -- and something like Rcpp as a guide on the other side to show you how to connect C/C++ libraries to R.
No magic bullet.
A cheaper and less reliable way is to use a cmdline tool that does what you want and write little helper functions that use system() to run that tool over the file, re-reading the output in R. Not pretty, not reliable, but possibly less challenging.
Possible, yes, easy, no.
You "could" use a combination of readChar and/or readBin on the file and parse out the contents. This would be highly dependent, though, on parsing the frame tags from the raw bytes of the ID3v2 tag (and mind you it would change if it was a version 1 tag). If would certainly be a lot of work to implement a straight R solution. Take this Python code for example, it's very clean straight python code but a lot of branching and parsing.
You can use exiftool with system command available in R. Optionally, you can create regexp to handle the fields you need... If I were you, I'd stick with Dirk's advice (as usual) =)!
Out here in 2021, I wanted to do this so I did the following...
Create a new playlist while in 'songs' view.
Select all songs and drag to the new playlist. Highlight that playlist
File> Library>Export Playlist. My default file was to save as .txt, if not, designate.
Open Excel to save as csv or read.delim() in r as the txt file is tab-separated
import to R
I'm trying to use Adobe's OggVorbis library. But I can't seem to get the Sound object to loop.
I even tried looping the _sound object inside the AudioDecoder.as in the "com.automatastudios.audio.audiodecoder" package.
Do you really have to reload the file and stream it over and over?
If you're streaming, then yes you'll have to jump back the beginning of the stream. A stream, by definition, is a constant link to the server and does minimal loading of files locally.
But, if you're not really streaming, you should have no problem loading up a file and caching it locally then playing it over and over.
Since you mention Alchemy, there may be other unknown issues if you're trying to use a converted C library, as opposed to native ActionScript.
[Note; I didn't know the OggVorbis library for Flex before now].