Easy question. I need to read a CSV file in .NET, and for that I'm using the Lumenworks CSV library.
The problem is that it seems this solution reads the entire CSV content into memory. I was wondering if there's another option that would let me run through the CSV content one element at a time, and therefore, consume less memory.
Something like XmlDocument vs. XmlReader.
Thanks
You can use StreamReader Class to load the file line by line to do some operations like searching, matching, etc., with the method StreamReader.ReadLine Method. One sample is contained in it to show how. This really costs little time.
Store the position or line number after once of operation, then in the next operation use the Stream.Seek Method to start load from the stored position.
Related
I need very simple text file logging. I'll only append lines to it. never change existing ones nor delete them. If it would be XML file it would be easier to bind to grids to view them. but question remains for both text files and xml files as they are in file system.
in web server there will be file locking while appending log entries. and maybe also while reading them. So this method has to be thread safe. At the same moment multiple instances can write date to file.
I know there are some third party tools like serilog etc but I want to know:
how can I append (not change) lines to text file (or xml file) without concerning about file locks ?
if I read xml file to dataset, add a new row to it and save it as xml I would use other entries made by other instances.
if I open a text file with streamwriter and append a line to it, other instances would get lock error.
I get the list of logs from admin panel again, file will be locked and instances wouldn't append logs.
any ideas ?
After long reserch hours and experiments I found out that using Nlog is the best option for me. most important thing is people who use it are very happy. I created small example page that writes a log everytime it called and tested it. I have a multithreaded application that calls this sample page again and again. If was fast enough so I could not see the counting numbers of threads. no problem raised so far.
So, I'll stick to Nlog.
best.
I'm using PHPExcel to read through Excel spreadsheets of various sizes and then import the cell data into a database. Reading through the spreadsheet itself works great and is very quick, but I've noticed that the time to actually load/open the file for PHPExcel to use can take up to 10-20 seconds (the larger the file, the longer it takes--especially if the spreadsheet is >1MB in size).
This is the code I'm using to load the file before iterating through it:
$filetype = PHPExcel_IOFactory::identify($file);
$objReader = PHPExcel_IOFactory::createReader($filetype);
$objReader->setReadDataOnly(true);
$objPHPExcel = $objReader->load($file);
What can I do to get the file to load faster? It's frustrating that the greatest latency in importing the data is just in opening up the file initially.
Thank you!
I've seen this same behavior with Ruby and an Excel library: a non-trivial amount of time to open a large file, where large is > 500KB.
I think the cause is two things:
1) an xlsx file is zip compressed, so it must first be un-compressed
2) an xlsx file is a series of XML files, which all must be parsed.
#1 can be a small hit, but most likely it pales in comparison to #2. I believe its the XML parsing that is the real culprit. In addition, the XML parser is a DOM-based parser, so the whole XML DOM must be parsed and loaded into memory.
I don't think there is really anything you can do to speed this up. A large xlsx file contains a lot of XML which must be parsed and loaded into memory.
Actually, there is something you can do. The problem with most of the XML parsers is that they first load the entire document in memory. For big documents, this takes a considerable amount of time.
A way to avoid this is to use parsers that allow streaming. So instead of loading all the XML files content in memory, you just load the part you need. That way, you can pretty much have only one row at a time in memory. This is super fast AND memory efficient.
If you are curious, you can find an example of a library using this technique here: https://github.com/box/spout
I have binary asd i want to save as a myvideo.mp4, So my question is that if you have binary and you want to store your file,
So how it would accomplished any one idea i didn't work yet with binary and streaming i am new with these all things so if any one know please share a piece of example which solve my problem.
like response.outStream in this i am getting binary so can i save as a myvideo.mp4 what would be that process actually this binary is generating for mp4 conversion so any idea.
thanks.
You will need to create a FileStream to write to and then you will read the source stream in a loop, reading a block and then writing that block to the FileStream. It should be fairly easy to find an example of reading a stream in a loop using its Read method.
I will be creating a structure more or less of the form:
type FileState struct {
LastModified int64
Hash string
Path string
}
I want to write these values to a file and read them in on subsequent calls. My initial plan is to read them into a map and lookup values (Hash and LastModified) using the key (Path). Is there a slick way of doing this in Go?
If not, what file format can you recommend? I have read about and experimented with with some key/value file stores in previous projects, but not using Go. Right now, my requirements are probably fairly simple so a big database server system would be overkill. I just want something I can write to and read from quickly, easily, and portably (Windows, Mac, Linux). Because I have to deploy on multiple platforms I am trying to keep my non-go dependencies to a minimum.
I've considered XML, CSV, JSON. I've briefly looked at the gob package in Go and noticed a BSON package on the Go package dashboard, but I'm not sure if those apply.
My primary goal here is to get up and running quickly, which means the least amount of code I need to write along with ease of deployment.
As long as your entiere data fits in memory, you should't have a problem. Using an in-memory map and writing snapshots to disk regularly (e.g. by using the gob package) is a good idea. The Practical Go Programming talk by Andrew Gerrand uses this technique.
If you need to access those files with different programs, using a popular encoding like json or csv is probably a good idea. If you just have to access those file from within Go, I would use the excellent gob package, which has a lot of nice features.
As soon as your data becomes bigger, it's not a good idea to always write the whole database to disk on every change. Also, your data might not fit into the RAM anymore. In that case, you might want to take a look at the leveldb key-value database package by Nigel Tao, another Go developer. It's currently under active development (but not yet usable), but it will also offer some advanced features like transactions and automatic compression. Also, the read/write throughput should be quite good because of the leveldb design.
There's an ordered, key-value persistence library for the go that I wrote called gkvlite -
https://github.com/steveyen/gkvlite
JSON is very simple but makes bigger files because of the repeated variable names. XML has no advantage. You should go with CSV, which is really simple too. Your program will make less than one page.
But it depends, in fact, upon your modifications. If you make a lot of modifications and must have them stored synchronously on disk, you may need something a little more complex that a single file. If your map is mainly read-only or if you can afford to dump it on file rarely (not every second) a single csv file along an in-memory map will keep things simple and efficient.
BTW, use the csv package of go to do this.
I'm trying to export an AdvancedDataGrid to CSV. This is easy enough for non-hierarchical data, but when using a HierarchicalCollectionView to show treed data it gets trickier.
Any suggestions on how to access each of the cells just as they appear on screen when all of the nodes are expanded?
If you've expanded all the nodes like you mentioned (you can use the AdvancedDataGrid's expandAll() function for this), you can then run the AdvancedDataGrid through the following CSV export utility class to access each of the cells as they appear on the screen:
https://onyxmueller.net/2011/08/20/advanceddatagrid-csv-export-utility-class/
However, I've found when dealing with a HierarchicalCollectionView as the data provider, that it is better to write some custom logic to "flatten" the data for CSV export.
Hierarchical data doesn't map well to CSV which is essentially flat. You are essentially trying to write nested objects into spreadsheet.
Accessing the data isn't that hard, you can just recursively work through getChildren() in the collection.
The hard bit is writing it into the CSV file in a way that can be retrieved later. The only really good ways of doing this is by ignoring the fact that you are writing to CSV. As soon as you get to the children field of the root object you are going to end up writing some horrible array parsing mechanism.
My solution? Write it out to JSON, and stick it in a single cell of the CSV. You'll save yourself a ridiculous amount of pain in the long run.