I need to export huge amount of data from ado.net datatable(which i get by db query) to excel.
I tried the following way : 1. Create excel object with workbook/worksheet # server side...and use memory stream to write whole document to client side.
But this gave me "out of Memory exception". bcoz my memory stream was was so huge.
So I replaced this with a new way - as follows :
Writing each row from datatable as a coma seperated string to client side. So, as and when we get each row ...we can write to client side ..no memory is used.
But by this way we can write to csv file...not to excel...
Does anybody know how to handle this situation.
Can I use silverlight to get data row by row from server, pass it to client side...build excel at client side.?
You should create it on the server, then copy it to the client in chunks.
For an example, see this answer.
If this is for XL 2007, then the workbook is basically in an OPEN XML file format.
If you can format the data in your datatable to conform to the OPEN XML, you can save the file and then just download the entire file.
Read up on OPEN XML at http://msdn.microsoft.com/en-us/library/aa338205.aspx
Related
I have read the FileWriter Filter :
" The File Writer filter can be used to write files to disc regardless of format. The filter simply writes to disc whatever it receives on its input pin, so it must be connected upstream to a multiplexer that can format the file correctly. You can create a new output file with the File Writer or specify an existing file; if the file already exists, it will be completely overwritten with the new data. "
So my question is :
I am using the FileWriter filter for writing my audio stream into the disc. Before writing the file in the disc i want to access that file , so can it be possible or should i make my own custom filter.
File writer filter does not not provide you with options to change file sharing mode while the file is being written to. Additionally, in most cases your accessing the file before it is finalized makes no sense: the files are rarely written incrementally, file finalization changes data in the middle of the file and your accessing data before the file is closed might get you bad/incomplete stream.
Roman R is right. Writers are for writing. If you need transform data - write your own Transform filter.
You can ask me directly here.
MS VS 2008, ASp.Net 3.5.
On the client side :
client selects start and end dates, report format as Excel, clicks "run report" button
On that click redirected to reportToExcel.aspx, in reportToExcel.aspx.vb in Page_Load event stored procedure is executed to retrieve report data :
oSQLDataReader = oSqlCommand.ExecuteReader()
Then:
Response.ContentType = "application/ms-excel"
Response.AddHeader("Content-Disposition", "attachment; filename=" + MyBase.UserSession.ReportName + ".xls")
Then Response.Write is used to write retrieved report data into Response object in XML format, like
Response.Write("<td>" & FormatColumnValue(oSQLDataReader.GetValue(I), arrColHeader(I + 1).ColumnFormat) & "</td>"), etc. Last callis Response.End().
I know Response.End should not be used, I plan to substitute is with
context.Response.Flush()
context.ApplicationInstance.CompleteRequest()
but I doubt it will improve response time.
Problem: on the client side take 6 mins to receive 32.5 MB of data. This is too long.
How to reduce this time ?
As I understood so far: chuncking is not possible for Excel report and anyway client wants to receive Excel report as one and whole.
In order to use Response.TransferFile : Excel file has to be created first, then zipped to reduce amount of data to download, then downloaded. For this to work Excel should be installed on the server, which is not acceptable in our case.
Deliver data as csv to client is not acceptable: client will have to import it to Excel, which they would not like to do.
Stored procedure executed from SQL management studio shows inconsistent run times: from 12 secs to 4 mins.
So, are there any other ways to reduce report 'delivery' time to the client ?
Thank you for all replies
You do need to get the stored procedure run time down. But that's a whole question and answer in itself.
Your method of writing out the HTML is slower than it needs to be. Essentially, you are doing repeated string concatenations, which are slow. Consider using a StringBuilder to construct the entire document before writing it to the Response stream.
Another option (and perhaps a better one) would be to try something like the free Excel Xml Writer library: http://www.carlosag.net/tools/excelxmlwriter/. I haven't used it, but I've heard good things about it. This would (I believe) let you write your Excel file on the server without needing Excel itself installed.
I am working on Windows Application development using c#. I want to read a csv file from a directory and imported into sql server database table. I am successfully read and import the csv file data into database table if the file content is uniform. But I am unable to insert the file data with invariant form ex.Actually my csv file delimiter is tab('\t') and after getting individual fields I have a field that contains data like dcc
Name
----
xxx
xxx yyy
xx yy zz
and i rerieved data like xxx,yyy and xx,yy,zz so the insertion becomes problem.
How could i insert the data uniformly into a database table.
It's pretty easy.
Just read file line-by-line. Example on MSDN here:
How to: Read Text from a File
For each line use String.Split Method with your tab as delimiter. Method documentation and sample are here:
String.Split Method (Char[], StringSplitOptions)
Then working insert your data.
If a CSV (or TSV) value contains a delimiter inside of it, then it should be surrounded by quotes. See the spec for more details: https://www.rfc-editor.org/rfc/rfc4180#page-3
So your input file is incorrectly formatted. If you can convince the input provider to fix this issue, that will be the best way to fix the problem. If not, other solutions may include:
visually inspecting and editing the file to fix errors, or
writing your parser program to have enough knowledge of your data expectations that it can correctly "guess" where the real delimiters are.
If I'm understanding you correctly, the problem is that your code is splitting on spaces instead of on tabs. Given you have read in the lines from the file, all you need to do is:
string[] fileLines;//from the file
foreach(string line in fileLines)
{
string[] lineParts=line.Split(new char[]{'\t'});
}
and then do whatever you want with each lineParts. The \t is the tab character.
If you're also asking about writing the lines to a database file...you can just read in tab-delimited files with the Import Export Wizard (assuming you're using Sql Server Mgmt Studio, but I'm sure there are comparable ways to import using other db management software).
I need to export huge amount of data from ado.net datatable(which i get by db query) to excel.
I tried the following way :
1. Create excel object with workbook/worksheet # server side...and use memory stream to write whole document to client side.
But this gave me "out of Memory exception". bcoz my memory stream was was so huge.
So I replaced this with a new way - as follows :
Writing each row from datatable as a coma seperated string to client side.
So, as and when we get each row ...we can write to client side ..no memory is used.
But by this way we can write to csv file...not to excel...
Does anybody know how to handle this situation.
Can I use silverlight to get data row by row from server, pass it to client side...build excel at client side.?
try spreadsheetgear
OR
smartxls
I'd keep the csv approach but write to a file and not the memory strea. After you've created the file, I'd use TransmitFile to get it to the browser. You can see more about using TransmitFile here.
I'm tasked with writing a solution for fixing a poorly performing legacy excel file generator.
The files I need to generate could get very large. Maybe up to a million rows with 40-50 columns. I guess I will stream directly to the user if possible, but I might just have to save the file to disk first and then create a link for the user.
I aim to make a performance test, testing if I can generate an xslx file with 1.500.000 rows and 50 columns, each cell containing a random 10 letter string... Will excel even handle files this big?
Note: In reality most files generated will never be larger than
300.000 rows and the absolute maximum is about 950.000 rows, but I like to
play it safe when stress testing thus
the 1.5M rows.
Do you have any suggestions as to how I should go about solving this task. Are there any components I should be aware of? Limitations in excel?
PS: I would appreciate if I didn't have to install Excel on the server.
There is a limit to the amount of rows the you can have in a spreadsheet (1M for Office 2007). I would generate a CSV file instead, which is really just a formatted text file that can be opened in Excel.
Excel 2007 supports a maximum worksheet size of 1,048,576 rows by 16,384 columns so your test with 1.5 million rows may not be feasible. Source
Edit: Excel 2003 supports even fewer rows: 65,536 rows by 256 columns. Source
If you are able to require that your users have the ability to open documents in the Excel 2007 (xlsx) format then that may be your best bet as it is just a XML document and can be generated without any requirement for Excel on the server.
If you need to support "all" version of excel/other office suite programs you should probably use CSV or another character delimited format.
The Open Document Format may also be of interest but excel users would need the ODF add-in to use the documents.
Edit 2: If you are looking at using CSV you may want to look at the FileHelpers library.
While I can't answer the maximum amount of data that Excel can handle, if you use the new .xlsx format, you are using MS's OpenXML format. The .xlsx file is actually a zip compressed file with all of the document data stored internally. The XML can be written just as any other XML, but you'll have to review the standards. There are a few commercial component implementations out there for doing this. You don't need Excel to write the format.
Here are some helpful links:
Office OpenXML - Wikipedia
Office Open XML C# Library - This looks like an open source library for reading/writing OpenXML
Read and write Open XML files - CodeProject - Another implementation of a R/W library
GemBox.Spreadsheet - A commercial .NET component for reading/writing office spreadsheets. Has a free version with limitations on number of rows you can read and write if you want to try it out.
NPOI Library - an implementation of Java's POI library for reading and writing office docs.
Simple OOXML - "A set of helper classes to make the creation of Open Office XML documents easier. Uses the Open Office SDK v 2.0. Modify or create any .docx or .xlsx document without Microsoft Word or Microsoft Excel."
Make sure that your tests are representative of the actual data. Excel handles simple numeric cells much more efficiently than simple text cells - especially when all of the text cells are unique. So, if your data is really going to be made up of 10 character unique strings, by all means use that as your test case. If it's actually going to be mostly numbers, make sure your tests reflect that fact.
For example. I built a simple test using SpreadsheetGear for .NET to generate a 300,000 row by 50 column Open XML (.xlsx) workbook. To create and save to disk with unique numbers took 13.62 seconds on my almost two year old overclocked QX6850 CPU, while creating and saving a 300,000 row by 50 column .xlsx workbook with 10 character unique strings took 78 seconds - 6 times longer for text than for numbers. I will paste the code below and you can run it with the free SpreadsheetGear trial which you can download here.
It is important to note that Open XML (.xlsx) is compressed, so if your data has much redundancy, you are likely to get smaller files with .xlsx than with .csv. This could have a big impact on performance if you are generating workbooks on a web server for consumption over a network.
SpreadsheetGear with the IWorkbook.SaveToStream method, and most other 3rd party Excel compatible libraries will let you save directly to the response stream in an ASP.NET application, so you will be able to avoid saving to disk on the server.
Disclaimer: I own SpreadsheetGear LLC
Here is the test code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using SpreadsheetGear;
namespace ConsoleApplication11
{
class Program
{
static void Main(string[] args)
{
var timer = System.Diagnostics.Stopwatch.StartNew();
int rows = 300000;
int sheets = 1;
var workbook = Factory.GetWorkbook();
var sb = new System.Text.StringBuilder();
int counter = 0;
bool numeric = true;
for (int sheet = 0; sheet < sheets; sheet++)
{
// Use the SpreadsheetGear Advanced API which is faster than the IRange API.
var worksheet = (sheet == 0) ? workbook.Worksheets[0] : workbook.Worksheets.Add();
var values = (SpreadsheetGear.Advanced.Cells.IValues)worksheet;
for (int row = 0; row < rows; row++)
{
for (int col = 0; col < 50; col++)
{
if (numeric)
values.SetNumber(row, col, ++counter);
else
{
sb.Length = 0;
// Make a 10 character unique string.
sb.Append(++counter);
System.Diagnostics.Debug.Assert(sb.Length <= 10);
// Make it 10 characters long.
while (sb.Length < 10)
sb.Append((char)('A' + (char)sb.Length));
values.SetText(row, col, sb);
}
}
}
}
Console.WriteLine("Created {0} cells in {1} seconds.", counter, timer.Elapsed.TotalSeconds);
workbook.SaveAs(#"C:\tmp\BigWorkbook.xlsx", FileFormat.OpenXMLWorkbook);
Console.WriteLine("Created and saved {0} cells in {1} seconds.", counter, timer.Elapsed.TotalSeconds);
}
}
}
You might want to check out the NPOI library for reading and writing excel files at http://npoi.codeplex.com/. As for storing on the server that is an option but remember you will have to clean up the files after they have been download.
Take a look at the Simple OOXML project on Codeplex.
It might be what you are looking for.
PS. Excel is mainly spreadsheet software, no a database substitute. Are you sure you want to dump a million rows to the end user?
Excel can not handle millions of rows, try creating a CSV output file in stead, this can be read by Excel.
And it is not recommended to add a huge amount of data in an excel on user request
He will need to wait to long before downloading the file.
Assuming you can avoid exceeding the new row limitations in Excel 2007 (by splitting to other worksheets or files), Excel 2007's xlsx format should work just fine.
Since XLSX is a zip format, rather than creating in memory or writing to a disk, you should consider writing directly to an in-memory zip stream. The compression will keep memory usage low, and not writing to the file system will help with performance.
Another potential solution, depending on your circumstances: create a blank Access template, copy and write to it, and send it instead of an Excel file. Of course, this would be a shift for your application, but Access won't have the same row limit.