getting out of memory exception while using fileStream - asp.net

I am facing an Issue while uploading file from a web page to server. It works fine for files upto 200 MB, but starts throwing out of memory exception.
Could you please help me
I have pasted the code below
private void UploadToServer(HttpPostedFile oHttpPostedFile)
{
string CalCheckSum = string.Empty;
try
{
string FileName = getFileName(oHttpPostedFile.FileName.Trim());
if (File.Exists(Server.MapPath("Upload") + "\\" + System.IO.Path.GetFileName(FileName)))
{
File.Delete(Server.MapPath("Upload") + "\\" + System.IO.Path.GetFileName(FileName));
}
string serverFilePath = Server.MapPath("Upload") + "\\" + System.IO.Path.GetFileName(FileName);
FileStream fs = new FileStream(serverFilePath, FileMode.CreateNew);
string strFileFormName = serverFilePath;
// Uri oUri = new Uri(strUrl);
// DFB: Upload goes into stream
Stream myStream = oHttpPostedFile.InputStream;
string _name = oHttpPostedFile.FileName;
string _contentType = oHttpPostedFile.ContentType;
// DFB: Create buffer for stream
Byte[] myBuffer;
myBuffer = new byte[10240];
if (myStream.Length == 0)
{
//Zero Bytes file Can not be processed
CalCheckSum = string.Empty;
return;
}
else if (myStream.Length > 10240)
myBuffer = new byte[10240];
else
myBuffer = new byte[myStream.Length];
StringBuilder filecontent = new StringBuilder();
int fileLength = (int)myStream.Length;
int length = (int)myStream.Length / myBuffer.Length + 1;
int lastPacketLength = (int)myStream.Length % 10240;
int count = 1;
while (myStream.Read(myBuffer, 0, myBuffer.Length) > 0)
{
if (count == length)
fs.Write(myBuffer, 0, lastPacketLength);
else
fs.Write(myBuffer, 0, myBuffer.Length);
count++;
}
fs.Close();
fs.Dispose();
myStream.Dispose();
myBuffer = null;
myStream = null;
FileStream fileStream = File.OpenRead(serverFilePath);
byte[] pbytCombinedArrays = new byte[fileLength];
int numBytesToRead = fileLength;
int numBytesRead = 0;
while (numBytesToRead > 0)
{
// Read may return anything from 0 to numBytesToRead.
int n = fileStream.Read(pbytCombinedArrays, numBytesRead, numBytesToRead);
// Break when the end of the file is reached.
if (n == 0)
break;
numBytesRead += n;
numBytesToRead -= n;
}
fileStream.Dispose();
fileStream.Close();
}

You get the exception because you are trying to allocate more memory than is allowed for the application. Web applications are generally limited to about 300 MB.
The solution would be to avoid reading the entire file into memory. It's simply too large for a web application to handle all at once.

Related

Speed up webcrawler

I wrote this script, which works perfectly, except it is taking about 5 minutes (give a few) to run. What would be the best way to speed this up?
Kind regards, Rob
System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
sw.Start();
int divnumber = 17476;
HtmlAgilityPack.HtmlWeb web = new HtmlAgilityPack.HtmlWeb();
while (divnumber < 18500)
{
string DivUrl = "https://www.hattrick.org/nl/World/Series/?LeagueLevelUnitID=" + divnumber;
HtmlAgilityPack.HtmlDocument doc = web.Load(DivUrl);
var ClassShy = doc.DocumentNode.SelectNodes("//a[#class='shy']");
if (ClassShy != null)
{
ClassShy.ToList();
int i = 0;
foreach (var item in ClassShy)
{
i++;
}
int divForList = divnumber - 17475;
if (i > 4)
{
Console.WriteLine("VI." + divForList + " - " + i);
}
}
divnumber++;
}
sw.Stop();
TimeSpan ts = sw.Elapsed;
string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}:{3:00}", ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);
Console.WriteLine("\nDone in " + elapsedTime);
sw.Reset();

Request was aborted

I tried to split large file into small 4KB chunks and send each chunks to oracle cloud but request is aborting. Please check the splitting code whether it is correct or not or am I doing wrong something else.
FileStream rdr = new FileStream(fileToUpload, FileMode.Open, FileAccess.Read);
byte[] inData = new byte[4096];
long chunkSize= rdr.Length;
long chunkBytesToRead = chunkSize;
using (Stream reqStream = request.GetRequestStream())
{
while (rdr.Position < rdr.Length)
{
int chunkBytesRead = 0;
int maxread = 4096;
while (chunkBytesRead < chunkSize)
{
int bytesRead = rdr.Read(inData,0,maxread);
if (bytesRead == 0)
{
break;
}
reqStream.Write(inData, 0, bytesRead);
chunkBytesToRead -= bytesRead;
if (chunkBytesToRead < 4096)
maxread = (int)chunkBytesToRead;
chunkBytesRead += bytesRead;
}
}
reqStream.Close();
}

unable to download xlsx file using POI

Hi i working on poi poc to generate xlsx file in webapplication i was able to render excel standalone well but when i integrate codebase in my project i was getting below error.
below is code base base to download xlsx file in webapplication
XSSFWorkbook workbook = new XSSFWorkbook();
XSSFSheet spreadsheet = workbook.createSheet("Sheet1");
XSSFRow row = spreadsheet.createRow(0);
XSSFCell cell;
while (response.nextResultSet()) {
resultSet = response.getResultSet();
Object[] columnMetaData = resultSet.getColumnNames();
int columnCount = columnMetaData.length;
//Columns Loop
ArrayList<String> columns = new ArrayList<String>();
for (int i = 1; i < columnCount; i++) {
String columnName = (String) columnMetaData[i];
columns.add(columnName);
cell = row.createCell(i-1);
cell.setCellValue(columnName);
}
int i=1;
while (resultSet.nextRow()) {
row = spreadsheet.createRow(i);
i++; // counter for each row of data
for (int j = 0; j < columnMetaData.length; j++)
{
String keyVal = String.valueOf(columnMetaData[j]);
String value = (String)resultSet.getValue(keyVal);
cell = row.createCell(j);
cell.setCellValue(value);
}
}
log.info("value if i--->" + i);
for (int k = 1; k < columnCount; k++) {
spreadsheet.autoSizeColumn(k-1);
}
}
ByteArrayOutputStream outByteStream = new ByteArrayOutputStream();
workbook.write(outByteStream);
context.getResponse().setHeader("content-disposition", "inline;filename=" + calendar.getTimeInMillis() + ".xlsx");
context.getResponse().setContentType("application/Excel");
context.getRequest().setAttribute("called_from", "excel");
//context.getResponse().setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
//context.getResponse().setHeader("Content-Disposition", "attachment; filename=testxls.xlsx");
}
OutputStream os = null;
os = context.getResponse().getOutputStream();
byte [] outArray = is.toByteArray();
context.getResponse().setContentLength(outArray.length);
os.write(outArray);
log.info(os.toString());
os.flush();

Write excel file in C# 2.0 without using any third party dlls/interop/sdk

I want to write a excel file (xls/xlsx) in C#.
Constraints:
I don't want to use following things while writing:
1. Third Party dll( My client is not allowing me to use third party dll )
2. SDK(Same as above reason as am not allowed to download)
3. Not allowed to use interop as in target server there is no office installed.
4. OpenXml
Info:
1. Using VS 2005.
2. Using ASP.Net (C# 2.0)
Write a csv file which can be opened directly by Excel.
I think this will help you
[DllImport("user32.dll")]
private static extern uint GetWindowThreadProcessId(IntPtr hWnd, out uint lpdwProcessId);
public DataTable readexcelV1(string path, bool isHeader, params string[] sheetname)
{
bool isFirstTime = true;
string filePath = path;
Microsoft.Office.Interop.Excel.Application ExcelApp = new Microsoft.Office.Interop.Excel.Application();
ExcelApp.Visible = false;
Microsoft.Office.Interop.Excel.Workbook wb = ExcelApp.Workbooks.Open(filePath, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value, Missing.Value);
string ActiveSheetName = "";
if (sheetname.Count() == 0)
ActiveSheetName = ((dynamic)wb).ActiveSheet.Name;
else
ActiveSheetName = Convert.ToString(sheetname[0]);
Microsoft.Office.Interop.Excel.Worksheet sh = (Microsoft.Office.Interop.Excel.Worksheet)wb.Sheets[ActiveSheetName];
//////////////////////////
excellt.Range excelRange = sh.UsedRange;
Int32 Userows = excelRange.Rows.Count;
Int32 Usecolumns = excelRange.Columns.Count;
int Pointrow = 1;
int expVar = 75000;
int Pointcolumn = expVar;
int co = 0;
int r = 0;
expVar = Userows > expVar ? expVar : Userows;
decimal lcount = Convert.ToDecimal(Userows) / Convert.ToDecimal(expVar);
Int32 lcountint = Convert.ToInt32(Math.Ceiling(lcount));
DataTable dt = new DataTable();
if (isHeader == true)
{
int duplicateColum = 1;
for (int j = 1; j <= excelRange.Columns.Count; j++) // Header Names
{
if (excelRange.Cells[1, j].Value2 != null)
{
if (!dt.Columns.Contains(Convert.ToString(excelRange.Cells[1, j].Value2).Trim()))
{
dt.Columns.Add(Convert.ToString(excelRange.Cells[1, j].Value2).Trim());
}
else
{
dt.Columns.Add(Convert.ToString(excelRange.Cells[1, j].Value2).Trim() + duplicateColum.ToString());
duplicateColum++;
}
}
else
{
dt.Columns.Add("Column" + j.ToString());
duplicateColum++;
}
}
}
else
{
for (int j = 1; j <= excelRange.Columns.Count; j++) // Header Names
dt.Columns.Add("Column" + j.ToString());
}
string cell = string.Empty;
string colCharset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
int colCharsetLen = colCharset.Length;
if (dt.Columns.Count > colCharsetLen)
{
cell = colCharset.Substring(
(dt.Columns.Count - 1) / colCharsetLen - 1, 1);
}
cell += colCharset.Substring(
(dt.Columns.Count - 1) % colCharsetLen, 1);
for (int i = 0; i < lcountint; i++)
{
string cell1 = "A" + Convert.ToString(Pointrow);
string cell2 = cell + Convert.ToString(Pointcolumn);
excellt.Range rng = sh.get_Range(cell1, cell2);
object[,] x = (object[,])rng.get_Value(excellt.XlRangeValueDataType.xlRangeValueDefault);
int NumRow = isHeader == true && isFirstTime == true ? 2 : 1;
int loopUpto = Userows > expVar ? (Pointcolumn - Pointrow) : expVar;
loopUpto += 2;
isFirstTime = false;
while (NumRow < loopUpto)
{
dt.Rows.Add();
co = 0;
for (int c = 1; c <= Usecolumns; c++)
{
dt.Rows[r][co] = Convert.ToString(x[NumRow, c]);
co++;
}
NumRow++;
r++;
}
Pointrow += expVar;
Pointcolumn += expVar;
Pointcolumn = (Pointcolumn >= Userows) ? Userows : Pointcolumn;
progressBar1.Value += 10;
}
GC.Collect();
GC.WaitForPendingFinalizers();
if (ExcelApp != null)
{
ExcelApp.Quit();
int hWnd = ExcelApp.Application.Hwnd;
uint processID; GetWindowThreadProcessId((IntPtr)hWnd, out processID);
Process[] procs = Process.GetProcessesByName("EXCEL");
foreach (Process p in procs)
{
if (p.Id == processID)
p.Kill();
}
Marshal.FinalReleaseComObject(ExcelApp);
}
return dt;
}
I hope client is paying by the hour because these are stupid solution constraints (unless there are extenuating circumstances)
However, the XML format for an Excel spreadsheet is actually not that bad to write code for, The newer version of EXcel will open an .xml file and treat it as a spreadsheet. You can embed formatting, headers, etc. in the .xml format and since XML is structured, it is far more maintainable than writing lots of bytes of data via code.
Use excel, create an document in the right format, use that as a template, and tweek till it works as needed. Make sure you have a decent XML editor and lots of patience.
I think a Text-Tab Delimited will be much simpler and faster to write than attempting a CSV generator. CSV's can sometimes have weird quirks if you aren't careful during your design phase and carefully examine all expected variations of your data. CSV's will require text qualifiers etc. Unless you're expecting your data to have TAB's all over the place, a TAB-Delimited file is pretty easy to do.

Is there an ISO or public mapping of locations to "olson database" format (e.g. TZInfo / TZ) [duplicate]

Given: LDAP stores location of users.
How do I drive their timezones using their location? Any pointers are accepted, Java language preferred.
Thanks in advance.
This depends on information the "location" contains? You'd somehow need to map the location to a timezone name, preferably the Olson style timezone names, because they are more detailed and easier to map, as they are locations themselves.
If it's an approximate addres (like country and city or so) then several geolocation services do include timezones in their information, so you can call these services and see.
If it's a geolocation with latitude and longitude then a site called Earthtools can give you the timezone. http://www.earthtools.org/webservices.htm#timezone
There is this database that provides mappings from cities and countries to timezones: http://citytimezones.info/cms/pending_requests.htm
Unfortunately it uses Windows timezone names, but you can this data http://unicode.org/repos/cldr/trunk/common/supplemental/windowsZones.xml from Unicode.org to map between Windows timezone names and the Olson TZ names.
I haven't actually done this but the following should work:
You can first use GeoGoogle java library to get the longitude/latitude from the city-state-country.
Next, you can use EarthTools (and some java code of your own) mentioned by Lennart to get the timezone :)
We are using the MaxMind GeoIP database to get information regarding the location of a user. They have a paid version (99.8% accuracy) as well as a free version (99.5% accuracy).
They also provide you with APIs in Java, C, PHP etc that will enable you to query its database which you can download and keep locally (updates provided each month). The database provides you info regarding the client city, state, country etc on the basis of IP addresses.
Hope this helps.
Try this code for use Google Time Zone API from Java:
String get_xml_server_reponse(String server_url){
URL xml_server = null;
String xmltext = "";
InputStream input;
try {
xml_server = new URL(server_url);
try {
input = xml_server.openConnection().getInputStream();
final BufferedReader reader = new BufferedReader(new InputStreamReader(input));
final StringBuilder sBuf = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null)
{
sBuf.append(line);
}
}
catch (IOException e)
{
Log.e(e.getMessage(), "XML parser, stream2string 1");
}
finally {
try {
input.close();
}
catch (IOException e)
{
Log.e(e.getMessage(), "XML parser, stream2string 2");
}
}
xmltext = sBuf.toString();
} catch (IOException e1) {
e1.printStackTrace();
}
} catch (MalformedURLException e1) {
e1.printStackTrace();
}
return xmltext;
}
private String get_UTC_Datetime_from_timestamp(long timeStamp){
try{
Calendar cal = Calendar.getInstance();
TimeZone tz = cal.getTimeZone();
int tzt = tz.getOffset(System.currentTimeMillis());
timeStamp -= tzt;
// DateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss",Locale.getDefault());
DateFormat sdf = new SimpleDateFormat();
Date netDate = (new Date(timeStamp));
return sdf.format(netDate);
}
catch(Exception ex){
return "";
}
}
class NTP_UTC_Time
{
private static final String TAG = "SntpClient";
private static final int RECEIVE_TIME_OFFSET = 32;
private static final int TRANSMIT_TIME_OFFSET = 40;
private static final int NTP_PACKET_SIZE = 48;
private static final int NTP_PORT = 123;
private static final int NTP_MODE_CLIENT = 3;
private static final int NTP_VERSION = 3;
// Number of seconds between Jan 1, 1900 and Jan 1, 1970
// 70 years plus 17 leap days
private static final long OFFSET_1900_TO_1970 = ((365L * 70L) + 17L) * 24L * 60L * 60L;
private long mNtpTime;
public boolean requestTime(String host, int timeout) {
try {
DatagramSocket socket = new DatagramSocket();
socket.setSoTimeout(timeout);
InetAddress address = InetAddress.getByName(host);
byte[] buffer = new byte[NTP_PACKET_SIZE];
DatagramPacket request = new DatagramPacket(buffer, buffer.length, address, NTP_PORT);
buffer[0] = NTP_MODE_CLIENT | (NTP_VERSION << 3);
writeTimeStamp(buffer, TRANSMIT_TIME_OFFSET);
socket.send(request);
// read the response
DatagramPacket response = new DatagramPacket(buffer, buffer.length);
socket.receive(response);
socket.close();
mNtpTime = readTimeStamp(buffer, RECEIVE_TIME_OFFSET);
} catch (Exception e) {
// if (Config.LOGD) Log.d(TAG, "request time failed: " + e);
return false;
}
return true;
}
public long getNtpTime() {
return mNtpTime;
}
/**
* Reads an unsigned 32 bit big endian number from the given offset in the buffer.
*/
private long read32(byte[] buffer, int offset) {
byte b0 = buffer[offset];
byte b1 = buffer[offset+1];
byte b2 = buffer[offset+2];
byte b3 = buffer[offset+3];
// convert signed bytes to unsigned values
int i0 = ((b0 & 0x80) == 0x80 ? (b0 & 0x7F) + 0x80 : b0);
int i1 = ((b1 & 0x80) == 0x80 ? (b1 & 0x7F) + 0x80 : b1);
int i2 = ((b2 & 0x80) == 0x80 ? (b2 & 0x7F) + 0x80 : b2);
int i3 = ((b3 & 0x80) == 0x80 ? (b3 & 0x7F) + 0x80 : b3);
return ((long)i0 << 24) + ((long)i1 << 16) + ((long)i2 << 8) + (long)i3;
}
/**
* Reads the NTP time stamp at the given offset in the buffer and returns
* it as a system time (milliseconds since January 1, 1970).
*/
private long readTimeStamp(byte[] buffer, int offset) {
long seconds = read32(buffer, offset);
long fraction = read32(buffer, offset + 4);
return ((seconds - OFFSET_1900_TO_1970) * 1000) + ((fraction * 1000L) / 0x100000000L);
}
/**
* Writes 0 as NTP starttime stamp in the buffer. --> Then NTP returns Time OFFSET since 1900
*/
private void writeTimeStamp(byte[] buffer, int offset) {
int ofs = offset++;
for (int i=ofs;i<(ofs+8);i++)
buffer[i] = (byte)(0);
}
}
String get_time_zone_time(GeoPoint gp){
String erg = "";
String raw_offset = "";
String dst_offset = "";
double Longitude = gp.getLongitudeE6()/1E6;
double Latitude = gp.getLatitudeE6()/1E6;
// String request = "http://ws.geonames.org/timezone?lat="+Latitude+"&lng="+ Longitude+ "&style=full";
long tsLong = 0; // System.currentTimeMillis()/1000;
NTP_UTC_Time client = new NTP_UTC_Time();
if (client.requestTime("pool.ntp.org", 2000)) {
tsLong = client.getNtpTime();
}
if (tsLong != 0)
{
tsLong = tsLong / 1000;
// https://maps.googleapis.com/maps/api/timezone/xml?location=39.6034810,-119.6822510&timestamp=1331161200&sensor=true
String request = "https://maps.googleapis.com/maps/api/timezone/xml?location="+Latitude+","+ Longitude+ "&timestamp="+tsLong +"&sensor=true";
String xmltext = get_xml_server_reponse(request);
if(xmltext.compareTo("")!= 0)
{
int startpos = xmltext.indexOf("<TimeZoneResponse");
xmltext = xmltext.substring(startpos);
XmlPullParser parser;
try {
parser = XmlPullParserFactory.newInstance().newPullParser();
parser.setInput(new StringReader (xmltext));
int eventType = parser.getEventType();
String tagName = "";
while(eventType != XmlPullParser.END_DOCUMENT) {
switch(eventType) {
case XmlPullParser.START_TAG:
tagName = parser.getName();
break;
case XmlPullParser.TEXT :
if (tagName.equalsIgnoreCase("raw_offset"))
if(raw_offset.compareTo("")== 0)
raw_offset = parser.getText();
if (tagName.equalsIgnoreCase("dst_offset"))
if(dst_offset.compareTo("")== 0)
dst_offset = parser.getText();
break;
}
try {
eventType = parser.next();
} catch (IOException e) {
e.printStackTrace();
}
}
} catch (XmlPullParserException e) {
e.printStackTrace();
erg += e.toString();
}
}
int ro = 0;
if(raw_offset.compareTo("")!= 0)
{
float rof = str_to_float(raw_offset);
ro = (int)rof;
}
int dof = 0;
if(dst_offset.compareTo("")!= 0)
{
float doff = str_to_float(dst_offset);
dof = (int)doff;
}
tsLong = (tsLong + ro + dof) * 1000;
erg = get_UTC_Datetime_from_timestamp(tsLong);
}
return erg;
}
And use it with:
GeoPoint gp = new GeoPoint(39.6034810,-119.6822510);
String Current_TimeZone_Time = get_time_zone_time(gp);

Resources