How to optimize this code that read from xml file - asp-classic

We have ther following classic asp code that read from xml file, it shows a poor performance when there are more than 10 concurrent requests to that page , can someone figure out the performance problem in this code( we know one of the the problems which is using fileSystemObject but we do not have alternatives for it!):
set filesys=server.CreateObject("Scripting.FileSystemObject")
if filesys.FileExists(sourcefile) then
set source = Server.CreateObject("Msxml2.DOMDocument")
source.validateOnParse = false
source.resolveExternals = false
source.preserveWhiteSpace = false
source.load(sourcefile)
If source.ParseError.errorCode <> 0 Then
str_head=source.selectSingleNode("/LIST/ITEM/NEWSITEM/HEADLINE//").text
str_by=source.selectSingleNode("//LIST//ITEM//NEWSITEM//PROVIDER//").text
News_date_orig = source.selectSingleNode("/LIST/ITEM/NEWSITEM/CREATED//").text
str_date= formatdatetime(source.selectSingleNode("//LIST//ITEM//NEWSITEM//CREATED//").text,1)
set bodyNode=source.selectSingleNode("/LIST/ITEM/NEWSITEM//BODY//")
styleFile=Server.MapPath("/includes/xsl/template.xsl")
Set style = Server.CreateObject("Msxml2.DOMDocument")
style.validateOnParse = false
style.resolveExternals = false
style.preserveWhiteSpace = false
style.load(styleFile)
news_full = bodyNode.transformNode(style)
if len(news_full) < 10 then
news_full = str_abstract
end if
DiscriptionKeyWord = stripHTMLtags(news_full)
DiscriptionKeyWord=StrCutOff(DiscriptionKeyWord, 200, "...")
headerTitle=str_head
Set style=nothing
Set source = nothing
end if
set filesys= nothing
The following is stripHTMLtags function:
Function stripHTMLtags(HTMLstring)
Set RegularExpressionObject = New RegExp
With RegularExpressionObject
.Pattern = "<[^>]+>"
.IgnoreCase = True
.Global = True
End With
stripHTMLtags = RegularExpressionObject.Replace(HTMLstring, "")
Set RegularExpressionObject = nothing
End Function
UPDATE: I put a timer to show the execution time for the function that read xml file and found that it takes around 3 seconds on the production server while it takes less than 1 second on my PC! What does tht mean?! I'm lost.

Option Explicit
If your scripts don't begin with Option Explict then make that change now. Then fix all the compile errors that show up. Doesn't help performance but when I see evidence that this biggest of all scripting mistakes is being made it just needs mentioning.
FileSystemObject
I doubt the performance problem is a result of the FileSystemObject, all you are doing is creating an instance and testing for the existance of a file. That is hardly likely to cause a problem.
Having said that I would ditch the FileSystemObject anyway. Just let the script throw an error if there is a problem. Use IIS Manager to map 500.100 status codes to an ASP page that presents a friendly "Something bad happened" page to the user. (500.100 is the status of a request when the script throws an exception). Also test the boolean result of the DOM load method, throw an error when parse error is not 0 also. That way you hand over all the ugly exception handling to the 500.100 handling page and your code can remain clean just dealing with the nominal path of the code.
Tidy up the paths
Perhaps there is a reason why you are using "//" a lot in your paths (but inconsistently) but I'm going to assume there isn't so we can simplify some of the paths:
Dim newsItem: Set newsItem = source.selectSingleNode("/LIST/ITEM/NEWSITEM")
Dim str_head: str_head = newsItem .selectSingleNode("HEADLINE").text
Dim str_by: str_by = newsItem .selectSingleNode("PROVIDER").text
Dim News_date_orig: News_date_orig = newsItem .selectSingleNode("CREATED").text
Dim str_date: str_date = formatdatetime(News_date_orig, 1)
Dim bodyNode: Set bodyNode = newsItem.selectSingleNode("BODY")
Cache XSLTemplate
An area where you could gain some real perfomance improvement is to cache the XSL Transform in the application object (which is possible due to XSLTemplate being a free threaded object). Like this:
Dim xslTemplate
If IsObject(Application("xsl_template")) Then
Set xslTemplate = Application("xsl_template")
Else
Set style = Server.CreateObject("Msxml2.FreeThreadedDOMDocument.3.0")
style.async = false
style.validateOnParse = false
style.resolveExternals = false
style.preserveWhiteSpace = false
style.load Server.MapPath("/includes/xsl/template.xsl")
Set xslTemplate = CreateObject("MSXML2.XSLTemplate.3.0")
xslTemplate.stylesheet = xsl
Set Application("template") = xslTemplate
End If
Dim xslProc: Set xslProc = xslTemplate.createProcessor()
xslProc.input = bodyNode
xslProc.transform()
news_full = xslProc.output
The effort to read, parse and compile the XSL transform is only done once in the whole lifetime of the application.
The most likely culprit
To be honest I suspect the most likely culprit is stripHTMLtags. This sounds like a whole load of string handling and the performance of VBScript string handling is poor. Its especially poor when code is not properly optomised to be aware of the string handling performance limitations (e.g. excessive and repeative string concatenations). Its also likely to be where the most actual VBScript occurs which is often the cause of a performance problem.

You can load the XML into a string on application startup and store it in an Application object.
Then, instead of using source.load, use source.loadXML with this string - you will not be accessing the filesystem anymore. See the documentation for loadXML.
Make sure you use a version of MSXML 3.0 or over to use this method.
Server.CreateObject("Msxml2.DOMDocument.6.0")
Update - seeing as you have thousands of these files and you believe it is infeasible to store them in memory in the Application object, you should be using a database to handle the concurrency issues you are seeing. Store the contents of the files keyed to something like their current filename/path - retrieve them from the database and load to a document using the same loadXML mechanism.

Related

iisExpress local httpruntime.cache always null

strange problem here. On local development in asp.net webforms (4.5 / 4.7) I am finding httpruntime.Cache always null even when properly set. I attempted it on another iis express workstation and found the same behavior, even with a tester single page web page. That same page in production IIS 7.5 works and is storing and delivering from cache. The code specifically is below, but I have tried a tester storing a simple string in httpruntime.Cache.
var cache = System.Runtime.Caching.MemoryCache.Default;
var luCacheKey = "lu_" + dsName;
var ic = HttpRuntime.Cache.Get(luCacheKey) as ICollection;
if (ic == null) {
and from the tester
var item = HttpRuntime.Cache.Get("x");
if (item == null)
{
HttpContext.Current.Cache.Insert("x", "test" , null, DateTime.Now.AddHours(1), Cache.NoSlidingExpiration);
Response.Write("added to cache<br>");
}
else {
Response.Write("already in cache");
}
So, I am wondering if there is something perhaps in web.config that I could look at or is this expected IIS express behavior? Note, System.runtime.Caching does work properly.
var cache = System.Runtime.Caching.MemoryCache.Default;
var ic = cache[luCacheKey] as ICollection;
if (ic == null)
{
var filterCriteria = new BinaryOperator("LookupGroup", dsName, BinaryOperatorType.Equal);
var lookups = xpoSession.GetClassInfo(typeof(Lookups));
ic = xpoSession.GetObjects(lookups, filterCriteria, new SortingCollection(), 0, 0, false, false);
var cachePolicy = new System.Runtime.Caching.CacheItemPolicy() { AbsoluteExpiration = DateTime.Now + TimeSpan.FromMinutes(30) };
cache.Add(new System.Runtime.Caching.CacheItem(luCacheKey, ic), cachePolicy);
You incorrectly add your object to the cache.
Instead of DateTime.Now follow the docs and put DateTime.UtcNow. This resolves a common issue where your machine is in a "non-zero" time zone which prevents the inner logic of the cache to manage your expirations correctly.
From the docs
To avoid possible issues with local time such as changes from standard time to daylight saving time, use UtcNow rather than Now for this parameter value.
https://msdn.microsoft.com/en-us/library/4y13wyk9(v=vs.110).aspx
Adding more information as follow up on why the behavior may change between servers.
This change in behavior may be caused by having .NET 4.7 installed on the machine. The article linked below says that Microsoft will fix this in the next version of .NET and in the next hotfix.
Quoting parts of the Microsoft page:
Symptoms:
Assume that you have Microsoft .NET Framework 4.7 installed on a
computer. When you try to insert items into the Cache object by using
the Cache.Insert (string, object, CacheDependency, DateTime, TimeSpan)
Insert overload method, you may notice that the inserted Cache items
expire much earlier or later than the specified DateTime (expiration
time).
Cause:
The internal implementation of System.Web.Caching.Cache uses
Coordinated Universal Time (UTC) time-stamp for absolute expiration.
But this particular Cache.Insert (string, object, CacheDependecy,
DateTime, TimeSpan) Insert overload method does not make sure whether
the expiration time is converted to UTC. Therefore, expiration for
items that are inserted into the Cache object by using this overload
will occur earlier or later than expected, depending on the computer
time zone difference from Greenwich Mean Time (GMT).
Workaround:
The temporary workaround for this issue is to use either the Cache.Add method or a different Cache.Insert overload method.
Resolution:
This issue will be fixed in the next version of the .NET Framework, and will also be available in the next hotfix for the .NET Framework 4.7.
References:
https://support.microsoft.com/en-us/help/4035412/fix-expiration-time-issue-when-you-insert-items-by-using-the-cache-ins
http://vimvq1987.com/2017/08/episerver-caching-issue-net-4-7/

libgit2sharp.Patch outofmemory

I try to use libgit2sharp.Patch to find how much line added or deleted, but i got an error while i try to run it. When i run my asp.net mvc project in debug mode, it doesn't have any problem, until i run it without debug mode, i got my web load too long and didn't show the page. When i run in debug mode again, finally an error appear from libgit2sharp.Patch variable with error message system.outofmemory. This is how i implement libgit2sharp.Patch
Patch treePatchInfo = repo.Diff.Compare<Patch>(firstTree, compareTree, null, compareOptions: compareOptions);
commitChangeValue = from s in treeChangeInfo
let patch = treePatchInfo[s.Path]
select new CommitChangeModel
{
ChangeKind = s.Status,
LinesAdded = patch.LinesAdded,
LinesDeleted = patch.LinesDeleted,
OldPath = s.Path,
Patch = patch.Patch,
Path = s.Path
};
If you're only interested in number of additions/removal per file, I'd suggest you to rather rely on the following construct would be more efficient.
var stats = repo.Diff.Compare<PatchStats>(...);
You can take a peek at PR #660 where it's been introduced to get a first grasp of its usage.
Note: Regarding the OOM Exception, we'd very interested in getting more information about it. Would you be so kind as to open an issue in the bug tracker so that we can get a deeper look at it?

VBScript Out Of Memory Error

I have a classic ASP CRM that was built by a third party company. Currently, I have access to the source code and am able to make any changes required.
Randomly throughout the day, usually after some prolonged usage by users, most of my pages start getting an Out of Memory error.
The way that the application is built, is all the pages and scripts pull core functions from a Global.asp file. In that file are embeds to other global files as well, but the error presented shows
Out Of Memory
WhateverScriptYouTriedToRun.asp Line 0
Line 0 is the include for the global.asp file. Once the error occurs, after an unspecified amount of time the error occurence subsides for some time but then begins to reoccur again. With how the application is written, and the functions it uses, and the "diagnostics" I've already done - it seems to be a common used function that is withholding data such as recordset or something of that nature and then not releasing it properly. Other users then try to use the same function and eventually it just fills up causing the error. The only way for me to effectively clear the error is to actually restart IIS, Recycle the App Pool, and Restart the SQL Server Services.
Needless to say, myself and my users are getting annoyed....
I can't pinpoint the error due to the actual error message presented being Line 0 - but from there I have no idea where in the 20K lines of code it could be hanging up. Any thoughts or ideas on how to isolate or at least point me in the right direction to begin clearing this up? Is there a way for me to increase "memory" size for VBScript? I know there is a limitation but is it set at say...512K and you can increase it to 1GB?
Here are things I have tried:
Removing SQL Inline statements into Views
Going through several hundred scripts and ensuring that every OpenConnection & OpenRecordSet is followed by an appropriate Close.
Going through the Global File and commenting out any large SQL statements such as ApplicationLog (A function that writes the executed query into a table).
Some smaller script edits.
Common Memory Leak
You say you are closing all recordsets and connections which is good.
But are you deleting objects?
For example:
Set adoCon = new
Set rsCommon = new
'Do query stuff
'You do this:
rsCommon.close
adocon.close
'But do you do this?
Set adoCon = nothing
Set rsCommon = nothing
No garbage collection in classic ASP, so any objects not destroyed will remain in memory.
Also, ensure your closes/nothings are run in every branch. For example:
adocon.open
rscommon.open etc
'Sql query
myData = rscommon("condition")
if(myData) then
response.write("ok")
else
response.redirect("error.asp")
end if
'close
rsCommon.close
adocon.close
Set adoCon = nothing
Set rsCommon = nothing
Nothing is closed/destroyed before the redirect so it will only empty memory some of the time as not all branches of logic lead to the proper memory clearance.
Better Design
Also unfortunately it sounds like the website wasn't designed well. I always structure my classic ASP as:
<%
Option Explicit
'Declare all vars
Dim this
Dim that
'Open connections
Set adoCon...
adocon.open()
'Fetch required data
rscommon.open strSQL, adoCon
this = rsCommon.getRows()
rsCommon.close
'Fetch something else
rscommon.open strSQL, adoCon
that = rsCommon.getRows()
rsCommon.close
'Close connections and drop objects
adoCon.close
set adoCon = nothing
set rscommon = nothing
'Process redirects
if(condition) then
response.redirect(url)
end if
%>
<html>
<body>
<%
'Use data
for(i = 0 to ubound(this,2)
response.write(this(0, i) & " " & this(1, i) & "<br />")
next
%>
</body>
</html>
Hope some of this helped.
Have you looked at using a memory monitoring tool to see how much memory fragmentation is happening? My guess at a possible cause is that some object of a size is trying to be created but there isn't enough room in the memory to store it as one contiguous chunk. Imagine needing room to store an object that would take 100 MB and while there may be several hundred megabytes free, the largest contiguous chunk is 90MB then this doesn't fit.
Debug Diagnostic Tool v1.1 would be a tool where Bernard's articles may help in understanding how to use the tool.
Another thought is the question of how much string concatenation is there in the code? I remember where I used to work had problems with doing a lot of string concatenation operations that sucked up memory that may be another idea to consider.
Yeah, I could see some shock at that kind of number the first few times you see it but then if you understand what the code is doing it may make sense for why so much space gets reserved right off the bat at times.
I haven't used that debug tool specifically but I did have a tool that took a snapshot of memory when pages were hung so I couldn't tell if there was a performance impact of the tool or not. Course in my case I used a similar tool in 2004 so it has been a few years since I've had to research this kind of issue.
Just going to throw this in here, but this problem has taken a long time to solve. Here's a breakdown of what we did:
We took all the inline SQL and made SQL Views, every SELECT statement is now handled with a VIEW first.
I took every single SQL INSERT and UPDATE (as much as I could without breaking the system) and put them into Stored Procedures.
#2 was the one item that really made the biggest difference
Went through several thousand scripts, and ensured that variables were properly disposed of, and all the DB Open Connections were followed correctly with a Close Connection and same with Open/Close RecordSet.
One of the slow killers was doing something like:
ID = Request.QueryString("ID)
at the top of the page. Before redirecting, or closing a page, there is always a:
Set ID = Nothing
or the complete removal of the inference.

ASP.NET Unexpected and Different Behavior in Different Environments

I have an ASP.NET site (VB.NET) that I'm trying to clean up. When it was originally created it was written with no error handling, and I'm trying to add it in to improve the User Experience.
Try
If Not String.IsNullOrEmpty(strMfgName) And Not String.IsNullOrEmpty(strSortType) Then
If Integer.TryParse(Request.QueryString("CategoryID"), i) And String.IsNullOrEmpty(Request.QueryString("CategoryID"))
MyDataGrid.DataSource = ProductCategoryDB.GetMfgItems(strMfgName, strSortType, i)
Else
MyDataGrid.DataSource = ProductCategoryDB.GetMfgItems(strMfgName, strSortType)
End If
MyDataGrid.DataBind()
If CType(MyDataGrid.DataSource, DataSet).Tables("Data").Rows.Count > 0 Then
lblCatName.Text = CType(MyDataGrid.DataSource, DataSet).Tables("Data").Rows(0).Item("mfgName")
End If
If MyDataGrid.Items.Count < 2 Then
cboSortTypes.Visible = False
table_search.Visible = False
End If
If MyDataGrid.PageCount < 2 Then
MyDataGrid.PagerStyle.Visible = False
End If
Else
lblCatName.Text &= "<br /><span style=""fontf-size: 12px;"">There are no items for this manufacturer</span>"
MyDataGrid.Visible = False
table_search.Visible = False
End If
Catch
lblCatName.Text &= "<br /><span style=""font-size: 12px;"">There are no items for this manufacturer</span>"
MyDataGrid.Visible = False
table_search.Visible = False
End Try
Now, this is trying to avoid generating a 500 error by catching exceptions. There can be three items on the query string, but only two matter here. In my test environment and in Visual Studio when I run this site, it doesn't matter if that item is on the query string. In production, it does matter. If that third item isn't present (SubCategoryID) on the query string, then the "There are no items for this manufacturer" displays instead of the data from the database.
In the two different environments I am seeing two different code execution paths, despite the same URLs and the same code base.
The site is running on Server 2003 with IIS 6.
Thoughts?
EDIT:
In response to the answer below, I doubt it's a connection error (though I see what you're getting to), as when I add the SubCategoryID to the query string, the site works correctly (displaying data from the database).
Also, if please let me know if you have any suggestions for how to test this scenario, without deploying the code back to production (it's been rolled back).
I think you should try to print out the exception details in your catch block to see what the problem is. It could anything for example a connection error to your database.
The error could be anything, and you should definitely consider printing this out or logging it somewhere, rather than making the assumption that there's no data. You're also outputting the same error message to the UI for two different code paths, which makes things harder to debug, especially without knowing if an exception occurred, and if so, what it was.
Generally, it's also better not to have a catch for all exceptions in cases like this, especially without logging the error. Instead, you should catch specific exceptions and handle these appropriately, and any real exceptions can get passed up the stack, ideally to a global error handler which can log it and/or send out some kind of error notification.
I discovered the reason yesterday. In short it was because when I copied my files from my computer into my dev-test environment, I missed a file, which ironically caused it to work, rather than not. So in the end it would have functioned the same in both environments.

Flex slow first Http request

When i use loader.load(request); for the first time, my flex freeze for 10 secondes before posting the data (i can see the web server result in real time).
However if redo a similar POST with other data but same request.url, it's instantaneous.
// Multi form encoded data
variables = new URLVariables();
variables.user = "aaa";
variables.boardjpg = new URLFileVariable(data.boardBytes, "foo.jpg");
request = new URLRequestBuilder(variables).build();
request.url = "http://localhost:8000/upload/";
loader.load(request);
How can i see what is taking so long ?
Thanks !
Ok, this is an old question, anyway I find it searching for other things so quick adding this
URLFileVariables nor URLRequestBuilder are core classes in AS3, so I guess you're using some custom library to build your request. I don't know which library you use, but it seems that the purpose is to serialize some binary data to build a POST. Serializing usually takes some times the first time (lookup initialization and the like) and goes faster next, a well known example is Remoting in his different flavours

Resources