Background: We are trying to migrate a large, complex web application written in classic ASP from Windows Server 2003 to Windows Server 2012 R2.
Everything is working without errors, but the new server is extremely slow to serve the ASP pages. With a single user on the site, response times in the order of 2-3 seconds for ASP pages are common. Equally large AJAX calls and JavaScript pages are served and process in under 100ms.
When the site receives a moderate level of load (more than approx. 50 users) it becomes unusably slow. Normal load for the the production site is several thousand users.
There does not appear to be a correlation between the amount of data returned or the database connection. We are using SQL Server 2008 R2 for the database.
The Web application server is in a DMZ and uses a hosts file entry for the database server which is in our general intranet. Database queries process extremely fast (within milliseconds).
I've tried profiling the web server memory usage, disk I/O and network usage, and found no evidence of memory leaks. Query profiling shows no lag in processing database calls.
Update after running Failed Request Tracing
I set up tracing to be triggered for classic ASP requests taking longer than 1 second
Maximum time shown by detail logs for each trace from request start to request completion: 140ms
Total request times logged ranged from 1094ms to 1453ms - so the actual request is taking an order of magnitude longer than the events logged by the failed request trace.
What are common fixes for this performance problem?
There are reports of classic ASP sites being slow if the connection string uses the machine name\instance name instead of an IP address, especially if SQL is running on a non-standard port. Maybe try changing the connection string, e.g.:
Server=10.10.10.123,1433\myInstanceName;
Reference: forums.iis.net
I am unable to comment since I do not have enough reputation, so asking question as an answer. I will remove this once I get the answer
What is the driver you are using to connect in your connection string ?
I did see your comment on host file, can you please try direct IP in the connection string. Please do not remove the host file.
Can you try a small new web application in asp just with minimal database listing. is that also slow?
Again try the same new application without a database connection and time the difference.
Do you have on error resume next in the code, are you failing on any file / log permission but that is not getting reported.
Try disabling Microsoft Defender. There are serious slowdowns after an update. They modified vbscript.dll which is what executes ASP code.
If you cannot live without Windows Defender, you can replace vbscript.dll with an older version.
Related
After mutiple days of searching Google, I still have the following issue:
I have an internal website, which is hosted on Win2012 server, and uses ODP.NET to connect to an oracle db.
The issue is that no matter what i do, no matter the IIS configuration or server load, the TTFB for loading ceratain pages is huge - more than 20s.
When the same site is hosted on a different host (test enviroment), the TTFB is <1s.
System setup - the productive server is win2012 server, the test sever is identical. Identical - same OS, same patch level ,same services, same setup.
Both machines are virtual, with the productive having 24GB RAM and 2xXeon 2.2Ghz
So far i have conducted the following test - changed the oracle host in tnsnames to the host IP - no effect. Stopped all services running on the server - no effect. System load does not at any time exceed 50% CPU time , 45% memory, 4Mbps / out of a 2x100Mbps. Disk I/O operations are acceptable - no more than 5MB/s at a maximum. ODP.NET trace shows gaps of
Slight improvement was noticed when the number of worker processes was increated from 1 do 3 , but he web site requires a repeat of the log on after the server changes the WP which currently handles the user.
In short- from server A the site loads a page for 25-30s, from server B (identical) the page loads in 1-2s.
My request is : would you please recommend where should i am my efforts, or is there a windows based tool which would allow me to break down the TTFB to a more detailed view than the one offered by chrome/IE.
Thank you kindly for your responses.
Start by enabling logging and see if there is anything that stands out:
<configuration>
<oracle.dataaccess.client>
<add key="TraceFileName" value="c:\temp\odpnet1.trc"/>
<add key="TraceLevel" value="63"/>
<add key="TraceOption" value="0"/>
</oracle.dataaccess.client>
The "certain pages" part throws me off a bit. Your test server is connected to the same database, yes? The test server consistently has better performance? In other words, you are sure this isn't related to query performance? Does the test server go thru the same networks to get to the server (tracert oracleserver)?
This question is related to a connection reset question I asked recently. I am also pursuing the info in this SO article and this SO article. What I want to know is the best way to track down the cause of an IIS Thread Pool crash. I am also pursuing this problem with our vendor. Here are the details.
I have an IIS application. It uses a very simple ASP page, modeled on a sample supplied by our vendor, when we purchased our first COM toolkit quite a few years ago. This application runs on IIS on Windows Server 2003. The application is a separate web service, not a virtual root, and it's at a different port number than 80. In other words, we're not using another NIC or virtual IP address. The site has little traffic, but was configured to have a DMZ. The whole configuration looks a bit funky to me.
Address verification requests are sent to the IIS application from browser-less applications that use http protocol. Street number, street name, city, and state are sent with a few other pieces of identification, and a response is sent back. One of the applications is written in C; the other is written in Clojure.
Sending "single-shot" requests to the newly built IIS application works just fine from both applications. Sending a lot of requests, and it's been hard to pin down whether that's over 25 or some other number, causes the IIS thread pool to crash. That's more or less what the error logs say.
This application has run without incident for years on a W2K/IIS server. The ASP page talks to an address verification engine by means of a COM toolkit that we purchased from the vendor. We had to upgrade to a new server (and 2012 COM toolkit) only because the newest COM toolkit would not install on W2K, and the new COM toolkit contained support for the new USPS address verification regulations. We'll validate fewer addresses without the new toolkit, and the only way you can barcode addresses and get that discount is to validate addresses. (When you send out 29,000 motor vehicle excise bills, the discount helps.)
What is the best way to track down this problem, so I can find the culprit? I'm looking for links to good information as much as answers. I am sorry to be vague; I know the rules of SO, and endeavor to provide as much detail as possible. I can re-edit this post and provide log entries, if someone wants to see those. The bottom line is my Clojure client (handles the batch requests) starts getting resets, and crashes, when the IIS thread pool crashes.
Epilogue:
We believe the rapid opening and closing of the COM object is the issue, and our writing the ASP page this way was sanctioned by our vendor. To get around this problem and the fact we also needed to install MS Access for additional address verification functions, we wound up writing an ActiveState Perl program to solve multiple needs.
First, we have purchased an ActiveState product that will the our Perl program run as a service. The service will start with parameters and a port number that allow the http-based requests to be sent, just like the were before. In this case the COM object will be opened once when the Perl program starts, and close when the Perl program exits.
The other functions provided by the Perl program are not germane to this post, but the writing of this Perl program obviates the need for IIS and MS Access, which were required with the old configuration.
So, if you want to take something from this post, it would be open a COM object on opening of a program and close on program close, at least for W2K Server 2003 and higher.
We had an issue at work with respect to application slowness. We did an application pool memory dump and then analysed the dump.
http://support.microsoft.com/kb/919790
I'm trying to run a trace with Sql Server Profiler against an Asp.NET Website Application running in Visual Studio development server.
However, whenever the trace is running, all db requests from the web application fails giving me the error message:
"Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding."
If I stop the trace, the web application works again.
Any input on this is appreciated.
You just need to increase the CommandTimeout on the sql connection while you are debugging and the application pool timeout values as well.
Once you get around the profiler timeout issue you should look at tuning your database (if you havent already, although it doesnt sound like it).
I have had a similar issue recently and it turned out to be IO blocking due to high reads on certain querys/statements. Getting the profiler to run on top of an already sluggish database was difficult. We had to run the profiler in ten minute sections at quieter times, although this does not help to identify the biggest issues with the heaviest loads.
Once we got the profiler to capture data (on sql server 2005) and implemented the indexes and statistics recommeneded by the Database Tuning Advisor (DTA) the database was running at expected peformance levels again.
I would recommend you read this free ebook on sql server profiler....
http://www.red-gate.com/products/SQL_Response/offers/mastering_sql_profiler_ebook.htm
It details how to run lightweight traces that will help the DTA recommend indexes and statistics that will improve the performance of your database and also identify some slow running queries that could be located in your code.
The trace you are running could be tipping your database over the edge, so running it in 10-20 minute sections might be more feasable.
If you have IO blocking issues this affects the overall sql server in general and management studio will seem non responsive at times.
Is it possible that you're accidentally stuck in single-user mode?
Try this:
ALTER DATABASE [database name] SET MULTI_USER;
my application is developed on classic asp, but also uses asp.net as I am migrating the application on .Net. Its using SQL server as database and hosted on Windows server 2003.
Now the problem is that the application continue to work perfectly fine for a long time but then after some time SQL server gives timeout error and it could fulfill any of the requests made. It doesn't get fixed even when I restart my SQL server or even IIS, ultimately I have to restart my server every time which only fixes the problem.
Any idea what might be causing the problem? Just to give a rought idea, the site is used by around 300 people at peak times.
Any idea what might be causing the problem? Just to give a rought idea, the site is used by around 300 people at peak times. I am certainely closing connection everywhere, my end code on each page closes the connection. If an error occurs before the end page, the expection handler closes the connection. So I am sure that closing the connection isn't an issue. And that there are no open connection if I see the sql logs. Our server, only one box, has SQL Server, IIS, iMail (our mail server). After I had restarted SQL Server, it did not solve the problem. Only restarting Windows Server, it worked. From perfom, IO usage is quite high. Is there any suggestions?
Thanks,
At the very least, are you closing the connection to the database, once you are done using it in the code? Also, how does your connection string look like? does it use connection pool?
EDIT: I saw your comments. Are there pending transactions to be committed?
It sounds a lot like there's an unmanaged resource of some kind that you aren't cleaning up properly. We don't have enough information to know exactly what that resource might be, so all we can do is guess.
My first instinct is database connections, except that restarting that restarting sql server should fix it if that were the case. Next on the list is file handles and threads, so if you do any multithreading work or extra file io that would be something to look at. Remember, in ASP.Net, the using statement (not directive) is your friend.
First, you need to talk to your DBA... they can check the number of open connections, table locks, slow-running queries, etc.
My gut reaction is that you aren't closing your connections somewhere, or your connection pool is too low.
Are you doing regular database maintenance? Rebuilding / defragmenting indexes, recalculating statistics (unless it's set to do this automatically). Check the size of your transaction log, etc.
I've got a number of ASP.Net websites (.Net v3.5) running on a server with a SQL 2000 database backend. For several months, I've been receiving seemingly random InvalidOperationExceptions with the message "Internal connection fatal error". Sometimes there's a few days in between, while other times there are multiple errors per day.
The exception is not limited to one site in particular, though they share business and data access assemblies. The error seems to always be thrown from SqlClient.TdsParser.Run(). It sometimes is thrown from old-school direct SqlCommand.Execute() calls, while other times it is thrown from Linq2Sql code.
I've been assured by the network guys that there are no errors or packets lost on their end. Has anyone else experienced this? Could it be a driver problem? We have been unable as of yet to pinpoint a specific trigger for this exception.
We're running II6 on Windows Server 2003.
After a few months of ignoring this issue, it started to reach a critical mass as traffic gradually increased. Under heavy load, including some crawlers, things got crazy and these errors poured in nonstop.
Through trial and error, we eventually tracked down a handful of SqlCommand or LINQ queries whose SqlConnection wasn't closed immediately after use. Instead, through some sloppy programming originating from a misunderstanding of LINQ connections, the DataContext objects were disposed (and connections closed) only at the end of a request rather than immediately.
Once we refactored these methods to immediately close the connection with a C# "using" block (freeing up that pool for the next request), we received no more errors. While we still don't know the underlying reason that a connection pool would get so mixed up, we were able to cease all errors of this type. This problem was resolved in conjunction with another similar error I posted, found here: Why is my SqlCommand returning a string when it should be an int?
Sounds like the database connection is getting dropped or timing out.
We recently had similar issues moving to IIS 6 from IIS 5 connecting to SQL 2000. Our issue was solved by increasing number of ephemeral ports available.
Look at the usage of the ephemeral ports by the IIS server. The default max no. of ports available is normally 4000. You might want to consider increasing this if the sites on your server are particularly busy or your application is making a lot of database calls.
You can monitor these first to see if going over max limit.
Search Microsoft Knowledge base for "MaxUserPort" and "TcpTimedWaitDelay" and make necessary registry changes. Make sure you back up registry or snapshot server before making the changes. Will need to reboot for changes to take effect.
You should double check your database and recordset connection are being closed after use. Not closing will use up this port range unnecessarily.
Check the efficiency of your stored procedures anyway as they might be taking longer than they need too.
"If you rapidly open and close 4000 sockets in less than four minutes, you will reach the default maximum setting for client anonymous ports, and new socket connection attempts fail until the existing set of TIME_WAIT sockets times out." - from http://support.microsoft.com/kb/328476
Check your server's LOG folder (\program files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG or similar) for files named SqlDump*.mdmp and SqlDump*.txt. If you do find any you'll have to take it to Product Support.
I was creating a new EF Core project and was trying to create the database to an external Linux server instead of a Windows Server or local one. After hours of searching I found out that I am using MySQL instead of the Microsoft SQL server.
I found it weird that everyone was using 1433 instead of the usual 3306. So to fix my 'Internal connection fatal error' I had to set up a docker instance of SQL Server bound to its default port of 1433.
It literally was that simple. In the docker repo look for "microsoft-mssql-server" and run the image as described neatly in the description below. Everything works now and I am able to push my database from my EF Core project to an external server.