After mutiple days of searching Google, I still have the following issue:
I have an internal website, which is hosted on Win2012 server, and uses ODP.NET to connect to an oracle db.
The issue is that no matter what i do, no matter the IIS configuration or server load, the TTFB for loading ceratain pages is huge - more than 20s.
When the same site is hosted on a different host (test enviroment), the TTFB is <1s.
System setup - the productive server is win2012 server, the test sever is identical. Identical - same OS, same patch level ,same services, same setup.
Both machines are virtual, with the productive having 24GB RAM and 2xXeon 2.2Ghz
So far i have conducted the following test - changed the oracle host in tnsnames to the host IP - no effect. Stopped all services running on the server - no effect. System load does not at any time exceed 50% CPU time , 45% memory, 4Mbps / out of a 2x100Mbps. Disk I/O operations are acceptable - no more than 5MB/s at a maximum. ODP.NET trace shows gaps of
Slight improvement was noticed when the number of worker processes was increated from 1 do 3 , but he web site requires a repeat of the log on after the server changes the WP which currently handles the user.
In short- from server A the site loads a page for 25-30s, from server B (identical) the page loads in 1-2s.
My request is : would you please recommend where should i am my efforts, or is there a windows based tool which would allow me to break down the TTFB to a more detailed view than the one offered by chrome/IE.
Thank you kindly for your responses.
Start by enabling logging and see if there is anything that stands out:
<configuration>
<oracle.dataaccess.client>
<add key="TraceFileName" value="c:\temp\odpnet1.trc"/>
<add key="TraceLevel" value="63"/>
<add key="TraceOption" value="0"/>
</oracle.dataaccess.client>
The "certain pages" part throws me off a bit. Your test server is connected to the same database, yes? The test server consistently has better performance? In other words, you are sure this isn't related to query performance? Does the test server go thru the same networks to get to the server (tracert oracleserver)?
Related
I have 3 machines - one which is IIS, one with a database and one from which I test the efficiency of my application - which means:
Using The Grinder I run 1000 instances of my application (hosted on the IIS and operating with the database on the machine with SQL Server). And using perfmon I observe that there really are 1000 requests.
BUT the problem is that connecting to this application (IIS) from another computer is very slow. I suppose there is some bottleneck on the IIS side but I cannot find it - CPU usage is less than 10%.
I think I changed every option in the IIS Manager and machine.config and web.config files - nothing seems to have any effect.
First thing is you need to confirm if you have a slowness issue while browsing the site
Check the IIS logs and look for the time-taken field. If the time taken is more than 10 seconds then it is considered as a Slowness.
The slowness might be because of several reasons. It might be because of the Network or might be because something in your code might be causing it.
I suggest you to capture a Network trace using Netmon or WireShark in case if its a Network related.
If its not Network you can collect a Process dump using Debug diag 2 update 2 tool.
You can check the below link to collect the dumps and check them and try to find out if there is any slowness:
https://msdn.microsoft.com/en-us/library/ff420662.aspx
Background: We are trying to migrate a large, complex web application written in classic ASP from Windows Server 2003 to Windows Server 2012 R2.
Everything is working without errors, but the new server is extremely slow to serve the ASP pages. With a single user on the site, response times in the order of 2-3 seconds for ASP pages are common. Equally large AJAX calls and JavaScript pages are served and process in under 100ms.
When the site receives a moderate level of load (more than approx. 50 users) it becomes unusably slow. Normal load for the the production site is several thousand users.
There does not appear to be a correlation between the amount of data returned or the database connection. We are using SQL Server 2008 R2 for the database.
The Web application server is in a DMZ and uses a hosts file entry for the database server which is in our general intranet. Database queries process extremely fast (within milliseconds).
I've tried profiling the web server memory usage, disk I/O and network usage, and found no evidence of memory leaks. Query profiling shows no lag in processing database calls.
Update after running Failed Request Tracing
I set up tracing to be triggered for classic ASP requests taking longer than 1 second
Maximum time shown by detail logs for each trace from request start to request completion: 140ms
Total request times logged ranged from 1094ms to 1453ms - so the actual request is taking an order of magnitude longer than the events logged by the failed request trace.
What are common fixes for this performance problem?
There are reports of classic ASP sites being slow if the connection string uses the machine name\instance name instead of an IP address, especially if SQL is running on a non-standard port. Maybe try changing the connection string, e.g.:
Server=10.10.10.123,1433\myInstanceName;
Reference: forums.iis.net
I am unable to comment since I do not have enough reputation, so asking question as an answer. I will remove this once I get the answer
What is the driver you are using to connect in your connection string ?
I did see your comment on host file, can you please try direct IP in the connection string. Please do not remove the host file.
Can you try a small new web application in asp just with minimal database listing. is that also slow?
Again try the same new application without a database connection and time the difference.
Do you have on error resume next in the code, are you failing on any file / log permission but that is not getting reported.
Try disabling Microsoft Defender. There are serious slowdowns after an update. They modified vbscript.dll which is what executes ASP code.
If you cannot live without Windows Defender, you can replace vbscript.dll with an older version.
Last night one of the websites (.NET 4.0 forms) hosted on my Win 2008 R2 (IIS 7.5) Server started to time out throwing the following error for all connected users.
TYPE System.Web.HttpException
MESSAGE Request timed out.
DETAIL System.Web.HttpException (0x80004005): Request timed out.
The outage was confined to just one website within IIS, the others continued to work fine.
Unfortunately I was unable to identify why the website was timing out. Here are the steps I took:
First thing I did was look at the task manager which revealed normal CPU and memory usage. Network activity was also moderate.
I then opened IIS to look at the live connections under 'Worker Processes'. There were about 60 live connections, so it didn't look like anything DDoS related.
Checked database connectivity (hosted on a separate server), all fine!
I then reset the website on IIS. That didn't work
I tried to then do a complete iisreset...still no luck :(
In the end (and under some duress) the only thing I could think to do to resolve this was to restart the server.
Restarting the server worked but I am nervous not knowing why this happened in the first place. Can anyone recommend any checks that I failed to carryout? Is there an official checklist for working through these sorts of IIS problems? I have reviewed the IIS logs but don't see anything unusual on the run up to the outage.
Any pointers or links to useful resources to help me understand and mitigate against this in future will be much appreciated.
EDIT
The only time I logged into the server that day was to add an additional web handler component (for remote deploy) to IIS Web Deploy. I'm doubtful this caused the outage as the server worked for for 6 hours after.
Because iisreset didn't helped and you had to restart whole machine, I would suspect it was a global resources shortage and mostly used website (or most resource consuming) was impacted. It could be because of not available RAM, network connections congestion due to some malfunctioning calls (for example a lot of CLOSE_WAIT sockets exhausting connections pool, we've seen that in production because of malfunction of external service). It could be also one specific client problem, which was disconnected after machine restart so eventually the problem disappeared.
I would start from:
Historical analysis
review Event Viewer to see any errors/warnings from that period of time,
although you have already looked into IIS logs, I would do it once again with help of Log Parser Lizard to make some statistics like number of request per client, network bandwith per client, average response time per client and so on.
Monitoring
continuously monitor Performance Counters:
\Processor(_Total_)\% Processor Time,
\.NET CLR Exceptions(_Global_)\# of Exceps Thrown / sec,
\Memory\Available MBytes,
\Web Service(Default Web Site)\Current Connections (per each your site name),
\ASP.NET v4.0.30319\Request Wait Time,
\ASP.NET v4.0.30319\Requests Current,
\ASP.NET v4.0.30319\Request Queued,
\Process(XXX)\Working Set,
\Process(XXX)\% Processor Time (XXX per each w3wp process),
\Network Interface(XXX)\Bytes total / sec
run Performance Analysis of Logs (PAL) Tool during time of failure to make a very detailed analysis of performance counters data,
run netstat -ano to analyze network traffic (or TCPView tool even better)
If all this will not lead you to any conclusion, create a Debug Diagnostic rule to create a memory dump of the process for long running requests and analyze it with WinDbg and PSSCor extension for .NET debugging.
I have a w3wp.exe that is restarting on my IIS server (see specs below). Memory gradually climbs to ~3G then it randomly restarts itself about every 1-2min.
Memory Usage:
The odd thing is that once this memory drop (what looks like a restart - btw...the app pool does not get recycled/restarted) happens GET requests are queued but then serviced as soon as the service warms/starts up (causing a delay in responses to our clients - who were initially reporting delayed reponse times on occasion).
I have followed this link to get a stack dump once the .exe restarts (private bytes go to ~0) but nothing gets logged (no .dmp file) with diag debug once the service restarts.
I see tons of warnings in my webserver (IIS) log but that's it:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
ASK: I'm not sure if this is a memory limitation, if cacheing is not playing well with my threads/tasks, if cacheing is blowing up, if there is watchdog service restarting my application, etc,. Has anybody run across something similar with w3wp.exe restarting? It's hard to tell because diagdebug is not giving me a dump once it restarts.
SPECS:
MVC4 Web API servicing GET requests (code is debug build with debug=true)
Uses MemoryCache with Model and Business Objects with cache eviction set to 2hrs...uses
Task (TPS) for each new request.
Database: SQL Server 2008R2
Web Servers: Windows Server 2008R2 Enterprise SP1 (64bit, 64G RAM)
IIS 7.5
One application pool...no other LOB applications running on this server
Your first step is to reproduce the problem in your test environment. Setup some kind of load generation app (you can write it yourself pretty easily) and get the same problem happening. Then turn off debug in web.config and see if that fixes the issue. Then change it to be a release build and test again.
I've never used memorycache - try reducing the cache eviction time or just turn it off and see if that fixes the issue. Good luck :)
My environment: Windows Server 2008, IIE 7.0, ASP.NET
I developed a Silverlight client.
This client gets updates from the ASP.NET host through a WCF web service.
We get 100% CPU usage and connection drops when we have a very low number of users (~50).
The server should clearly be able to handle a lot more than that.
I ran some tests on our DEV server and indeed 100 requests / s maxes out the CPU.
What's odd is that even if the service is replaced by a dummy sending back hardcoded data the service still maxes out the CPU.
The thread count looked very low at about 20 so I thought there was some contention somewhere.
I changed all configuration options I could find to increase the worker threads (processModel, httpRuntime and the MaxRequestsPerCPU registry entry).
Nothing changed.
Then I stopped the IIS server and ran the web service as a console (removing all the ASP authentication references).
The service maxed out the CPU as well.
Then comes the magic part: I killed the console app and restarted IIS and now the service runs a 5-60% CPU with 100 requests / s and I can see 50+ worker threads.
I did the same thing on our preprod machine and had the same magic effect.
Rebooting the machines keeps the good behaviour.
So my question is: what happened to fix my IIS server? I really can't understand what fixed it.
Cheers.
Find out the root cause of the high CPU usage, and then you can find a fix,
http://support.microsoft.com/kb/919791