I've got a Windows 2008 Enterprise R2 Server running Ektron 8.02 SP3 that is causing me some trouble that I can't diagnose.
So my question isn't for a solution but simply how to better profile a .net application / windows server. Whenever you try to POST a form built by the software, it takes 8 seconds and change to return the page (on this specific server, it doesn't do it on other machines with the same codebase). It appears that it's trying to make a connection to something for 8 seconds, fails, then returns the page without error. Some more facts:
This is a beefy VM that is not being over utilized
The database is running on the same machine, so there is no lag there.
The 8 second delay even happens when submitting from the server itself
The event viewer for the server doesn't report any errors that seem related
Profile MSSQL doesn't report any issues either
Microsoft Network Monitor doesn't report any glaring networking issues, though its hard to say as the tool doesn't report long connection attempts clearly (from what I've seen of it)
I feel sufficiently confident that there is some process during the form POST on the server that is trying to make a connection somewhere, failing, and continue through the rest of the process. It doesn't report any errors and the forms all submit fine.
So, all that being said, is there anything else I can do to debug this? I feel like I'm shooting in the dark. Thanks for any help you can offer.
It sounds like this isn't an application you wrote. If so, the potential solutions are different.
A first level step would be to use Windows' Performance Monitor. Select the options (process and/or performance attribute to watch), then run the process through its paces.
If Performance Monitor can't help, you'll need to get into the real guts of the app. Most likely this will be more difficult than you want unless you're writing the app. You can debug a .NET app that is not your own using Reflector (paid) or ILSpy (free). HOWEVER, this can violate the license agreement (and likely does if the app has been purchased).
Apart from that, you're going to need to contact the product developer if you want to both stay sane and find this problem quickly.
Related
Good day.
Background: ASP.NET MVC 3 / SignalR.
I'm developing typical chat component which will be used in a big portal. I wrote small emulator which opens new window with chat component and new SignalR connection in it.
The problem:
Everything works fine, but: Suddenly i noticed that if i open more than X windows/tabs with same component in one browser - server stops to receive any responses from any clients connected with it.
The X is:
5 for IE and Chrome
14 for FF.
I suppose it's a browser issue, because the value differs from one to another. Additionally i noticed that requests can stay pending for minutes! And there is no timeout..
Maybe someone faced with something like this, or know what can cause this strange behavior?
Thanx in advance, Jan.
I'm going to assume you're using IIS 7/7.5 on windows 7. If you are indeed using Windows7 then you need to use IIS Express since it only supports 10 concurrent connections (Client OS limit). Alternatively you can use Windows Server to get the full benefits of IIS.
This is the first post in three days of googling that came close to my experience. I am using the .NET client and have tried isolating the issue by creating an app with just the problem areas. The difference in our situation is this: I don't seem to have some connection limit on the client. In fact, I have no issues until I deploy my .Net hub to the windows server 2008 hyper vps. IIS express runs everything as smooth as butter. I've checked to make sure the asynchronous services and features are running and I've followed the performance tuning articles that david mentions in the documentation, adjusting the setting for thread queue maximums and the like. Note that the minute-two minute delays you describe are identical to mine. I updated to the latest version of SignalR (.4) from nuget and it broke my solution for some reason.
Can you tell me what version you're on? I suspect it could have something to do with security on the physical host where my vps is running. I am going to try it on a different virtualization platform to rule out hyper v issues.
Please, see my answer here: Is there a way to get number of connections in Signalr hub group?
The idea is to overwrite connectionid.
I have Visual Studio 2010 and a pretty large web application project running on IIS 7. Startup for the web application is over a minute (75 seconds). I've attached ANTS to it and very little of the 75 seconds is my code. Most of it seems to be something like CreateAppDomainWithHostingEnvironment and BuildManager stuff. Now I know that ASP .NET will compile dynamically the first time but I certainly don't expect it to compile for that long. Why could I be experiencing this problem and what are some ways I can try to fix it or try to better understand what is taking so much time. Aksi the CPU utilization doesn't seem to be that high. I have an awesome machine.
The problem with the 75 second startup is that for developers working on this, everytime they make a change they have to wait this 75 seconds.
I am using .NET 4.0
EDIT
I ran Microsoft Network Monitor on my machine to see if there was anything suspicious going on the network. There wasn't as far as I can tell though i wasn't sure what to look for (I am familiar with network monitor though so I did have an idea of what I was doing). I tried to run it in release build and though it may have improved the performance a little bit its not really significant
EDIT
I have SQL Session state. As far as i can tell, the connection string is pointing local. For some reasons though, when examinning ANTS, i'm getting a lot PollLockedSessionCallback on many threads. The function seems to be called over 70 times. Does this help at all?
Try building the application in release mode. You can set this in the Build tab of the properties window. You might also consider pre-compiling when publishing the application before deployment.
Are you trying to access anything via a network share at startup? If so, bring those resources local for startup comparison.
Sometimes a visual FoxPro App doesn't find files in a FileShare, which are there.
for example when checking in a loop File() on a existing file on a Network share about 5% of the tries don't find the file.
This works on most machines but sometimes it doesn't work. In the curren scenario I've a Windows Server 2K8 as file server (perhaps a SMB2 issue?)
I would patch your 2K8 server to SP1 (and any Windows 7 clients too), this will take care of any SMB2 issues. Those issues were around CDX index file corruption, though.
It's also possible that this is due to the caching that SMB2 uses, which can produce 'File Not Found' errors. The client registry settings involved are:
FileInfoCacheLifetime
FileNotFoundCacheLifetime
DirectoryCacheLifetime
There is a discussion regarding this on Alaska Software's website, and a useful MSI installer which can be run per workstation to adjust the settings. This company produces a product called Xbase++ but I would guess it is close enough to Visual FoxPro in terms of low-level file IO and locking.
Not positive if its an issue of Fox, or your network. Going way back in time, I had a client that had problems somewhat similar. Took Foxpro out of the equation and just used Windows Explorer and it would hang for a moment. It ended up that their network cards were set to energy saving mode and would basically time-out / shut down due to inactivity. The network drive share would apparently be released. Until the network card would re-connect and get established again, they had issues. By changing so the network card NEVER went into energy save mode, problem went away for them.
Yes. I have versions of fox pro deployed on various different servers, with various versions of windows server and never experienced an issue as described.
Maybe you could try a similar test using a different programming discipline, .Net , access, Ruby ...., etc
Post you test loop, just out of interest ?
Some background info:
We have several websites running on a 64-bit machine with IIS6
These websites all have the same core code, but different skins and content
We have a SQL 2005 database which is fairly heavily used throughout the site
Historically we've used SQL stored procs, but have been gradually transitioning to NHibernate. The majority of our code uses NHibernate now, but not all.
These sites have been running fine on our live web server for a while, although we get a few errors a day regarding SQL connectivity / deadlocking.
Last Thursday we noticed the sites going very slow, then checking task manager revealed one of the websites was hogging over 1.6Gb of memory. Ever since then we've been restarting the app and watching it slowly increase in size over the course of the day.
We apparently have a memory leak (or at least, that's the effect), but I'm losing hair trying to work out how to trace it.
It only appears to be happening on this one website, even though as far as I am aware nothing had changed in the code before it started happenning. It is, however, our busiest website so it could be a traffic issue.
Debug Diagnostics hasn't revealed any issues.
Refreshing certain pages very quickly causes the memory to jump up rapidly, then fall slightly, but all the time the gradual progression is upwards.
I cannot replicate the issue on our test servers or locally. Probably because the traffic has something to do with it.
My suspicion is that the problem lies in database connectivity / locking. However, I'm not sure how that would cause the problem specified.
Any ideas?
Edit
Okay so not exactly sure I've found the problem but we're getting closer. It's definately SQL related. The error log reveals lots of errors since last thursday.
It all happened after we ran some windows updates on our servers. One of the updates failed on the SQL server so not sure if this caused some problems.
The warnings we're getting are:
SQL Server has encountered XX occurence(s) of I/O requests taking longer than 15 seconds to complete on file .. tempdb.mdf
Where XX is anything between 17 and 90! Does that sound like a deadlocking issue?
Followed by the following erors:
Unable to complete login process due to delay in opening server connection
These coincide with our log times for when the websites have been "blipping".
We've increased the page file size on SQL server to the recommended size, as it was set to a max of 4Gb, but recommended was 12Gb. I think we may need to roll back the windows updates we did on Thursday if that doesn't fix it.
Unfortunately I can't get into Activity monitor as it tells me Timeout expired!
Edit
Okay after a reboot I'm into Activity monitor. How many sleeping processes would you say would be normal? We have roughly 127 sleeping. That's serving over 10 websites.
If there is a deadlock or timeout issue, will NHibernate not clean up its connections properly?
Okay so in the end it seems it's quite complex. Sql deadlocks and data problems, heightened it seems by anti-virus software that was locking up or choking on a file.
Turning off the anti-virus reduced the problems, but we still need to resolve the underlying data issues.
Anyone got tips for diagnosing SharePoint / ASP.Net "Request Timed Out" messages?
We've recently taken on the support and development of a client's MOSS public facing website. We've recreated a version of the site (a manual process - no Solution's here!) on 3 separate dev servers and are experiencing extremely slow warmup times. I'm used to waiting up to a minute after an IIS Reset but we are having to go through 2 Asp.Net "Request Timed Out" error messages. In general the site seems to be taking about 5 minutes to load up. Try doing custom development against that!
The strange thing is that on the staging and production servers the site takes about 40 seconds to warm up. They are slightly more powerful servers with a separate DB server but I wouldn't have thought the difference should be that great? I don't have any trouble with other SharePoint sites on my dev servers - just this one. It does contain a lot of custom code and DLLs so I understand that it may take a little longer to load these up but 5 minutes seems ridiculous.
The servers I'm testing this on are SharePoint 2007 (Feb CU), Win2003/IIS6, SQL 2005.
Does anyone have any tips for diagnosing the bottleneck here? I'm not sure if this is expected behaviour or a problem somewhere in the stack?
Cheers,
James.
Have you run any performance monitoring over the servers? This is essential for finding where the bottlenecks are. See here and here for recommendations.
If custom code has been deployed, check for an unusually high exception count or garbage collection/memory leak problems. This is most likely to be where the problem is. The best way to narrow this down is with a tool such as ANTS Profiler which will show memory leaks and performance issues. You could also Turn on ASP.NET tracing and set debug="true" in web.config and get some idea on slow executing code (although with all those timeouts this might not be so helpful).
Also do you know if any regular maintenance was performed on the SQL Server? (See some tips here.) Has SharePoint SP2 been installed (this performs some reindexing for you)?