I have an ASP.Net 3.5 website running in IIS 6 on Windows Server 2003 R2. It is a relatively small internal application that probably serves less than ten users at any given time. The server has 4 Gig of memory and shows that 3+ Gig is available while the site is active.
Just minutes after restarting the web application Performance monitor shows that there is a whopping 4,294,967,293 sessions active! I am fairly certain that this number is incorrect; at the time this reading there were only 100 requests to the website.
Has anyone else experienced this kind odd behavior from perf mon? Any ideas on how to get an accurate reading?
UPDATE: After running for about an hour the number of active sessions has dropped by 4. So it does seem to be responding to sessions timing out.
Could be an overflow, but my money's on an underflow. I think that the program started with 0 people, someone logged off, and then the number of sessions went negative.
Well, 2^32 = 4,294,967,296, so sounds like there's some kind of overflow occurring. Can't say exactly why.
We have the same problem. It looks like MS has a Hotfix available: http://support.microsoft.com/kb/969722
Update 9/10/2009: Our IT department contacted MS for the Hotfix. It fixed our issue. We are running .NET 2.0 if it matters any.
I am also showing a high number, currently 4,294,967,268.
Every time I abandon a Session, the Sessions abandoned count goes up by 1, and the Sessions Active count decreases by 1. Currently my abandoned session count = 16, so this number probably started at 4,294,967,84.
Is there a fix for this?
My counters were working fine, but one morning I logged in remotely to the production server, and the counter was on this huge number (which is as somebody mentioned very close to 2^32 indicating an underflow). But the only difference from the day before when everything worked was the fact that during the night, windows had installed updates.
So for some reason these updates caused this pretty annoying error.
Observing the counter a little more, I found out that whenever the application is restarted - after some time with no traffic, the counter starts correctly at zero. When users start logging on, it increments fine. When they start logging off again, it still decrements fine until it reaches what is supposed to be zero. At that point it goes bananas...
Sigh...
If you have to use your existing statistics, I opened the log file in Excel and used a formula to bring a more accurate value. I cannot guarantee its accuracy, but the results did look okay:
If B2 is the (aspnet_wp)\Sessions Active value , and the formula sits in C2
/* This one is quicker as it doesn't have to do the extra calculations */
=IF(B2>1073741824,4294967296-B2,B2)
Or
/* This one is clearer what is going on */
=IF(B2>power(2,30),(4*power(2,30))-B2,B2)
P.S. (I feel your pain - I have to explain why they have 4.2 billion sessions opening whereas a second earlier they had 0!)
Related
The web site that I have run for long time, sometimes it will have some speed issues, but after we clean up the MSSQL data, it will work fine.
But this time, it doesn't work any more, we always got Timeout error, and the IIS causes the CPU runs very high.
We took out some features, and the site runs back OK, but slow without error.
For example, when we do a search, if we have less than 10 results, the page/output is really fast.
When we have more than 200 results, the page is very slow, almost take about 15 to 20 secs to output the whole page
I know if you have more data to output, of course it takes more time to run, but we used to have more than 500 results, it ran/output very fast also.
Do you know anywhere I should look at to solve this speed problem?
You need to look at the code to see what is executed with displaying those results. Implement some logging or step through the execution of a result with a debugger.
If you have a source control system, now is the time to review what changes have been made between fast code and now slow code.
10 results could be taking 1 second to display which is barely tolerable but as you say 200 results takes 20 seconds. So the problem is some bad code somewhere. I think someone has made a code change.
I would start with breaking down the issue. For example sql server times and iis times. You can separate different parts of code and measure execution times, etc.
SQL Server Profiler is good tool to start with and for ASP.NET You can start with simple trace and page tracing.
Some more info about testing and performance
I'm using JMeter for load testing. I'm going through and exercise of finding the max number of concurrent threads (users) that our webserver can handle by simply increasing the # of threads in my distributed JMeter test case, and firing off the test.
Then -- it struck me, that while the MAX number may be useful, the REAL number of users that my website actually handles on average is the number I need to make the test fruitful.
Here are a few pieces of information about our setup:
This is a mixed .NET/Classic ASP site. Upon login, a browser session (with timeout) is created in both for the users.
Each session times out after 60 minutes.
Is there a way using this information, IIS logs, performance counters, and/or some calculation that will help me determine the average # of concurrent users we handle on our production site?
You might use logparser with the QUANTIZE function to determine the peak number of requests over a suitable interval.
For a 10 second window, it would be something like:
logparser "select quantize(to_localtime(to_timestamp(date,time)), 10) as Qnt,
count(*) as Hits from yourLogFile.log group by Qnt order by Hits desc"
The reported counts won't be exactly the same as threads or users, but they should help get you pointed in the right direction.
The best way to do exact counts is probably with performance counters, but I'm not sure any of the standard ones works like you would want -- you'd probably need to create a custom counter.
I can see a couple options here.
Use Performance Monitor to get the current numbers or have it log all day and get an average. ASP.NET has a Requests Current counter. According to this page Classic ASP also has a Requests current, but I've never used it myself.
Run the IIS logs through Log Parser to get the total number of requests and how long each took. I'm thinking that if you know how many requests come in each hour and how long each took, you can get an average of how many were running concurrently.
Also, keep in mind that concurrent users isn't quite the same as concurrent threads on the server. For one, multiple threads will be active per user while content like images is being downloaded. And after that the user will be on the page for a few minutes while the server is idle.
My suggestion is that you define the stop conditions first, such as
Maximum CPU utilization
Maximum memory usage
Maximum response time for requests
Other key parameters you like
It is really subjective to choose the parameters and I personally cannot provide much experience on that.
Secondly you can see whether performance counters or IIS logs can map to the parameters. Then you set up proper mappings.
Thirdly you can start testing by simulating N users (threads) and see whether the stop conditions hit. If not hit, you can go to a higher number. If hit, you can use a smaller number. Recursively you will find a rough number.
However, that never means your web site in real world can take so many users. No simulation so far can cover all the edge cases.
I have a thoroughly complex kernel processing audio input data. It will run for a couple of minutes, 60 times a second, and then hang. That's on the GPU; on the CPU it will run for hours. The input data are constantly changing, but each variable is always within proscribed ranges. I have inserted test code before uploading the inputs to the kernel each frame; in this test code, I can force these inputs to be well below their valid input range, but it still will eventually crash. (Say the valid range for a particular input is 0->400; I can force it to 0->1 and it will STILL eventually crash. I can force it to be below 0.1 and it will still ultimately bite the dust.) However, if I force the input variables to zero, the GPU will happily dance for hours. Of course, that input-free dance is not so particularly interesting.
I'm at a loss so far, though I have clues. I can make it crash much faster than 2 minutes if an input variable is high in its approved range. I can make it crash in less then 10 seconds under the right circumstances. BUT, I can't seem to _back_off_of_ those certain circumstances such that they go away. As said above, I can force the input vars into ridiculously small portions of their valid range, and the kernel (let's call him Harlan Sanders) will eventually go belly-up. BUT, if they're forced to actual zero, no problems puppy, we can run all day long.
To repeat, I'm a bit at a loss - although I have things that look like clues, I have not yet figured out what they are hinting at, though I've been trying for a few days. Frankly, I do not expect to find a real solution by asking here; whenever I stumble over a problem in opencl it seems that my fate is to be the first to articulate that particular problem. I guess this is part of the fun of being in on a technology during its infancy!!!!!!!!!! BUT, I want to do some serious, sustainable work with this "baby" (or, maybe, "toddler").
Op details: MacBook Pro 2010, OS 10.6.8, nv 330M GPU, xcode 3.2.5, shorts, teeshirt.
bonus P.S. for those who've read this far, including a related question:
My laptop, soldier that it has proved to be, is not powerful enough for the next stage. I must sell some stocks/bonds and purchase a Mac Pro. I'm looking at the ATI 5870. So, PERHAPS my problem will simply go away when I compile the .cl for the ATI??? Maybe I have run into a bug in the nV implementation. Maybe my kernel is so complex that I'm running into undetected resource limits (it's 1300 lines of code). So, SINCE I run fine on the CPU, perhaps I'll have no bugs, or different bugs, on the ATI card???
Any thoughts?
Thanks, guys & dolls --
Dave
Use "cl_" data types on the CPU side, because maybe you are not coping data the right way, or it is not being understood by the GPU. This could lead to GPU hangs on invalid pointers while handing the data.
You should also try -Werror, and read the error output. You can be doing smt wrong.
Without any code, we can only guess. But I haven't found any bug in the actual OpenCL NV or ATI implementations.
Make sure you release all resources. Events returned by Enqueue functions must be released. This error sometimes occurs after accessing buffers out of range.
Environment.WorkingSet incorrectly reports the memory usage for a web site that runs on Windows 2003 Server.(OS Vers: Microsoft Windows NT 5.2.3790 Service Pack 2, .NET Vers: 2.0.50727.3607)
It reports memory as Working Set(Physical Mem.): 1952 MB (2047468061).
Same web site runs locally on Windows Vista with a Working Set(Physical Mem.): 49 MB (51924992).
I have limited access to the server and support is so limited :(.
so i have computed the total memory by traversing with VirtualQuery.
Total of pages with state: MEM_FREE is 1300 MB.
(I guess server have 4 GBs of RAM and PAE is not enabled, max user mode virtual address is 0x7fff0000.)
So, i know working set is not only about virtual memory. But, is it normal to have such a high working set while its very low on another machine?
I think the problem is related to what is described in this article:
MAY 04, 2005
Fun with the WorkingSet and int32
I finally found an honest to goodness bug in the .NET framework.
... the
WorkingSet returns the amount of memory being used by the process as
an integer (32 bit signed integer). OK, so the maximum value of an
integer is 2,147,483,647 -- which is remarkably close to the total
amount of memory that a process can have in its working set.
... There is actually a switch in Windows that will allow a process to use
3 gig of memory instead of 2 gig. This switch is often turned on when
dealing with Analysis Services -- this thing can be a memory hog. So
now what happens is that when I poll the WorkingSet I get a negative
number, a really big negative number. Usually, in the realm of
-2,147,482,342.
... The problem was the overflow bit.
Working set is returned to the .NET framework as a binary value. The
first bit of an integer is the sign bit. 0 is positive, 1 is negative.
So, when the value turned from (binary)
1111111111111111111111111111111 to (binary)
10000000000000000000000000000000 the value goes from 2147483647 to
-2147483647.
OK, so I still have to fix this. Here is what I came up with (in C#):
long lWorkingSet = 0;
if (process.WorkingSet >= 0)
lWorkingSet = processWorkingSet;
else
lWorkingSet = ((long)int.MaxValue*2)+process.WorkingSet;
Hopefully that fixes the problem for now.
The real question will come in down the road. Microsoft knows about
this problem. I still have find out how they are going to fix this for
Win64...where this trick will no longer work.
http://msdn2.microsoft.com/library/0aayt1d0(en-us,vs.80).aspx:
There's gonna be a Process.WorkingSet64 variable, and they're
deprecating WorkingSet.
On a tangent, though, I thought it was impossible for a managed
process to come near the 3gb limit, because the runtime splits the
memory into multiple heaps. Is this not true?
At a guess, Environment.WorkingSet is probably returning the value from GetProcessWorkingSetSize, which is basically what has been set with SetProcessWorkingSetSize. It's basically whatever the system has picked as the largest working set size it would like to see for this process, not necessarily anything to do with how much memory it's actually using. The basic effect is that when/if the process uses more memory than that, the system's working set trimmer goes to work seeing if it can get some of its memory paged out to disk.
HI
We am getting time outs in our asp.net application. We are using sql server 2005 as the DB.
The queries run very fast in the query analyser . However when we check the time through the profiler it shows a time that is many times more than what we get in query analyser.
(paramter sinffing is not the cause)
Any help is much appreciated
thanks
We are on a SAN
Cleared the counters. The new counters are
ASYNC_NETWORK_IO 540 9812 375 78
WRITELOG 70 1828 328 0
The timeout happens only on a particular SP which a particular set of params. if we change the params and access the app it works fine. We ran the profiler and found that the SP batchcompleted statement comes up in the profiler after the timeout happens on asp.net side. If we restart the server everything works fine
if we remove the plan from the cache the app works fine. However we have taken into consideration parameter sniffing in the sp. what else could be the reason
If I was to take a guess, I would assume that the background database load from the webserver is elevating locks and causing the whole thing to slow down. Then you take a large-ish query and run it and that causes lock (and resource) contension.
I see this ALL THE TIME with companies complaining of performance problems with their client-server applications when going from one SQL server to a cluster. In the web-world, we get those issues much earlier.
The solution (most times) to lock issues with one of the following:
* Refactor your queries to work better (storing SCOPE_IDENTITY instead of calling it 5 times for example)
* Use the NO LOCK statement everywhere it makes sense.
EDIT:
Also, try viewing the server with the new 2008 SQL Management Studio 'Activity Monitor'. You can find it by right-clicking on your server and selecting 'Activity Monitor'.
Go to the Processes section and look at how many processes are 'waiting'. Your wait time should be near-0. If you see alot of stuff under 'Wait Type', post a screen shot and I can give you an idea of what the next step is.
Go to the Resource Waits section and see what the numbers look like there. Your waiters should always be near-0.
And 'Recent Expensive Queries' is awesome to look at to find out what you can do to improve your general performance.
Edit #2:
How much slower is it? Your SAN seems to be taking up about 10 seconds worth, but if you are talking 20 seconds vs. 360 seconds, then that would not be relevent, and there is no waits for locks, so I guess I am drawing a blank. If the differene is between 1 second and 10 seconds then it seems to be network related.
Run the following script to create this stored proc:
CREATE PROC [dbo].[dba_SearchCachedPlans]
#StringToSearchFor VARCHAR(255)
AS
/*----------------------------------------------------------------------
Purpose: Inspects cached plans for a given string.
------------------------------------------------------------------------
Parameters: #StringToSearchFor - string to search for e.g. '%<MissingIndexes>%'.
Revision History:
03/06/2008 Ian_Stirk#yahoo.com Initial version
Example Usage:
1. exec dba_SearchCachedPlans '%<MissingIndexes>%'
2. exec dba_SearchCachedPlans '%<ColumnsWithNoStatistics>%'
3. exec dba_SearchCachedPlans '%<TableScan%'
4. exec dba_SearchCachedPlans '%CREATE PROC%MessageWrite%'
-----------------------------------------------------------------------*/
BEGIN
-- Do not lock anything, and do not get held up by any locks.
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
SELECT TOP 100
st.TEXT AS [SQL],
cp.cacheobjtype,
cp.objtype,
DB_NAME(st.dbid) AS [DatabaseName],
cp.usecounts AS [Plan usage],
qp.query_plan
FROM sys.dm_exec_cached_plans cp
CROSS APPLY sys.dm_exec_sql_text(cp.plan_handle) st
CROSS APPLY sys.dm_exec_query_plan(cp.plan_handle) qp
WHERE CAST(qp.query_plan AS NVARCHAR(MAX)) LIKE #StringToSearchFor
ORDER BY cp.usecounts DESC
END
Then execute:
exec dba_SearchCachedPlans '%<MissingIndexes>%'
And see if you are missing any recommended indexes.
When SQL server creates a plan it saves it, along with any recommended indexes. Just click on the query_plan column text to show you the graph. On the top there will be recommended indexes you should implement.
I don't have the answer for you, because I'm not a guru. But I do remember reading on some SQL blogs recently that SQL 2008 has some extra things you can add to the query/stored procedure so it calculates things differently. I think one thing you could try searching for is called 'hints'. Also, how SQL uses the current 'statistics' makes a difference too. Look that up. And how the execution plan is only generated for the first run--if that plan doesn't work with different parameter values because there would be a vast difference in what would be searched/returned, it can present this behavior I think.
Sorry I can't be more helpful. I'm just getting my feet wet with SQL Server performance at this level. I bet if you asked someone like Brent Ozar he could point you in the right direction.
I've had this exact same issue a couple of times before. It seemed to happen to me when a particular user was on the site when it was deployed. When that user would run certain stored procedures with their ID it would timeout. When others would run it, or I would run it from the DB, it would run in no time. We had our DBA's watch everything they could and they never had an answer. In the end, everything was fixed whenever I re-deployed the site and the user was not already logged in.
I've had similar issues and with my case it had to do with the SP recompiling. Specifically it was my use of temp tables vs table variables.