When using a rmm pool, is it possible to query how much of the pool is occupied ?
Yes as of https://github.com/rapidsai/rmm/pull/222;
rmm.get_info()
MemoryInfo(free=6867517440, total=34058272768)
Related
It's sad that I found it's not allowed by ceph cli to decrease the value of pg_num for a specific pool.
ceph osd pool set .rgw.root pg_num 32
The error is shown:
Error EEXIST: specified pg_num 32 <= current 128
The tutorial from placement-groups is about to tell me what is it and how to set the best value of it. But there is seldom any tutorial about how to reduce the pg_num without re-installing ceph or delete the pool firstly, like ceph-reduce-the-pg-number-on-a-pool.
The existed SO thread ceph-too-many-pgs-per-osd shows us how to decide the best value. If I met the issue, how can I recover from the mess?
If it's not easy to reduce the value pg_num, what's the story behind it? Why doesn't ceph expose the interface to reduce it?
Nautilus version allows pg_num changes without restrictions (and pg_autoscale).
If you want to increase/reduce pg_num/pgp_num values without having to create, copy & rename pools (as suggested on your link), the best option is to upgrade to Nautilus.
Is it possible to use powershell to detect an IIS worker process thread deadlock and then trigger an Application Pool recycle?
The Reason: I have a website that calls an assembly in the GAC that sometimes just holds the thread in deadlock and the only fix is to recycle the Application Pool of the website in IIS, the annoying thing is that a user has to notice it before I am told, it would be good to automate this process.
This is how I would do it. First we have to find a way to detect if the thread is deadlocked. I am guessing that you can probably do that with something as simple as a ping. Then we have to Recycle the App Pool.
# Load IIS module:
Import-Module WebAdministration
# Set a name of the site we want to recycle the pool for:
$site = "Default Web Site"
# Get pool name by the site name:
$pool = (Get-Item "IIS:\Sites\$site"| Select-Object applicationPool).applicationPool
#Check to see if the site is deadlocked.
if(!(Test-Connection -ComputerName Server -Quiet))
{
#Site is Deadlocked. Recycle the application pool:
Restart-WebAppPool $pool
}
below some of the counter values while doing load testing using 250 users:
> Gen2 heap size : 1124196
> #bytes in all heaps : 2172104
> #GC Handles: 926
> # of pinned objects: 11 Large Object Heap size: 87128
> # total commited bytes: 3350528
> # total reserved bytes: 33546240
they were increasing, increasing till they reach that limit.
after the test finished, the memory shown in task manager for w3wp.exe is not releasing until an IIS Reset is applied.
also the application is not accessible till an IIS Reset is applied (getting com+ activation failed)
Anyone, had benn in that situation before?
Thanks
Yes, you need a tool like ANTS Memory Profiler. Aside from that, make sure you are closing MemoryStreams, anything IO related, SqlConnections...etc. Try to us using statements on anything that implements IDisposable. Check for static references to objects tied to your Page instances.
I have an ASP.NET MVC website that gets about 6500 hits a day, on a shared hosting platform at Server Intellect. I keep seeing app restarts in the logs and I cannot figure out why.
I've read Scott Gu's article here: http://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx
and implemented the technique, and here's what shows up in my log:
Application Shutdown:
_shutDownMessage=HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
_shutDownStack=at
System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at
System.Environment.get_StackTrace() at
System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at
System.Web.Hosting.HostingEnvironment.InitiateShutdown() at
System.Web.Hosting.PipelineRuntime.StopProcessing()
It seems to occur about every five minutes.
Are there any other ways to debug this?
UPDATE: Here are the application pool settings mentioned by Softion:
CPU
Limit : 0
Limit Action : no action
Limit Interval : 5 Minutes
Process Model
Idle Timeout : 20 Minutes
Ping Maximum Response Time : 90 Seconds
Startup Time Limit : 90 Seconds
Rapid-Fail Protection
Enabled : True
Failure Interval : 5 Minutes
Recycling
Private Memory Limit : 100 MB
Regular Time Interval : 1740 Minutes (29 Hours)
Request Limit : 0
Specific Times : none
Virtual Memory Limit : 0
You can easily grab the reason of the shutdown by HostingEnvironment.
You read Scott Gu article, but you missed its comments.
var shutdownReason = HostingEnvironment.ShutdownReason;
If the reason is HostingEnvironment, check the IIS application pool parameters controlling recycling. I've put a red dot near each one. Check the description in the bottom help box in your own copy for full info.
You can ask your provider to give you the applicationHost.config file where all these parameters are set. They find it in C:\Windows\System32\inetsrv\config. I'm sure you can also get them using some .NET api.
For 6500 hits a day, which is a very low hit rate, i'm betting the "Idle time-out" is set to 5mn.
Update (moved comments to here //jgauffin)
CPU Limit 0 = disabled.
Process Model Idle Timeout : 20 Minutes (20mn without a request recycles your app).
Rapid-Fail Protection enabled (5mn). You need to know the maximum failures count. If your app throws more than this exception count in 5mn it will we recycled.
Private Memory Limit : 100 MB. Yes you should profile, this is a low limit.
Regular Time Interval : 1740 Minutes (29 Hours): it will recycle every 29h.
Request Limit : 0 (disabled).
Virtual Memory Limit : 0 (disabled).
Rapid-Fail Protection enabled (5mn). You need the maximum failures count. If your app throws more than this exception count in 5mn it recycles. If it recycles every 5mn this should be the thing to check. There should be 0 unhandled exception in secondary worker threads. Wrap your code into a try catch there.
re update:
The settings asked to the provider help, but is way better to ask for information on the reason of the restarts like I mentioned on my original answer i.e. the actual log entries of the restarts like I mentioned on my orig answer. From those you can know specifically what was triggered, I've seen happen one hitting different limits.
You really have to:
profile your application with a
realistic amount of test data
My money is on hitting resource limits set by your hosting provider.
Before going crazy with optimization without a target, contact your provider and ask them to give you information on the restarts.
Typical recycles:
idle x amount of time / like 15 mins
more than x amount of memory / like 200 MB
more than x % processor over y time / like 70 over 1 minute
a daily recycle
Once you know the case, you have to find out what's taking those resources. For this you have to profile your application with a realistic amount of test data. Knowing if it is memory or processor can help on knowing what to look for.
Is IIS set to recycle the app pool frequently?
Is there some kind of runaway memory leak in the app pool?
It requires a bit of know how on what your app does here's a list of things that can cause the app to restart/reset or even shut down
StackOverflowException
OutOfMemoryException
Any unhandled exception that crashes a thread
CodeContracts use Environment.FailFast when a contract violation occurs
Exceptions are quite easy to track if you can reproduce the issue with a debugger attached you can go into Visual Studio and enable all exceptions when they are thrown not caught by user code. It will sometimes reveal intresting stuff that otherwise is hidden away.
Greetings,
I have a large .Net web app which runs on a farm of blades with the code base on a NAS. Every once in a while slight fluctuations in the response time of the nas cause .NET to think that something in the bin has changed and kick off a recycle of the app pool. No change has actually occurred. Is there a way to disable .Net's monitoring of changes to the bin?
This will put an end to it. Now you must do iisreset to recycle your app pool.
'This is supposed to turn off the monitoring for directory deletes
'See https://connect.microsoft.com/VisualStudio/feedback/Workaround.aspx?FeedbackID=240686
'This incurs the penalty of an IISRESET or manually restarting the containing AppPool after every upgrade.
Dim pi As PropertyInfo
Dim o As Object
Dim m As MethodInfo
pi = GetType(System.Web.HttpRuntime).GetProperty("FileChangesMonitor", BindingFlags.NonPublic Or BindingFlags.Public Or BindingFlags.Static)
o = pi.GetValue(Nothing, Nothing)
m = o.GetType().GetMethod("Stop", BindingFlags.Instance Or BindingFlags.NonPublic)
m.Invoke(o, New Object() {})
There also seems to be a hot fix that appears to be able to disable the "feature" with a Registry entry:
http://support.microsoft.com/kb/911272
Not entirely sure that this works on the bin or not.
The simple answer is, you cannot. If IIS detects a change in one of the binaries, it must reload in order to ensure that it is using the correct binaries.
The real questions are:
a) Is IIS truly detecting a change, or is it due to the "fluctuation" causing the OS to think the files aren't there momemtarily?
b) If the "fluctuations" are the root of the issue, what is causing them? How can that be addressed?