We Just ran into a problem with our website. We are running a Oracle 12c database and using IIS 8 on a Windows 12 Azure server. ODP.NET is the provider we use. The website is build in C# and run as a 64bit web application.
For Whatever reason, Oracle doesn't release our connections, and after a while we got an Oracle Error :
ORA-12520: TNS:listener could not find available handler for requested type of server
We reviewed all of our connections string, adding :
Min Pool Size=10;Max Pool Size=100;Connection Lifetime=120;Connection Timeout=60;Incr Pool Size=5; Decr Pool Size=2
We more than double the MAX Process and Sessions that was defined in the Oracle configuration.
It's help but only in delaying the problem.
For now what we have done is a batch file which run every 10 minutes and kills all inactive sessions that are still there after 30 minutes (ALTER SESSION KILL....IMMEDIATE).
That's where we are, we have enable the ODP.NET trace and waiting to look at it tomorrow, but by this time, what can cause this problem?
Finally what I found in the ODP.NET Trace was
TID: dac (ERROR) Oracle error code=28002; Oracle msg=ORA-28002: the password will expire within 6 days
Seems that this Error doesn't stop the access to the DB but hang the session in the pool.
After we fixed that, all was back to normal.
Related
Due to slow query, nebula graph was down.
I ran I ran enter code here, and then the connection to the nebulagraph database was lost. The error message is java.net.SocketException:Broken pipe(Write failed)`.
I would like to ask if this is normal and if it is possible to optimize the client to automatically recognize and reinitialize the connection pool?
The application is java client :com.vesoft.client-3.3.0 and the connection pool is com.vesoft.nebula.client.graph.SessionPool.
This is normal, the database kernel restarted the application and the current connection will also be invalid. It will be better that we can catch the corresponding exception in the business layer and re-initialize the session pool. In addition session pool and connection pool are two different concepts.
In my IIS Server, I have many application pools (like 6 to 7) and there are many ASP.NET applications running on each of them (ex. 25 applications per pool). They all are connected with Oracle database by using ADO.NET.
All applications are just working fine, but sometimes we get error like
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I know the possibilities for this like we are not closing our database connections properly. So here is my headache... I don't want to go each and every project to see where we forget to close connections it is very time taking task for us.
So is there any way to identify from which application connections are remaining opened? Can we see from IIS itself? Can we make some kind of utility to track from which project connection are remaining opened?
I'm not sure that it's a probleme of the connection to database. I think that you application are not disposing the context then the garbage collector can't clear memory. you can try to reduce the time for recylcling your application pools then check if you memory usage is decreasing or not.
I have a ASP.NET application running on IIS 8. My application sometimes does not respond to request with clients logging a timeout and then IIS starts sending out 503 Unavailable response codes. Each outage seems to last 5 minutes which corresponds to a 5 minute limit interval for Kill w3wp.
I have the App Pool configured for a Kill w3wp limit action with limit percentage of 80%.
While it might seems the pool is getting killed due to high CPU usage, I could not find any records in the event viewer. In fact, it had gone to idle state because of no requests in about 25 minutes.
The crash happened with 3 simple identical requests issued when the pool was idle.It cannot be reproduced manually in any environment.
This is happening about once a week. Since its a critical application, its running on a separate server with no other applications installed.
I have changed the Limit from Kill to throttle but I am not sure its a good idea. Anyways I've got to find a way to reproduce this.
Can the pool fail to exit an idle state?
If there is no traffic, w3wp.exe may go idle aka stop running. But that is not the same as it being truly stopped and it would not cause a 503. It would just start back up on the next request.
A 503 is more likely because your IIS App Pool is crashing or literally being stopped.
I would check Windows Event Viewer to see if says anything around that time frame around w3wp crashing. You can also set IIS to do more detailed logging anytime your w3wp app pool is stopped, started, recycled.
Docs about enabling more app pool logging:
https://technet.microsoft.com/en-us/library/cc753412%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
You could also search various IIS logs for errors:
https://stackify.com/beyond-iis-logs-find-failed-iis-asp-net-requests/
If you set auto recycle option then remove it and make it schedule recycle instead.
It seems as the memory consumption by your application tends to saturation level, IIS being recycled
I have a w3wp.exe that is restarting on my IIS server (see specs below). Memory gradually climbs to ~3G then it randomly restarts itself about every 1-2min.
Memory Usage:
The odd thing is that once this memory drop (what looks like a restart - btw...the app pool does not get recycled/restarted) happens GET requests are queued but then serviced as soon as the service warms/starts up (causing a delay in responses to our clients - who were initially reporting delayed reponse times on occasion).
I have followed this link to get a stack dump once the .exe restarts (private bytes go to ~0) but nothing gets logged (no .dmp file) with diag debug once the service restarts.
I see tons of warnings in my webserver (IIS) log but that's it:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
ASK: I'm not sure if this is a memory limitation, if cacheing is not playing well with my threads/tasks, if cacheing is blowing up, if there is watchdog service restarting my application, etc,. Has anybody run across something similar with w3wp.exe restarting? It's hard to tell because diagdebug is not giving me a dump once it restarts.
SPECS:
MVC4 Web API servicing GET requests (code is debug build with debug=true)
Uses MemoryCache with Model and Business Objects with cache eviction set to 2hrs...uses
Task (TPS) for each new request.
Database: SQL Server 2008R2
Web Servers: Windows Server 2008R2 Enterprise SP1 (64bit, 64G RAM)
IIS 7.5
One application pool...no other LOB applications running on this server
Your first step is to reproduce the problem in your test environment. Setup some kind of load generation app (you can write it yourself pretty easily) and get the same problem happening. Then turn off debug in web.config and see if that fixes the issue. Then change it to be a release build and test again.
I've never used memorycache - try reducing the cache eviction time or just turn it off and see if that fixes the issue. Good luck :)
I have a set of SSRS05 reports being called through the 2005 web service from our site that runs on Win2k8 and IIS7. The reports are stored remotely on another server from the web server.
In our QA environment, if we have a report that runs for longer than 2 minutes, I get the following error:
Message : The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
Status : KeepAliveFailure
The problem is that the SQL server shows that the stored procedure is executed correctly, the report server shows that the report was rendered correctly (after ~15 minutes), but the web server receives that error consistently after 2 minutes (120 seconds).
This same report runs with no problems on our old environment with ran IIS6. The issue I'm running into is that it looks like long running reports seem to run fine in our production environment. The QA environment as a replicate of our production environment, just with less juice.
Any ideas? I've checked the Keep-Alive status and made sure it's enabled. I've tried changing the connection settings timeout from 120 (default) to 300 seconds and it still fails at 120. I feel that we've been able to isolate to IIS7, but I'm at a lost where to look to change the timeout setting.
Any chance you're running in SharePoint Integrated Mode? If so, see "An ASP.NET Timeout Error Occurs After Two Minutes of Report Processing" in Troubleshooting Configuration Errors.