I am doing a code review and found this piece of code. Does throwing an exception inside a thread bring down IIS? Read the comments.
Thread unhandledExceptionThread = new System.Threading.Thread(new System.Threading.ThreadStart(delegate()
{
System.Threading.Thread.Sleep(90 * 1000);
throw new ApplicationException("Thread 1");
}));
unhandledExceptionThread.Name = "IntentionalCrasher";
Thread environmentExitThread = new System.Threading.Thread(new System.Threading.ThreadStart(delegate()
{
System.Threading.Thread.Sleep(100 * 1000);
Environment.Exit(-500);
}));
environmentExitThread.Name = "Thread 2";
unhandledExceptionThread.Start();
environmentExitThread.Start();
It won't bring "IIS down" but it will terminate the ASP.NET worker process. It will restart.
Your code snippet first tries to die by throwing an exception. 10sec later, if that didn't help, it takes matters into its own hands and kills the process.
Alternatively, it could call AppDomain.Unload on the current domain. But this is a reasonable way to restart the worker process.
Make sure rapid fail protection is disabled. This feature is the devil because it permanently shuts down your app after 5 failures. Permanently without notification. On any kind of error.
Related
I have a Quarkus application where I use the event bus.
the code in question looks like this:
#ConsumeEvent(value = "execution-request", blocking = true)
#Transactional
#TransactionConfiguration(timeout = 3600)
public void consume(final Message<ExecutionRequest> msg) {
try {
execute(...);
} catch (final Exception e) {
// some logging
}
}
private void execute(...)
throws InterruptedException {
// it actually runs a long running task, but for
// this example this has the same effect
Thread.sleep(65000);
}
Why do I still get a
WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 63066 ms, time limit is 60000 ms: io.vertx.core.VertxException: Thread blocked
I'm I doing something wrong? Is the blocking parameter at the ConsumeEvent annotation not enough to let that handle in a separate Worker?
Your annotation is working as designed; the method is running in a worker thread. You can tell by both the name of the thread "vert.x-worker-thread-0", and by the 60 second timeout before the warnings were logged. The eventloop thread only has a 3 second timeout, I believe.
The default Vert.x worker thread pool is not designed for "very" long running blocking code, as stated in their docs:
Warning:
Blocking code should block for a reasonable amount of time (i.e no more than a few seconds). Long blocking operations or polling operations (i.e a thread that spin in a loop polling events in a blocking fashion) are precluded. When the blocking operation lasts more than the 10 seconds, a message will be printed on the console by the blocked thread checker. Long blocking operations should use a dedicated thread managed by the application, which can interact with verticles using the event-bus or runOnContext
That message mentions blocking for more than 10 seconds triggers a warning, but I think that's a typo; the default is actually 60.
To avoid the warning, you'll need to create a dedicated WorkerExecutor (via vertx.createSharedWorkerExecutor) configured with a very high maxExcecuteTime. However, it does not appear you can tell the #ConsumeEvent annotation to use it instead of the default worker pool, so you'd need to manually create an event bus consumer, as well, or use a regular #ConsumeEvent annotation, but call workerExectur.executeBlocking inside of it.
I am experiencing some randomly happening unhandled exception causing w3wp to crash. I want to trace the cause of that exception. I already have a global Application_Error handler override in my MvcApplication class, so the crash must be caused by some out-of-http-context exception. In order to replicate the problem I genereate one myself in a timer callback, and try to trace it. Simplified code like
public static class MonitorTimers
{
public static Timer _taskMonitorTimer = new Timer(state: null, dueTime: 1000, period: 1000, callback: (state) =>
{
throw new Exception("Ouch! Me dead.");
});
}
In my local development environment (iisexpress launched by VS2017) and test environment (IIS 8.5), when the app starts and then crashes, the following can be seen in event viewer:
The most useful Event 1325 and 1026 sourced from ASP.NET and .NET Runtime shows the stack trace - just the thing I need.
My problem is, in my production machine (also IIS 8.5) I can't find the useful event 1325. Only a crash report, bearing no more information than I know. So I don't know what caused the error. I could surround my timer callback with try...catch block but the error could well be caused by something else (unmanaged libraries, error in static class initialization) then I still can't trace.
So suggestions on why event 1325 is missing or some tools that can show the log and analyse the stack trace is greatly appreciated. Thank you.
So, in your case you generate Exception diring Loading of application domain.
When CLR load application domain it firstly init static fields. So, if your code has problem with static fields, then exception will throw until it specifies Application_Error handler.
One more point, Is your application take a lot of memory? There are 2 cases when application can not write logs and execute code in catch block: StackOverflowException and OutOfMemoryException. Can you check is it has some memory leaks or infinite recursion?
One more point: set in visual studion setting to break when any exception throwed.
One more point: It is better to move your initialization logic from static constructors to ApplicationStart or something like this. You can do it temporary, for catch the bag and then move it to previous state.
I have a signalr server which is hosted in IIS.
There is a function in the hub which starts 600 processes in windows and then kills them.
//start 600 processes
for (int i = 0; i < 600; i++)
{
try
{
Process myProcess = Process.Start(startInfo);
proclist.Add(myProcess);
Task.Delay(10).Wait();
}
catch(Exception e)
{
feedback = "Process " + i + " cannot be started: " + e.Message;
break;
}
feedback = "All processes are running.";
}
//kill them
foreach (var proc in proclist)
{
try
{
proc.Kill();
Task.Delay(10).Wait();
}
catch (Exception e)
{
feedback = "Process " + proclist.IndexOf(proc) + " cannot be killed: " + e.Message;
break;
}
feedback = "All Processes are killed.";
}
However, when I call this function in Client I get an Exception whiling killing the processes:
Process 104 cannot be killed: Die Anforderung kann nicht verarbeitet werden, da der Prozess beendet wurde(The request cannot be proceeded, because the process is already terminated.)
It seems that I can only keep 104 processes runing. And the rest of them terminate immediately after start.
I tried the same thing in a Console Application, and all processes can be started and killed.
I tried to consume a lot of memory using another application and I could also keep 104 processes running.
I tried to consume a lot of memory using another application and I could also keep 104 processes running.
I also checked all possible IIS configuration and I could not find any settings which is related to this issue.
So I would like to ask whether anyone knows how to start more procecces in an ASP.NET application.
I will appreciate it very much if someone can help me. Thanks!
I strongly suggest you do not execute 600 (or any multiple of hundred) processes under ASP.NET. You will really strain the resources on the Aspnet_wp.exe process which could hurt the performance of the IIS box.
You need to re-think the design.
If this was me, I would consider creating an external process outside of ASP.NET which could do the hard work for you. For example, maybe you can create a Windows service (or even just a .NET console application running on the server) that waits (i.e. listens) on a file system folder for a file (you can name the file anything you like e.g. start.txt) to be created which you could do when a request to your website is made. That service will then execute the 600 exe files for you.
I'm not familiar with lasttest, so my suggestion might not be adequate. However, I do not believe you will achieve what you are looking for using your current design. It's going to hurt performance, and in fact, I'm not suprised that a limit of running processes has been reached. I do not know of any documentation that points to how many exe files you can run in Aspnet_wp.exe, but that's likely because the ASP.NET team never expected anyone to attempt this.
We have a Java class that listens to a database (Oracle) queue table and process it if there are records placed in that queue. It worked normally in UAT and development environments. Upon deployment in production, there are times when it cannot read a record from the queue. When a record is inserted, it cannot detect it and the records remain in the queue. This seldom happens but it happens. If I would give statistic, out of 30 records queued in a day, about 8 don't make it. We would need to restart the whole app for it to be able to read the records.
Here is a code snippet of my class..
public class SomeListener implements MessageListener{
public void onMessage(Message msg){
InputStream input = null;
try {
TextMessage txtMsg = (TextMessage) msg;
String text = txtMsg.getText();
input = new ByteArrayInputStream(text.getBytes());
} catch (Exception e1) {
// TODO Auto-generated catch block
logger.error("Parsing from the queue.... failed",e1);
e1.printStackTrace();
}
//process text message
}
}
Weird thing we cant find any traces of exceptions from the logs.
Can anyone help? by the way we set the receiveTimeout to 10 secs
We would need to restart the whole app for it to be able to read the records.
The most common reason for this is the listener thread is "stuck" in user code (//process text message). You can take a thread dump with jstack or jvisualvm or similar to see what the thread is doing.
Another possibility (with low volume apps like this) is the network (most likely a router someplace in the network) silently closes an idle socket because it has not been used for some time. If the container (actually the broker's JMS client library) doesn't know the socket is dead, it will never receive any more messages.
The solution to the first is to fix the code; the solution to the second is to enable some kind of heartbeat or keepalives on the connection so that the network/router does not close the socket when it has no "real" traffic on it.
You would need to consult your broker's documentation about configuring heartbeats/keepalives.
I have a big web application on asp.net 2.0.
Usere open object editor there and makes some changes. They can not open the same object at one time.
After they press "save" btn all changes process to save on server via postback.
I am using transaction for save. There are a lot of procedures, checks and others to be done before saving operation is OK.
using (SqlConnection con = .........)
{
SqlTransaction trans = null;
try
{
con.Open();
trans=con.BeginTransaction(IsolationLevel.ReadUncommitted);
........operations.........
trans.Commit();
}
catch (Exception e)
{
try { if (trans != null) trans.Rollback(); }
catch { }
throw new MyException("SQL Exception: " + e.Message, e);
}
finally
{
if (con != null && con.State == ConnectionState.Open) con.Close();
}
}
For me this code is quite safe.
But periodicaly happens:
one process from this web application on saving operation on mssql became "sleeping/awaiting".
and others processes called by other users became locked by this process and organiza a queue.
One of them threw timeout excetion.....but others are waiting.
So, my question is: does my code have some bad operation that allows command to became sleeping/awaiting?
May be there are some tricks?
Since you are doing a lot DB of operations inside a transaction there is a chance that a database lock is blocking your application.
You can use the sp_who2 stored procedure (there are more details on it here) to see if there any blocks on your server, by checking the BlkBy column of the result.
You might also want to check the following links on Sql Server locks and deadlocking
Может ли это быть связано с репликацией. В момент когда наблюдаются описанные выше тормоза репликация начинает сильно тормозить. И пишет:
The process could not connect to Publisher 'эта бд'. (Source: MSSQL_REPL, Error number: MSSQL_REPL20084)
Get help: http://help/MSSQL_REPL20084
· TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Source: MSSQLServer, Error number: 10060)
Get help:
· An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. (Source: MSSQLServer, Error number: 10060)
Get help:
· Login timeout expired (Source: MSSQLServer, Error number: 0)
Get help:
· The merge process failed to execute a query because the query timed out. If this failure continues, increase the query timeout for the process. When troubleshooting, restart the synchronization with verbose history logging and specify an output file to which to write. (Source: MSSQLServer, Error number: 0)
Get help:
Эта БД является источником репликации.
But the main problem is that the first process does not threw timeout exception. It is sleeping as log as we do not kill him. That is the main problem.