I'm running an application which has a background process which runs every 5 minutes. It runs in a clustered environment (WAS 7.1) but the process should only run on one server at a time. For this reason I've opted for EJB Timer using the javax.ejb.TimedObject interface. The timer is initially scheduled via a Web application (Spring MVC) in the same EAR. This has been running fine for over a year.
Recently I have upgraded the application to Spring 3.2.4 and made use of profiles. Now when the server recreates the EJB as part of it's lifecycle it sometimes is complaining that it cannot find one of the bean definitions. This bean happens to be defined twice, once withing a beans tag with a profile of 'live' and another with 'test'. The Web Application that initially schedules the timer uses a contexParam in the web.xml of 'spring.profiles.active' set to 'live'. This ensures the timer initially starts with the 'live' configuration. I suspect that when the server recreates the EJB it is recreating the spring context but the profile parameter is being lost somewhere.
Has anyone come across this before or have any suggestions as to how this might be solved?
Thanks in advance.
Related
I am working on an app which has the web component (visited via browser) and background task processing component, to which web component delegates some long running stuff.
I've just hit an issue when I refreshed my web browser only to find it loading indefinitely (first spotted in AJAX, but later in normal request).
It did not really look apparent but as soon as I shut down the background Symfony command which also utilizes EntityManager the browser get unblocked and proceeds with request.
My app uses RabbitMQ to store job requests which are publish by web component. The Symfony command uses the same "backbone" to create RabbitMQ consumer and take consume those jobs.
I tried, without any result:
Restarting Apache
Restating RabbitMQ
Purging RabbitMQ queue
Using different EntityManagers for web and command
I use OldSoundRabbitMqBundle (link) to facilitate communication between those two.
The web component gets stuck regardless of action being called (not related to RabbitMQ producer).
Has anyone stumbled upon similar issue?
This happens on dev box, I haven't got around giving it a spin on a production server, nor would I until I find out more about this.
It would seem that I misused the locking mechanism in Postgres. Indeed the task processing component is a long-running task, but given that it is Symfony command, Doctrine connection is being established as early as possible.
Now comes the tricky part: I used the LOCK TABLE statement to lock some tables away from concurrent access (EXCLUSIVE type). Without closing the connection (not entity manager), those locks are left intact, until I restart the command (every 10th task).
This was the root cause.
I am still investigating some edge-cases, but since I moved away to advisory locking, I had no more lock-ups.
This question already has answers here:
How can I make a ServletContextListener stop the Java EE application?
(2 answers)
Closed 6 years ago.
I have implemented a ServletContextListener in a Java Servlet web app via the #WebListener annotation. In my contextInitialized method I do some set-up work and verify that expected resources are available.
If I determine in that contextInitialized method that something is wrong, how do I stop the web app from continuing onwards with executing servlets? Those servlets should not execute if the environment is not suitable (such as no database available).
How to gracefully handle a faulty environment for a servlets-based web app?
No, it appears that the ServletContextListener interface was not designed with the intent to be able to prevent the launch of a web app.
As this Answer states, the Servlet spec says a ServletContextListener may somehow disable access to the web app when an Exception is encountered. That word may means optional, not required. Nor does the spec define exactly what stopping access to the web app means.
Apparently the implemented behavior in various web containers varies widely. Some do nothing, some log it and move on, some prevent the web app from being deployed.
My experience with Tomcat 8.0.33… Putting throw new RuntimeException ( "bogus stop servlet " ); in the contextInitialized method prevents the app from being deployed. The console during deployment in the IDE report reports “FAIL - Deployed application at context path / but context failed to start”. Unfortunately neither that console nor none of the logs capture the report of the actual Exception. So if you throw more than one Exception from one or more listeners, debugging will not be obvious.
As mentioned elsewhere in Stack Overflow, the most reliable solution is probably to have your ServletContextListener mark success or failure with a flag variable stored in the servlet session. Then have your servlet code retrieve and examine that flag. Your servlet code would then determine the appropriate course of action. Your web app would be deployed, but your own servlet(s) could choose to do nothing and send back some HTTP error code.
Similar questions:
Abort java webapp on startup
How can I make a ServletContextListener stop the Java EE application?
Prefered way to handle Java exceptions in ServletContextListener
Prevent Java EE application start on WebSphere on exception
ServletContextListener execution order
Stop Servlet context from initializing in init method
Side note: When adding or editing your ServletContextListener you may need to do a 'clean-and-build' operation on your project. Your IDE’s hot-swap or deploy-while-developing feature may not pickup on a new or changed listener. Trace your code or do some logging to verify.
I created a service with a RESTful API in ASP.NET, hosted in IIS. Inside this service, I would like to create an actor system with Akka.NET.
Upon creating the actor system:
var actorSystem = ActorSystem.Create("myActorSystem");
The following exception is thrown:
A first chance exception of type 'System.InvalidOperationException' occurred in System.Web.dll
Additional information: An asynchronous operation cannot be started at this time. Asynchronous operations may only be started within an asynchronous handler or module or during certain events in the Page lifecycle. If this exception occurred while executing a Page, ensure that the Page is marked <%# Page Async="true" %>. This exception may also indicate an attempt to call an "async void" method, which is generally unsupported within ASP.NET request processing. Instead, the asynchronous method should return a Task, and the caller should await it.
The actor system is inherently a concurrent system with asynchronous messages being exchanged between actors. As explained here, this actor system would not survive IIS taking down the AppDomain, which is probably why the aforementioned exception is thrown.
This article explains how to run background tasks in ASP.NET. However, I don't see how I could use this for my actor system, as I have no control over the lifecycle of background tasks that might be created by Akka.NET.
Is there a way to make this work, or should I abandon the idea of having an actor system in an ASP.NET application?
EDIT: I also saw a question on Stackoverflow about implementing a REST service using Akka. Any advice about a solution similar to the Spray toolkit, but working for Akka.NET would be welcome.
I've used Akka.NET and Akka.Remote inside ASP.NET MVC applications that are doing up to 1000 requests per second on EC2 - so I'll share some of the tips and tricks I used to get it up and running successfully. Had a prototype version that even used Akka.Cluster but ended up not shipping that version.
Best place to call ActorSystem.Create is inside Global.asax Application_Start().
Hang onto a static reference to the ActorSystem object inside Global.asax itself, using a static field or property. Helps ensure that the ActorSystem itself doesn't get garbage-collected in long-running applications.
Create a separate static helper class to initialize any top-level actors your applications needs - i.e. actors at the top of the /user/ hierarchy. This class should also provide actor paths that your ASP.MVC controllers and action methods can use for Tell and Ask operations.
Creating the ActorSystem is a bit of an expensive operation, because lots of system-level stuff gets fired up at once. It's definitely best to do this once at application startup and then just cache the result inside the Application class.
Creating individual actor instances is cheap - you should be able to do this no-problem inside ASP.NET MVC action methods. If you see this error come up again, please let us know what part in the request-handling process this error occurred and with which version of ASP.NET.
Edit: added some updated guidance for how to do this on ASP.NET Core
https://petabridge.com/blog/akkadotnet-aspnetcore/
Keep your ActorSystem as a shared property in some static class container - this way you may access it from the rest of your application. Actor system initialization/disposal can be done by:
Global.asax - use ActorSystem.Create(...) inside Global.asax Application_Start and dispose it with system.Shutdown() on Application_End.
OWIN - create actor system in OWIN's Startup.Configuration method and shut it down by binding to host.OnAppDisposing event (how-to link).
Remember that IIS will startup your web app only after first request and tear it down automatically after some time when it's idle. Therefore make sure, that your deployment script will ping application after publishing and set idle timeout (link) for long enough if you want your Akka actor system to run continuously.
Second option
Separate your Actor System logic and deploy it, for example, as a Windows Service (or Linux deamon). Turn on Akka.Remoting for it and create a proxy client, which will forward all application long-running sensitive tasks to external service. Similar solution is often used for things such as schedulers or event buses, when your application logic must be working continuously.
I was developing a simple Spring MVC application that invokes a Web Service over a JMS trasport. I need to deploy it on websphere application server. Everything seems to work fine when I reference the remote connection factory directly from my application, and even seems to work properly when I try to use a local JNDI Queue connection factory on my WAS, but then, I cannot shutdown my server.
It seems to be because its connections pool keeps locked by somebody. It never invokes Session.close over JMS session (it is managed by JAX-WS server framework, in this case Apache Axis2). I have done the same from an even simpler spring MVC application (just #Controller and views) and it stops fine. The application that makes pool to be locked just includes a more complex context with persistence and transaction management (annotation-driven).
Do you think that this transaction mananagement could affect JMS session management? Somebody have ever found a JMS connection pool locked by an spring application?
Thanks.
Regards.
That was the solution to avoid locking. We removed ServletContext.close, so our server can find the references to used resources, and can release connections on pool.
When IIS restarts an ASP.Net (2.0) web application, it can either:
Recycle the AppDomain: Unload the AppDomain and load a new AppDomain on the same process (e.g. when HttpRuntime.UnloadAppDomain() is called, when web.config is changed).
Recycle the process: unload the AppDomain and load a new one on a new process (e.g. when invoking Recycle command on an AppPool via inetmgr, or when memory limit is reached).
Due to some internal reasons (trouble with legacy native code we depend upon), we cannot allow the first option. We just can't load the application twice on the same process.
Can IIS be somehow told to never allow worker process reuse?
I've tried preventing it myself by tracking whether an application has already started on a process once using a Mutex, and if so - throwing an exception during Application_Start()); I've also tried terminating the process by calling Environment.Exit() during Application_End(). The problem with both methods is that it causes any requests that arrive during Application_End or Application_Start to fail (unlike a manual process recycle, which fails absolutely no requests because they are redirected to the new process right away).
I believe that "Recycle the AppDomain" comes under preview of ASP.NET runtime and IIS is not really involved anywhere (I am not 100% sure about this in case of integrated pipeline of IIS7). So I don't think that what you want is feasible. But there are couple of workaround that you may consider for your problem:
Ensure that you run start-up code (manipulating legacy code) to run only once - this should be possible via named system semaphores. Once system semaphore is created by app start-up in worker process, it will exists till process is recycled so you can have per process initialization.
If #1 is not possible then consider hosting code manipulating legacy code in a separate process all together - this process can expose relevant functionality via WCF services over named pipes. ASP.NET will consume them to use legacy code.
Couldn't find a way to tell IIS that worker processes are not to be reused. Can't afford fixing the underlying problem that forbids process reuse. Therefore, ended up calling Environment.Exit(0) in Application_End, although it may cause a small number of requests to fail with a connection reset.