I have a simple application with a Quartz scheduler started from a servlet listener. The code is available at https://github.com/ike3/quartz-and-ejb
The only interesting part is that the job calls a remote EJB:
InitialContext initialContext = new InitialContext();
PpaJobRemote remote = (PpaJobRemote) initialContext.lookup(beanName);
remote.process();
The code is packaged as EAR containing EJB-JAR and WAR and then deployed on the WildFly 18 application server. To be sure: there are no duplicated interfaces in EAR/lib and WEB-INF/lib.
Everything works fine until the module is restarted. Then suddenly the lookup fails with an exception:
Caused by: javax.ejb.NoSuchEJBException: EJBCLIENT000079: Unable to discover destination
for request for EJB StatelessEJBLocator for "fz44-test-jobs-ear/fz44-test-jobs-ejb/SendMailMessageJob",
view is interface ru.lanit.fz44.ejb.job.PpaJobRemote, affinity is None
The problem occurs only from a Quartz Job. I have servlet calling the same job in the same WAR - it works fine after such restart. So I think there is some weird bug between Quartz thread pool and Widlfly classloaders.
If I restart the whole server (not the single module) the error is gone. The problem is worse that deploy scripts in the CI server use jboss CLI effectively restarting the module now and then.
Can anyone suggest any workarounds of this issue?
Seems I figured it out. The problem is that I didn't shutdown the scheduler. Aparently Quartz reuses the same scheduler even after the app is restarted.
Adding this to the servlet listener fixed the issue.
public void contextDestroyed(ServletContextEvent event) {
try {
if (sched.isStarted()) {
sched.shutdown();
}
} catch (Exception e) {
LOGGER.error(e.getMessage(), e);
}
}
One more example that all the resources must be closed after use and forgetting to do so can lead to strange issues.
Related
I'm trying to port a working Hangfire setup embedded in a Kestrel webserver to a console app. I've modified the web app so it still provides the Hangfire dashboard but doesn't start its own Hangfire server.
The code I must port uses Autofac. I've added the Hangfire.Autofac package to the console app and have already performed all the steps detailed in the answer to Hangfire Autofac .net core 3.1
When I create a job (using the web app) the console app Hangfire server tries to execute the job but I get this failure message:
The requested service 'AED.ServicesLayer.JobProcessing.ProcessManager' has not been registered.
Investigating this we examine the setup of Autofac in the console app. This is how I set up my container.
IConfiguration config = new ConfigurationBuilder()
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.Build();
var containerBuilder = new Autofac.ContainerBuilder();
containerBuilder.RegisterInstance(Log.Logger).AsImplementedInterfaces();
containerBuilder.RegisterModule(new RepositoryModule(config));
containerBuilder.RegisterType<UserService>().As<IUserService>();
containerBuilder.RegisterInstance(config).As<IConfiguration>();
containerBuilder.RegisterModule(new JobProcessingModule(config));
var container = containerBuilder.Build();
When the app is executed, hitting a breakpoint in JobProcessingModule proves the following line of code is executed.
builder.RegisterType<ProcessManager>().As<IProcessManager>();
It is very curious that the containerBuilder instance passed to JobProcessingModule.Load(containerBuilder) is not the same containerBuilder object on which RegisterModule is invoked.
However, experiments with simplified injectables suggest that this is normal, and the injected items are nevertheless visible in the registrations for the container that is returned.
Re-examining the logged failure we note that the class is mentioned by class name and not by interface. Changing the registration by removing the interface registration, like so
builder.RegisterType<ProcessManager>();//.As<IProcessManager>();
caused the ProcessManager to be found in the Hangfire console host but caused run-time errors in the web application when creating the job.
Registering it both ways caused ProcessManager to be found by both, with a new problem surfacing: cannot resolve dependencies. This, however, is merely a new case of the same problem.
While this allows me to move forward with getting a console host working, I do not like code I do not understand. Why should the console host require registration by class name when the web app does not?
Whatever is causing this has also caused Hangfire.IBackgroundJobClient to fail to resolve to the background job client. This is a hangfire class so it really does seem like there is a fundamental problem.
A lengthy investigation eventually revealed, confirmed by experiments, that this code
_recurringJobManager.AddOrUpdate(
insertResult.ToString(),
pm => pm.RunScheduledJobs(insertResult), interval.CrontabExpression
);
is responsible for the behaviour described in the question. AddOrUpdate is a generic method. When it is not explicitly typed it acquires its type from the class of the object passed to it. When the method is explicitly typed as the interface, like so
_recurringJobManager.AddOrUpdate<IProcessManager>(
insertResult.ToString(),
pm => pm.RunScheduledJobs(insertResult), interval.CrontabExpression
);
it remains compatible with the object, but the type acquired by Hangfire is the interface, and the console application can resolve ProcessManager from its interface.
Why the problem was not manifest in the web hosted Hangfire server remains a puzzle, but at least now I'm puzzled by the absence of a problem in a situation I don't have.
Trying to figure out why my console app won't stop running.
Using the following approach in a dotnet core application main method:
await new HostBuilder().
...
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<MyHostedService>();
})
.UseConsoleLifetime()
.Build()
.RunAsync();
Publishing and scheduling that task from the Windows Task Scheduler using the following settings works:
All good so far. All code is properly executed. However, the task stays running, the process never ends. (not even after pressing refresh on the UI of the task scheduler)
Is this expected? If not, how do I get the process to terminate?
If expected, does it still make sense then, to use Generic Host / Hosted Service in a scheduled console app that just starts, runs, and stops?
Answer based on Microsoft.Extensions.Hosting 2.2.0
This behavior is expected, due to your usage of the Generic Host:
It keeps running until shut down or disposed, and you have no shutdown mechanism in place. I assume you expect the Generic Host to shut down after IHostedService.StartAsync(CancellationToken) of your MyHostedService ran to completion. This is not the case, because there might be other IHostedService implementations registered and executed in sequence, and/or a long running BackgroundService which returns control when its ExecuteAsync(CancellationToken) is not completing synchronously to allow other services to run in parallel.
To stop your application gracefully after your MyHostedService completes when running the host via RunAsync, you should constructor-inject the IApplicationLifetime into your MyHostedService and call StopApplication after your Task has completed.
internal class MyHostedService : IHostedService
{
private readonly IApplicationLifetime _appLifetime;
public MyHostedService(IApplicationLifetime appLifetime)
{
_appLifetime = appLifetime;
}
public async Task StartAsync(CancellationToken cancellationToken)
{
await Task.Delay(1000); //your scheduled work
_appLifetime.StopApplication();
}
public Task StopAsync(CancellationToken cancellationToken)
{
return Task.CompletedTask;
}
}
Also, the application may be stopped via either AppDomain.CurrentDomain.ProcessExit or Console.CancelKeyPress, both events are subscribed to by the ConsoleLifetime, which is pre-registered as the default lifetime implementation.
You can read more about lifetime management in the docs.
Microsoft.Extensions.Hosting 3.0.0 - currently in preview - marked IApplicationLifetime obsolete and recommends using IHostApplicationLifetime instead
I am using resin server + spring framework and c3p0 connection pooling. I have configured the connection pool with the following properties file. But somehow every 24 hours or so my website faces connection timeout errors and then i have to restart my resin server to make the website live again. Please tell me whats wrong in the following configuration file and what im missing here.
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.databaseURL=jdbc:mysql://localhost/my_database1_url
jdbc.StockDatabaseURL=jdbc:mysql://localhost/my_database2_url
jdbc.username=my_username
jdbc.password=my_password
jdbc.acquireIncrement=10
jdbc.minPoolSize=20
jdbc.maxPoolSize=30
jdbc.maxStockPoolSize=30
jdbc.maxStatements=100
jdbc.numOfHelperThreads=6
jdbc.testConnectionOnCheckout=true
jdbc.testConnectionOnCheckin=true
jdbc.idleConnectionTestPeriod=30
jdbc.prefferedTestQuery=select curdate();
jdbc.maxIdleTime=7200
jdbc.maxIdleTimeExcessConnections=5
So, a bunch of things.
c3p0 has built-in facilities for observing and debugging for Connection leaks. Please set the configuration parameters unusedConnectionTimeout unreturnedConnectionTimeout and debugUnreturnedConnectionStackTraces. Set an unreturnedConnectionTimeout that defines a period of time after which c3p0 should presume a Connection has leaked, and so close it. Set debugUnreturnedConnectionStackTraces to ask c3p0 to log the stack trace that checked out the Connection that did not get checked in properly. See Configuring to Debug and Workaround Broken Client Applications.
You are configuring c3p0 in a nonstandard way. That might be fine, or not, but you want to verify that the config that you intend to set is the config c3p0 gets. c3p0 DataSources dump their config at INFO on pool initialization. Please consider checking that to be sure you are getting the config you intend. Alternatively, you can check your DataSource's runtime config via JMX.
Besides the nonstandard means of configuration, several of your configuration properties seem amiss. prefferedTestQuery should be preferredTestQuery. numOfHelperThreads should be numHelperThreads.
The following are not c3p0 configuration names at all. Perhaps you are internally mapping them to c3p0 configuration, but you'd want to verify this. Here are the not-c3p0-property-names:
jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.databaseURL=jdbc:mysql://localhost/my_database1_url
jdbc.StockDatabaseURL=jdbc:mysql://localhost/my_database2_url
jdbc.username=my_username
jdbc.maxStockPoolSize=30
In a standard c3p0.properties form, what you probably mean is
c3p0.driverClass=com.mysql.jdbc.Driver
c3p0.jdbcURL=jdbc:mysql://localhost/my_database1_url
# no equivalent -- jdbc.StockDatabaseURL=jdbc:mysql://localhost/my_database2_url
c3p0.user=my_username
# no equivalent -- jdbc.maxStockPoolSize=30
Please see Configuration Properties. Again, c3p0 knows nothing about jdbc.-prefixed properties, but perhaps something in your own libraries or middleware picks those up.
Note: I love to see #NiSay's way of checking for Connection leaks, because I love to see people using more advanced c3p0 API. It will work, as long as you don't hot-update your DataSource's config. But you don't need to go to that much trouble, and there's no guarantee this approach will continue to work in future versions c3p0 makes no promises about ConnectionCustomizer lifecycles. ConnectionCustomizers are intended to be stateless. It is easier and safer to use c3p0's built-in leak check facility, described in the first bullet-point above.
As there could be possibility of connection leaks in the program (the probable cause of connection timeouts), you need to follow the below steps in order to identify the leaks.
Make as entry in your c3p0.properties file
c3p0.connectionCustomizerClassName = some.package.ConnectionLeakDetector
Create a class with name 'ConnectionLeakDetector' and place it in appropriate package. Below is the content of the class.
import java.sql.Connection;
import java.util.concurrent.atomic.AtomicInteger;
public class ConnectionLeakDetector implements com.mchange.v2.c3p0.ConnectionCustomizer {
static AtomicInteger connectionCount = new AtomicInteger(0);
#Override
public void onAcquire(Connection c, String parentDataSourceIdentityToken)
throws Exception {
}
#Override
public void onDestroy(Connection c, String parentDataSourceIdentityToken)
throws Exception {
}
#Override
public void onCheckOut(Connection c, String parentDataSourceIdentityToken)
throws Exception {
System.out.println("Connections acquired: " + connectionCount.decrementAndGet());
}
#Override
public void onCheckIn(Connection c, String parentDataSourceIdentityToken)
throws Exception {
System.out.println("Connections released: " + connectionCount.incrementAndGet());
}
}
The onCheckOut method will increment the count when the connection is acquired, where as onCheckOut will decrement it when the connection is released.
Execute some scenarios and observe the statistics on your console. If the count is more than 0, then the scenario executed has a connection leak. Try to fix them and you will observe the difference.
As a side note, you can increment the jdbc.maxPoolSize as a temporary solution until you deploy the fix.
I have an EJB3.0 timer which runs great.During application deployment i see this error in my WL logs,
An exception occurred while registering the MBean null.java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=admin,Name=weblogic.ejb.timer"
And during undeployment this
An unexpected error was encountered while attempting to remove any EJB Timers from the persistent store for the EJB 'TimerBean(Application: )
I don't use persistence store mechanism.I trigger the timer with servlet context.
We use WL 10.3.1,How can i overcome/catch this exception so,that it wouldn't be displayed during build process.
Thanks
The WLS ejb timers are persisted to a default store. The error messages seem to be related to it. Its likely that the ejb timer from a previous deployment is interfering. Does a server restart resolve this issue? You may want to try your app on WLS 10.3.4 to see if the issue has been resolved.
I have many servlets in a web applicaton; for some stange reason, only and only one of them always fails in initialization with the following error trace:-
00000045 ServletWrappe E SRVE0100E: Did not realize init() exception thrown by servlet MyServletX: java.lang.NullPointerException
at com.ibm.ws.webcontainer.WebAppPmiListener.onServletStartInit(WebAppPmiListener.java:120)
at com.ibm.ws.webcontainer.webapp.FireOnServletStartInit.fireEvent(WebAppEventSource.java:237)
at com.ibm.ws.webcontainer.util.EventListeners.fireEvent(EventListeners.java:48)
at com.ibm.ws.webcontainer.webapp.WebAppEventSource.onServletStartInit(WebAppEventSource.java:105)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:261)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:444)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:2841)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:220)
at com.ibm.ws.webcontainer.VirtualHost.handleRequest(VirtualHost.java:204)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1681)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:77)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:421)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:367)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:94)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:548)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:601)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:934)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1021)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1332)
I could not figure out if there is anything extra ordinary with this servlet. There is no init() method in this servlet and it extends HTTPServlet. Any idea what could be reason? I am using websphere server 6.0.x. How to get more debugging information in this case?
Well I don't know still cause of above error, but this is how it started working strangely:- i) Re-applied recommended fixes by IBM for my WAS version (especially there are IBM JDK upgrade related fix patches) ii) created a new profile of server iii) Install web application to new profile and it started working.
I don't think this is a product issue.
To debug this problem what i would suggest is to place a simple servlet (kind of Hello World) and deploy it to the server and see what happens.
initialization does not necessarily mean init() method alone.
If you have a static block in your servlet, if you have any variables that are initialized they would all be part of the initialization activity.
Look at the FFDC logs that were generated when this error occurred and that should provide you with clues.
As bkail mentioned, also ensure that yo have the latest fixpacks just to eliminate known problems with the product.
if the hello world servlet works, suggest you place hte servlet code here along with the SystemOut and System Err logs that correspond to this issue along with the relevant FFDC logs and i am sure most of us will be able to help you out with this
HTH
Manglu