Hosted service not terminating after Environment.Exit - .net-core

I've got a .NET core 3.1 app with a hosted service that runs as a console application on Windows.
In case of an error I'm trying to terminate the worker with Environment.Exit(1).
Now the problem is that, if Enviroment.Exit() is called before any await in ExecuteAsync, the application does not terminate. It logs Waiting for the host to be disposed. Ensure all 'IHost' instances are wrapped in 'using' blocks. and then hangs indefinitely.
When I await anything before the call to Enviroment.Exit() it also logs that, but it terminates as expected.
Here is the simplest code that I could come up with to reproduce the problem.
The NotTerminatingWorker hangs forever, the TerminatingWorker terminates. The only difference is a tiny Task.Delay:
public class Program {
public static async Task Main(string[] args) {
using var host = CreateHostBuilder(args).Build();
await host.RunAsync();
}
public static IHostBuilder CreateHostBuilder(string[] args) {
return Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) => { services.AddHostedService<NotTerminatingWorker>(); });
}
}
public class NotTerminatingWorker : BackgroundService {
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
Environment.Exit(1);
}
}
public class TerminatingWorker : BackgroundService {
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
await Task.Delay(1);
Environment.Exit(1);
}
}
I would expect that both behave the same way, but that's obviously not the case.
Any explanation for this would be greatly appreciated!
UPDATE: The application should be able to run both as a console application and as a Windows service. The non-zero return code is required to get it restarted if it crashes.
And apparently Windows does not restart services that exited with code 0.

I believe the behavior you're seeing is a side-effect of how the .NET Core runtime does its startup: it calls ExecuteAsync for each background worker and then waits for it to complete. So a synchronous ExecuteAsync can cause problems. I've used Task.Run to work around this.
In case of an error I'm trying to terminate the worker with Environment.Exit(1).
I recommend not using Environment.Exit at all. Instead, do a controlled shutdown by injecting IHostApplicationLifetime and calling StopApplication. This will trigger the stoppingToken for each of your background services, and if they ignore it, they will be forcibly terminated after a timeout.

Handling the hostLifetime events in the Main method did for me the job. This is working for me on .NET6
public static int Main(string[] args)
{
ExitCode = 0;
ILogger? logger = null;
try
{
var builder = CreateHostBuilder(args)
.Build();
var hostLifetime = builder.Services.GetRequiredService<IHostApplicationLifetime>();
logger = builder.Services.GetService<ILogger<Program>>();
// register on hostLifetime events for handling stopping and finalize
using var hostLtAppStopping = hostLifetime.ApplicationStopping.Register(() =>
{
// service is about to stop... do some cleanup stuff here
});
using var hostLtAppStopped = hostLifetime.ApplicationStopped.Register(() =>
{
logger?.LogDebug("Service graceful shout down, exit with code {exitCode}!", ExitCode);
Environment.Exit(ExitCode); // ExitCode is set by the caller of hostApplicationLifetime.StopApplication
});
// start the service
logger?.LogDebug("builder.Run()");
builder.Run();
}
catch (Exception e)
{
logger?.LogError(e, "Unhandled Exception occurred => exit with exit code 1!");
ExitCode = 1;
return ExitCode;
}
return ExitCode;
}

Related

Blazor WebAssembly SignalR HubConnection causing javascript error on reload

I have a SignalR HubConnection within my Blazor WebAssembly application and whilst it works most of the time, if I reload the page (via the browser reload) then I often am getting the following error in the console and the connection is not made:
Uncaught Error: The delegate target that is being invoked is no longer available. Please check if it has been prematurely GC'd.
at Object.invoke_delegate (dotnet.5.0.4.js:1)
at WebSocket. (dotnet.5.0.4.js:1)
Here's a rough, simplified view of the code where I create the HubConnection (and dispose it).
#inherits LayoutBase
#attribute [Authorize]
<AuthorizeView>
<Authorized>
//...
</Authorized>
<NotAuthorized>
//...
</NotAuthorized>
</AuthorizeView>
public class LayoutBase : LayoutComponentBase, IAsyncDisposable
{
[Inject] public IAccessTokenProvider AccessTokenProvider { get; set; }
private readonly HubConnection _hubConnection;
protected override async Task OnInitializedAsync()
{
_hubConnection = new HubConnectionBuilder()
.AddNewtonsoftJsonProtocol(c =>
{
//...
})
.WithUrl(notificationHubUrl, option => option.AccessTokenProvider = GetAccessToken)
.WithAutomaticReconnect()
.Build();
_hubConnection.Closed += HubConnectionOnClosed;
_hubConnection.Reconnected += HubConnectionOnReconnected;
_hubConnection.Reconnecting += HubConnectionOnReconnecting;
await _hubConnection.StartAsync()
await base.OnInitializedAsync();
}
private async Task<string> GetAccessToken()
{
var tokenResult = await AccessTokenProvider.RequestAccessToken(...)
// etc...
}
// .. Event Handlers
public ValueTask DisposeAsync()
{
_logger.LogInformation($"Disposing Hub: {_hubConnection.ConnectionId}");
_hubConnection.Closed -= HubConnectionOnClosed;
_hubConnection.Reconnected -= HubConnectionOnReconnected;
_hubConnection.Reconnecting -= HubConnectionOnReconnecting;
return _hubConnection.DisposeAsync();
}
}
Previously I had it as an injected service but I eventually simplified it to this structure but it continues to get this error on reload. It's not every time I reload but most times.
I have tried changing the dispose pattern without success. I can't find any information on the error anywhere else.
Any ideas?
I don't have a definitive answer as to the underlying reason but I suspect that this is a bug somewhere in the SignalR/dotnet framework resulting in the GCing of a delegate because something drops a reference to it.
One way I've managed to provoke this error reasonably consistently is to have a handler returning just a Task, e.g.
_hubConnection.On<TEvent>(eventType.Name, OnEvent);
where OnEvent looks like this:
// THIS IS THE BROKEN SIGNATURE - DO NOT USE
private async Task OnEvent<TEvent>(TEvent e)
{
}
A workaround which appears to have fixed it for me is to make the handler actually return something. This seems to make something deeper in the framework hold a reference for longer so that it doesn't get GC'ed. E.g.
// WORKS ON MY MACHINE - Note the return type of Task<object>
private async Task<object> OnEvent<TEvent>(TEvent e)
{
// ... Do stuff
return null;
}

rewrite an IHostedService to stop after all tasks finished

I have an application that normally should be a simple console application to be programmed as a scheduled task from time to time called by the windows task scheduler.
The program should launch some updates on two databases, one service per one database. Say ContosoDatabase should be updated by the ContosoService.
Finally it was written as an .NET Core app using, and maybe is not the best choice, the IHostedServices as base for the service, like this:
public class ContosoService : IHostedService {
private readonly ILogger<ContosoService> _log;
private readonly IContosoRepository _repository;
private Task executingTask;
public ContosoService(
ILogger<ContosoService> log,
IContosoRepository repository,
string mode) {
_log = log;
_repository = repository;
}
public Task StartAsync(CancellationToken cancellationToken) {
_log.LogInformation(">>> {serviceName} started <<<", nameof(ContosoService));
executingTask = ExcecuteAsync(cancellationToken);
// If the task is completed then return it,
// this should bubble cancellation and failure to the caller
if (executingTask.IsCompleted)
return executingTask;
// Otherwise it's running
// >> don't want it to run!
// >> it should end after all task finished!
return Task.CompletedTask;
}
private async Task<bool> ExcecuteAsync(CancellationToken cancellationToken) {
var myUsers = _repository.GetMyUsers();
if (myUsers == null || myUsers.Count() == 0) {
_log.LogWarning("{serviceName} has any entry to process, will stop", this.GetType().Name);
return false;
}
else {
// on mets à jour la liste des employés Agresso obtenue
await _repository.UpdateUsersAsync(myUsers);
}
_log.LogInformation(">>> {serviceName} finished its tasks <<<", nameof(ContosoService));
return true;
}
public Task StopAsync(CancellationToken cancellationToken) {
_log.LogInformation(">>> {serviceName} stopped <<<", nameof(ContosoService));
return Task.CompletedTask;
}
}
and I call it from main like this:
public static void Main(string[] args)
{
try {
CreateHostBuilder(args).Build().Run();
}
catch (Exception ex) {
Log.Fatal(ex, ">>> the application could not start <<<");
}
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host
.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) => {
var config = hostContext.Configuration;
if (args.Contains("Alonso")) {
services
.AddHostedService(provider =>
new AlonsoService(
provider.GetService<ILogger<AlonsoService>>(),
provider.GetService<IAlonsoRepository>()));
}
// if there also Cedig in the list, they can be run in parallel
if (args.Contains("Contoso")) {
services
.AddHostedService(provider =>
new ContosoService(
provider.GetService<ILogger<ContosoService>>(),
provider.GetService<IContosoRepository>()));
}
});
Now, the problem, is surely, that the application will not stop once all updates finished.
Is there a way to quickly rewrite the application in order to make it stop after the second service finishes its tasks?
I tried to put the Environment.Exit(0); at the end
public static void Main(string[] args) {
try {
CreateHostBuilder(filteredArgs.ToArray()).Build().Run();
}
catch (Exception ex) {
//Log....
}
Environment.Exit(0); // here
}
but it does not seem to help: the application is still running after all task are completed.
Following #Maxim's suggestion, I found this dirty but working workaround, by injecting the IHostApplicationLifetime and the lastService boolean:
public ConsosoService(
IHostApplicationLifetime hostApplicationLifetime,
// ...
bool lastService)
{ ... }
public async Task StartAsync(CancellationToken cancellationToken)
{
// do the job
if (_lastService)
_hostApplicationLifetime.StopApplication();
// stops the application and cancel/stops other services as well
}
HostedServices are background services. It's the other way around: they can react to application start and stop events, so that they can end gracefully. They are not meant to stop your main application when finished, they potentially live as long as the application does.
I'd say you will be better served with simple Tasks and awaiting all of them. Or send some events when your background jobs finishes its work and handle them in main.
Whatever trigger you may choose you can stop .net app by injecting IHostApplicationLifetime and calling StopApplication() method on it. In earlier versions it's just IApplicationLifetime.
Looking at IHost Interface documentation the method run() does not stop until the host is shutdown. seems that StopAsync() did not stop the service. so Environment.Exit(0); was never reached. maybe use CancellationToken to forcefully end the host, or inject Environment.Exit(0); in ContosoService class if possible even though not optimal.
Here is another approach without need for creating hosted service
using var host = CreateHostBuilder(args).Build();
await host.StartAsync();
using var scope = host.Services.CreateScope();
var worker = scope.ServiceProvider.GetService<Worker>();
await worker!.Run();
await host.StopAsync();
IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices(services => ConfigureServices(services));
void ConfigureServices(IServiceCollection services)
{
//main class which does the work
services.AddScoped<Worker>();
//do some DB operations
services.AddScoped<DbCtxt>();
}
Complete code https://github.com/raghavan-mk/dotnet/tree/main/DIInConsole

Windows service using BackgroundService cannot be stopped

I'm using a Net Core 3.X BackgroundService and after publishing my code, I've installed the executable generated as a Windows Service.
On my ExecuteAsync method I have some code like this:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
//do something
if(someConditionIsTrue)
{
await this.StopAsync(new CancellationToken());
}
}
}
That manual call on StopAsync stops the execution and exits from while loop, but when I go to services.msc I see that my Windows Service is still in running state although is not executing anything.
How can I stop the service automatically and not calling to "cmd \c sc stop..."?
Thanks
Your BackgroundService is stopping correctly, but the application is not being stopped.
To stop the application from a background service, inject a IHostApplicationLifetime and call it as such:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
//do something
if (someConditionIsTrue)
{
_hostApplicationLifetime.StopApplication();
}
}
}
You don't need to call this.StopAsync because that will (eventually) be called by StopApplication.

Start Service Bus Client from BackgroundService

I have a ServiceBusClient class that creates a QueueClient which is used to listen for messages on a bus. I have looked at the following articles to set this up:
Background tasks (Microsoft)
Hosted services (Microsoft)
Async and Await
My ServiceBusClient class that handles the QueueClient looks like this:
public class ServiceBusClient : IServiceBusClient
{
public ServiceBusClient(IEventService eventService, ServiceBusClientOptions options)
{
...
queueClient = new QueueClient(options.ConnectionString, options.QueueName);
}
public void Run()
{
RegisterOnMessageHandler();
}
private void RegisterOnMessageHandler()
{
...
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
var eventMessage = EventMessage.FromMessage(message);
await eventService.Write(eventMessage);
if (!token.IsCancellationRequested)
{
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
}
private Task ExceptionReceivedHandler(ExceptionReceivedEventArgs exceptionReceivedEventArgs)
{
// log errors
...
return Task.CompletedTask;
}
}
I was hoping to launch from an IHostedService or even by extending the BackgroundService. In the examples I find, work is constantly being executed in a while loop which does not fit my scenario since I am only trying to run a single command.
So I created a super simple implementation like this:
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
serviceBusClient.Run();
while (!cancellationToken.IsCancellationRequested)
{
// empty loop to keep running for lifetime of pod
}
}
If removing the async I obviously need to return something. I tried Task.CompletedTask but that required me to change the return type to Task<Task>.
If I have the async in place, I will need to await something, but I am not sure what.
This does not feel right. I would assume I would need to change something in the ServiceBusClient, but I am unsure what, since the ProcessMessagesAsync is async and does the heavy lifting in the background from my understanding.
All I want is for my web app to start listening for messages until it dies. How can I do that?
I gave up on using BackgroundService and implemented IHostedService instead.
public class MessageListenerService : IHostedService
{
private readonly IServiceBusClient client;
private readonly ITelemetryClient applicationInsights;
public MessageListenerService(IServiceProvider serviceProvider)
{
client = serviceProvider.GetService<IServiceBusClient>();
applicationInsights = serviceProvider.GetService<ITelemetryClient>();
}
public Task StartAsync(CancellationToken cancellationToken)
{
applicationInsights.TrackTrace(new TraceTelemetry("MessageListenerService is starting"));
client.Run();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
applicationInsights.TrackTrace(new TraceTelemetry("MessageListenerService is stopping"));
return client.Stop();
}
}
If you find issues with this code please let me know in the comments and I'll update as appropriate.
In the end we created a console app for it anyway.

Configuring Quartz.Net to stop a job from executing, if it is taking longer than specified time span

I am working on making a scheduler, just like Windows Scheduler using Quartz.Net.
In Windows Scheduler, there is an option to stop a task from running if it takes more than the specified time. I have to implement the same in my scheduler.
But I am not able to find any extension method/setting to configure Trigger or Job accordingly.
I request some inputs or suggestions for it.
You can write small code to set a custom timout running on another thread. Implement IInterruptableJob interface and make a call to its Interrupt() method from that thread when the job should be interrupted. You can modify the following sample code as per your need. Please make necessary checks/config inputs wherever required.
public class MyCustomJob : IInterruptableJob
{
private Thread runner;
public void Execute(IJobExecutionContext context)
{
int timeOutInMinutes = 20; //Read this from some config or db.
TimeSpan timeout = TimeSpan.FromMinutes(timeOutInMinutes);
//Run your job here.
//As your job needs to be interrupted, let us create a new task for that.
var task = new Task(() =>
{
Thread.Sleep(timeout);
Interrupt();
});
task.Start();
runner = new Thread(PerformScheduledWork);
runner.Start();
}
private void PerformScheduledWork()
{
//Do what you wish to do in the schedled task.
}
public void Interrupt()
{
try
{
runner.Abort();
}
catch (Exception)
{
//log it!
}
finally
{
//do what you wish to do as a clean up task.
}
}
}

Resources