TelemetryClient.Flush() takes about 2 minutes to run during application exit - azure-application-insights

Question

I have a .NET Core 3.1 console application running as a hosted service. When the application shuts down, it hangs for about 2 minutes. When breaking in I can see that it is hanging on TelemetryClient.Flush(), specifically inside InMemoryChannel.Flush().
So I watch the result in Fiddler and I can see the application is making a request out to https://dc.services.visualstudio.com/v2/track to report the telemetry, but that service doesn't respond. Eventually after 2 minutes the response fails with 500 and in the web response: "IIS 10.0 Detailed Error - 500.1013 - Internal Server Error".
I cannot tell if this is something I am doing wrong or not. So I shortened the application to the bare minimum.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
namespace DotNetCoreConsoleAppTest
{
public class Program
{
public static async Task Main()
{
await Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services
.Configure<TelemetryConfiguration>(options =>
{
options.InstrumentationKey = "<put your key here>";
})
.AddSingleton<TelemetryWriter>()
.AddHostedService<ProgramHost>();
})
.RunConsoleAsync()
.ConfigureAwait(false);
}
}
public class TelemetryWriter : IDisposable
{
private readonly TelemetryClient _telemetryClient;
public TelemetryWriter(IOptions<TelemetryConfiguration> telemetryConfiguration)
{
_telemetryClient = new TelemetryClient(telemetryConfiguration.Value);
}
public void WriteEvent() => _telemetryClient.TrackEvent("test");
public void Dispose() => _telemetryClient.Flush();
}
public class ProgramHost : IHostedService
{
private readonly TelemetryWriter _telemetryWriter;
private readonly IHostApplicationLifetime _hostApplicationLifetime;
public ProgramHost(
TelemetryWriter telemetryWriter,
IHostApplicationLifetime hostApplicationLifetime)
{
_telemetryWriter = telemetryWriter;
_hostApplicationLifetime = hostApplicationLifetime;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_telemetryWriter.WriteEvent();
_hostApplicationLifetime.StopApplication();
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
}
}
Some content from the web response:
<div class="content-container">
<fieldset><h4>Detailed Error Information:</h4>
<div id="details-left">
<table border="0" cellpadding="0" cellspacing="0">
<tr class="alt"><th>Module</th><td> iisnode</td></tr>
<tr><th>Notification</th><td> ExecuteRequestHandler</td></tr>
<tr class="alt"><th>Handler</th><td> iisnode</td></tr>
<tr><th>Error Code</th><td> 0x0000006d</td></tr>
</table>
</div>
<div id="details-right">
<table border="0" cellpadding="0" cellspacing="0">
<tr class="alt"><th>Requested URL</th><td> https://dc.services.visualstudio.com:443/InputServer.js</td></tr>
<tr><th>Physical Path</th><td> E:\sitesroot\0\InputServer.js</td></tr>
<tr class="alt"><th>Logon Method</th><td> Anonymous</td></tr>
<tr><th>Logon User</th><td> Anonymous</td></tr>
<tr class="alt"><th>Request Tracing Directory</th><td> C:\Resources\directory\f3eec886680f474eb56deb0e59f20036.Breeze.DiagnosticStore\FailedReqLogFiles\Web</td></tr>
</table>
<div class="clear"></div>
</div>
</fieldset>
</div>
The hanging stack trace is:
System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.GetResult() Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.DequeueAndSend(System.TimeSpan timeout) Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Flush(System.TimeSpan timeout) Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryChannel.Flush(System.TimeSpan timeout) Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.Channel.InMemoryChannel.Flush() Unknown
Microsoft.ApplicationInsights.dll!Microsoft.ApplicationInsights.TelemetryClient.Flush() Unknown
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.TelemetryWriter.Dispose() Line 43 C#
Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync() Unknown
Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync() Unknown
Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(Microsoft.Extensions.Hosting.IHost host, System.Threading.CancellationToken token) Unknown
Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.RunConsoleAsync(Microsoft.Extensions.Hosting.IHostBuilder hostBuilder, System.Threading.CancellationToken cancellationToken) Unknown
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.Program.Main() Line 16 C#
DotNetCoreConsoleAppTest.dll!DotNetCoreConsoleAppTest.Program.<Main>() Unknown
Despite all of this, the events DO appear in the logs inside of Application Insights. The only problem is my host application hangs. Is this a problem with the way I am trying to Flush()? Or is this an Application Insights service problem?

Answer 1

ZakiMa answered my question -- App Insights was having an outage.