End-to-end logging between multiple components using Application Insights (AI) is based on a hierarchical Request-Id header. So each component is responsible to honer a possible incoming Request-Id. To get the full end-to-end hierarchical flow correct in Application Insights the Request-Id header needs to be used as the AI Operation.Id and Operation.ParentId (as described here).
But when making a request with a Request-Id header to a Azure Function using a HttpTrigger binding for example (Microsoft.NET.Sdk.Functions 1.0.24) with integrated Application Insights configured (as described here) a completely new Operation.Id is created and used - causing the whole flow in AI to be lost. Any ideas on how to get around this?
Setting up a separate custom TelemetryClient might be an option. But seems to require a lot of configuration to get the full ExceptionTrackingTelemetryModule and DependencyTrackingTelemetryModule right - especially when using Functions v2 and Core (ref to AI config). Anyone go that successfully working?
This is not yet supported by Functions but should start working sometime early next year.
If you want to hack it through, you can add a reference to ApplicationInsights SDK for AspNetCore (v 2.4.1) and configure RequestTrackingTelemetryModule.
static Function1()
{
requestModule = new RequestTrackingTelemetryModule();
requestModule.Initialize(TelemetryConfiguration.Active);
}
private static RequestTrackingTelemetryModule requestModule;
This is pretty sketchy, not fully tested and has drawbacks. E.g. request collected is no longer augmented with functions details (invocation id, etc). To overcome it you need to get the real TelemetryConfiguration from the Function dependency injection container and use it to initialize the module.
It should be possible, but is blocked by some issue.
But even with the code above, you should get requests that respect incoming headers and other telemetry correlated to the request.
Also, when out-of-the-box support for correlation for http request is rolled out, this may break. So this is a hacky temporary solution, use it only if you absolutely have to.
Related
I have legacy .NET 4.6.2 application that had to be monitored with OpenTelemetry. After going back and forth with implementation, I managed to send traces to Kibana, however the most of the them contain endpoint addresses like this - api/{controller}/{action}, and not the actual endpoint.
I am investigating this and only in couple of occasions here and there is a word about OpenTelemetry is not handling properly endpoints that have route set in a attribute, like:
[Route("api/products/{productCode}")]
public HttpResponseMessage GetProductData(string code)
{
\\some code
}
There is no details how to be fixed, or I am not able to find them.
Is there anything that could be done in this case? Legacy system that I am working on contains around 20 controllers and more than 80 endpoints, so managing routes would be very challenging.
Edit: I have forgot to mention that this behavior doesn't happened in console exporter.
I am using Elastic APM agent (https://www.elastic.co/guide/en/apm/agent/dotnet/current/index.html) to instrument an ASP.NET MVC Application. I added the nuget packages and added the module entry in the web.config.
I am able to get data in the Kibana APM tab and nicely shows the time spent by each call. (see screenshot below).
Mu Question is: How can I drill down inside each of these calls to see where the time is spent in the stackstace? Is there something I am missing?
There are basically 2 ways the agent captures things:
Auto-instrumentation: in this case you don't write any code, the agent just captures things for you - this is what we see on your screenshot
Manual code instrumentation - for this you can use the Public Agent API and capture things programatically.
In a typical ASP.NET Classic MVC application the agent has auto instrumentation for outgoing HTTP calls with HttpClient, Database calls with EF6 (Make sure to add the interceptor) (SqlClient support is already work-in-progress, hopefully released soon). So unless you have any of these within those requests, the agent won't capture things out of the box.
If you want to capture more things, currently the way to go is to place some agent specific code - so basically manual code instrumentation - into your application and use the public agent API.
Background (TLDR: I need parallel queries)
I am building REST service that needs to be able to answer queries very fast.
As such I'm pre-loading a large part of the database into memory and answering using that data instead of making complex database queries for each request. This works great, and the average response time of the API is well below the requirements and a lot faster than direct database queries.
But I have a problem. The service takes about 5 minutes to start and pre-load all of its information. During this time it can not answer queries.
Problem
I want to change this so that during the pre-load phase it makes database queries until the in-memory cache is loaded.
This leads me to a problem. I need to have multiple active queries to my database. Anyone who has tried this in EF Core has problably seen this message.
System.InvalidOperationException: A second operation started on this context before a previous operation completed. This is usually caused by different threads using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
The first sentence on the linked page is
Entity Framework Core does not support multiple parallel operations
being run on the same DbContext instance.
I thought this would be easily solved by wrapping my cache-loading into its own class and the direct query into another, and then having both of these requiring their own instance of the Database Context. Then my service can in turn get these injected and use both of these dependencies in parallel.
This should be what I have:
I have also set up my database context so that it uses transient for all parts.
services.AddDbContext<IDataContext, DataContext>(options =>
options.UseSqlServer(connectionString), ServiceLifetime.Transient, ServiceLifetime.Transient
);
I have also enabled MultipleActiveResultSets=True
All of this however results in the exact same error as listed above.
Again, everything is Transient except the HandlerService which is Singelton as I want this to keep a copy of the cache in memory and not have to load it for every request.
What is it I have failed to understand about the ef-core database context, or DI in general?
I figured out what the problem was. In my case there is as described above, one singleton handler. This handler has one (indirect) context (through DI) for fulfilling requests until the cache is loaded. When multiple parallel queries are sent to the API before the cache is loaded, then this error occurs as each of these request are using the same context. And in my test I was always hitting the parallel requests as part of the startup and hence the singelton service was trying to use the same db context for multiple requests. My solution is to in this one place step outside the "normal" dependency injection and use the IServiceScopeFactory to get a new instance of the dependency used to resolve requests before the cache is loaded. Bohdans answer led me to this conclusion and ultimate solution.
I'm not sure whether it qualifies for a full answer but it's too broad for a comment.
When doing .NET core background services which are obviously singletons too I use IServiceScopeFactory to create services with a limited lifetime.
Here's how I create a context
using (var scope = _scopeFactory.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<DbContext>();
}
My guess is that you could inject it in your hander and use it like this too. So it would allow you to leave context as scoped instead of transient with is default setting btw.
Hope that helps.
I'm working on a project structured as N-Tier / 3-Tier Architectural Style. The layers (client-server) communicate with each other by HTTP and WCF. I want to use AppInsights to track everything in the project. However, I'm confused about where in the project to create or add AppInsight packages. I have created two instances of resources under one resource group on Azure. Then, I configured inst. key and other things regarding projects in the solution. Although I overrode handle attribute with mine(as explained in AppInsights docs), I couldn't see all call methods and exception tracks from client to server along with layers. Is it not possible to track exceptions in that project type or I'm mistaken?
UPDATE for additional info to define problem clearly: When I use one resource and one instrumentation key, AppInsights defines well whole project and shows relation of my solution and its dependencies. But, when one request failed due to exception in the server, I can't reach whole exception trace, just last method where request is asked from server. Because of that I use two resource instance, one for server and one for client(not real client, its like middle layer).
if you are using multiple apps/keys and multiple layers, you might need to introduce custom code to set/pass through something like a correlationId through http headers, and code that reads and uses that as application insights's "operation id". If you do the work across different applications/ikeys, searching across the apps will become more complicated.
instead, you might want to use just one ikey across all the layers, and instead use different ikeys for different environments, like dev/staging/prod instead.
there are some examples of reading/using operationId, like this blog post:
http://blog.marcinbudny.com/2016/04/application-insights-for-owin-based.html#.WBkL24WcFaQ
where someone does something similar with OWIN.
For Exception tracking, if the exceptions are happening at one of the other asp.net layers, you might need to enable some extra settings to collect exceptions or more information for them:
http://apmtips.com/blog/2016/06/21/application-insights-for-mvc-and-mvc-web-api/
I'm very new to Application Insights, and I'm thinking of using it for a set of services I plan on implementing with asp.net webapi. I was able to get the basic telemetry up and running very easily (right-clicking on a project on VS, Add Application Insights), but then I hit a block. I plan to have a correlation id set in the request headers for calls to downstream services, and I would like to tag all the telemetry related to one outside call with the same correlation id.
So far I've found that there is a way to configure a TelemetryInitializer, but if I understood correctly, this is run before I get to access the request, meaning I can't check if there is a correlation id that I should attach.
So I guess there might be 2 ways to solve this: 1) if I can somehow actually get access to the request headers before the initializer, that would obviously solver the problem, or 2) somehow get a hold of the TelemetryClient instance that is used to report the automatically generated telemetry.
Perhaps the last resort would be to turn off all of the automatic stuff and do all of it manually, when I could of course control what properties are set on the TelemetryClient. But this would be quite a lot more work, so I'd prefer to find some other solution.
You were rights saying that you should use TelemetryInitializer. All TelemetryInitializers are called when Track method is called on any telemetry item. Autogenerated request telemetry is "tracked" on request OnEnd, you should have all your custom headers available for you at that time.
Please also have a look at OperationId - this is part of the standard context managed by App Inisghts and is used exactly for the purpose of correlating requests with downstream execution. This is created and passed automatically, including traces (if you use trackTrace).
Moreover, we have built-in support in our UX for easily seeing all telemetry for a particular operation - it can be found in "Search->Details-->Related Items-->All telemetry for this operation"