I have an asp.net web page that obtains data from two web services, the main one and the fall-back one. Now those services are called consecutively. When running on my box the page render time is usually 6-10 s.
I decided to parellelize those calls so that the fall-back data is available faster if the main service fails.
Result res = null;
Task<Result> task = new Task<Result>(() => service1.method1());
task.Start();
res = service2.method();
if(res == null || res.Count == 0)
res = task.Result;
return res;
With that change the page render time grew significantly and reaches 60 s. I added some profiling code and I see that this piece of code typically runs in less than 3 s.
So I suspect that using Tasks somehow hinders the asp.net infrastructure performance.
What might cause that severe performace degradation?
Is there an better way of running those two external service calls in parallel?
Edit 1: I'm mostly interested in the result from service2. The result from service1 will be used only if service2 fails. However, I want to minimize the waiting time for the service1 result if service2 fails.
It turned out that the code executed within a Task called a lazily initialized singleton. The singleton's constructor tried to set up some caching if HttpContext.Current was available. Because HttpContext.Current was not available within a Task, the singleton did not use caching and this degraded performance.
Related
I have ASP.Net 4.7.2 window service which is processing NServiceBus messages. Currently it is deployed to On-Premise server. It has retry mechanism as well and working fine. Now I am going to containerizing it. While running into docker window container, it is doing SQL operation using Entity framework and giving exception as mentioned below:
The configured execution strategy 'SqlRetryingExecutionStrategy' does not support user-initiated transactions. Use the execution strategy returned by 'DbContext.Database.CreateExecutionStrategy()' to execute all the operations in the transaction as a retriable unit.
While running locally by installing manually or on On-Premise server, it is working fine but in container it is throwing exception.
Can any one help me what can be the root cause?
It sounds like the piece of code does manual transaction management and is not wrapped within an execution strategy execute.
if your code initiates a transaction using BeginTransaction() you are defining your own group of operations that need to be treated as a unit, and everything inside the transaction would need to be played back shall a failure occur.
The solution is to manually invoke the execution strategy with a delegate representing everything that needs to be executed. If a transient failure occurs, the execution strategy will invoke the delegate again.
https://learn.microsoft.com/en-us/ef/core/miscellaneous/connection-resiliency#execution-strategies-and-transactions
using var db = new SomeContext();
var strategy = db.Database.CreateExecutionStrategy();
strategy.Execute(
() =>
{
using var context = new SomeContext();
using var transaction = context.Database.BeginTransaction();
context.SaveChanges();
transaction.Commit();
});
``
We have a .NET application which is calling over OpenRia services on the server (IIS). This web service call is running a heavy calculation, where we are loading over LoadLibrary some DLL's, which we need to solve some linear systems. We need to go over a list of 1000 events. Every single event is a separate calculation and can be run independently from each other.
What we are doing is, that we create on a 64-core machine 60 tasks and every task is taking one event => run the calculation => take the next event => run the calculation and so on until the list is empty.
As soon the list is empty our calculation is finished.
We have now the strange behaviour that on the first run the calculation seems to run fast, but when we run the same calculation again it's getting slower on every run.
If we restart the server the calculation is running fast again.
We have done an analysis with PerfView and we have seen that on the second/third/fourth run the used threads from the IIS worker process are less than at the beginning.
On the first run the IIS worker process is using 60 threads (as we have defined) and on the second the process is using less than 60. On every run the actual threads used are less and less.
The first run the calculation needs around 3min. The second run we need 6min and the third run we are already around 15min.
What could be the problem? I have tried to use the ThreadPool, but I have the same effect as with the Tasks.
Here is some sample code:
//This part of code is called after the web service call
ConcurrentStack<int> events = new ConcurrentStack<int>();//This is a list of 1000 entries
ParallelOptions options = new ParallelOptions();
int interfacesDone = 0;
Task[] tasks = new Task[options.MaxDegreeOfParallelism];
for (int i = 0; i < options.MaxDegreeOfParallelism; i++)
{
tasks[i] = Task.Run(() =>
{
StartAnalysis(events);
});
}
Task.WaitAll(tasks);
private void StartAnalysis(ConcurrentStack<int> events)
{
while (!events.IsEmpty)
{
int index;
if (events.TryPop(out index))
{
DoHeavyCalculation();
}
}
}
ASP.NET processes requests by using threads from the .NET thread pool. The thread pool maintains a pool of threads that have already incurred the thread initialization costs.
Therefore, these threads are easy to reuse. The .NET thread pool is also self-tuning. It monitors CPU and other resource utilization, and it adds new threads or trims the thread pool size as needed.
I am considering using Hangfire https://www.hangfire.io to replace an older home-grown scheduling ASP.NET web site/app.
I have created a simple test project using Hangfire. I am able to start the project with Hangfire, submit (in code) a couple of very simple single and recurring tasks, view the dashboard, etc.
I'm looking for more suggestions for creating a little more complex code (and classes) for tasks to be scheduled, and I have a question about what happens with permanently scheduled tasks when re-publishing a Hangfire site to production.
I have read some of the documentation on the Hangfire site, reviewed the 2 tutorials, scanned the Hangfire forums, and searched StackOverflow and the web a bit. A lot of what I have seen shows you how to schedule something very simple (like Console.WriteLine), but nothing more complex. The "Highlighter" tutorial was useful, but that essentially shows how to schedule a single instance of a (slightly longer-running) task in response to an interactive user input. I understand how useful that can be, but I'm more interested in recurring tasks that are submitted and then run every day (or every hour, etc.) and don't need to be submitted again. These tasks could be for something like sending a batch of emails to users each night, batch processing some data, importing a nightly feed of external data, periodically calling a web service to perform some processing, etc.
Is there any sample code available that shows some examples like this, or any guidance on the most appropriate approach for structuring such code in an interface and class(es)?
Secondly, in my case, most of the tasks would be "permanent" (always existing as a recurring task). If I set up code to add these as recurring tasks shortly after starting the Hangfire application in production, how should I handle it when publishing updates to production (when this same initialization would run again)? Should I just call "AddOrUpdate" with the same ID and Hangfire will take care of it? Should I first call "RemoveIfExists" and then add the recurring task again? Is there some other approach that should be used?
One example would be a log janitor, which would run every weekday # 5:00PM to remove logs that are older than 5 days.
public void Schedule()
{
RecurringJob.AddOrUpdate<LogJanitor>(
"Janitor - Old Logs",
j => j.OnSchedule(null),
"0 17 * * 1,2,3,4,5",
TimeZoneInfo.FindSystemTimeZoneById("CST"));
}
Then we would handle it this way
public void OnSchedule(
PerformContext context)
{
DateTime timeStamp = DateTime.Today.AddDays(-5);
_logRepo.FindAndDelete(from: DateTime.MinValue, to: timeStamp);
}
These two methods are declared inside LogJanitor class. When our application starts, we get an instance of this class then call Schedule().
All,
I am using Change Feed Processor Library.Want to know the best way to handle service failure along with the exceptions/errors scenario's in ProcessChangesAsync method. Below are the events am referring to.
1) Service failure - Service having the processor library crashed in the middle of some operation. How to start the process from the same document(doc on failure instance)? is there any inbuilt mechanism where change feed will start with the last failed documents? E.g. Let assume,in current batch we have 10 docs.5 processed successfully and then service breaks because of network failure or by some other reasons.Will my process starts with 6th document once service is re-started? How to achieve this?
2) Exception and Errors- Any errors in ProcessChangesAsync method can be handle using try catch at the global level but how to persist those failure records and make them available for the next batch? Again,looking for any available inbuilt mechanism in change feed process.
1) The Processor Library, by default, checkpoints after a successful run of ProcessChangesAsync. In the latest library version, you can customize the Checkpointer to do manual checkpoints in case you need it. If for some reason the processor shuts down before checkpointing, then it will start processing next from the the last successful checkpoint stored in the Leases collection. In your case, it will start with the first document again, so you will never lose a change but you could experience double processing (this is an "at least once" model).
2) There is no built-in mechanism that you can leverage, handling exceptions within the ProcessChangesAsync is your responsibility. You could not only add a global try/catch but, in the case you are looping over the documents, add a try/catch inside the loop, to handle a failing document (maybe send it to queue for later analysis/post-process) without losing the batch. If you require logging for those errors (I'm assuming that's what you mean by persisting errors?), then the latest version is compatible with LibLog, so plugging your own custom logging is as simple as:
using Microsoft.Azure.Documents.ChangeFeedProcessor.Logging;
var hostName = "SampleHost";
var tracelogProvider = new TraceLogProvider(); //You can use any provider supported by LibLog
using (tracelogProvider.OpenNestedContext(hostName))
{
LogProvider.SetCurrentLogProvider(tracelogProvider);
// After this, create IChangeFeedProcessor instance and start/stop it.
}
Source
Extra info for the comments
To avoid exceptions halting the batch or causing a batch to be reprocessed, you can have handling like this:
public async Task ProcessChangesAsync(IChangeFeedObserverContext context, IReadOnlyList<Document> documents, CancellationToken cancellationToken)
{
try
{
foreach(var document in documents)
{
try
{
// Do your work for the document
}
catch(Exception ex)
{
// Something happened with the current document, handle it, send it to a queue / another storage to analyze, log it. This catch will make the loop continue with the next.
}
}
}
catch(Exception ex)
{
// Something unhandled happened, log it and avoid throwing it again so the next batch is processed
}
}
Consider the normal scenario where an ASP.NET Core Web API application executes the service Controller action, but instead of executing all the work under the same thread (thread pool thread) until the response is created, I would like to use non-pooled threads (ideally pre-created) to execute the main work, either by scheduling one of these threads from the initial action pooled thread and free the pooled thread for serving other incoming requests, or passing the job to a pre-created non-pooled thread.
Among other reasons, the main reason to have these non-pooled and long running threads is that some requests may be prioritized and their threads put on hold (synchronized), thus it would not block new incoming requests to the API due to thread pool starvation, but older requests on hold (non-pooled threads) may be waked up and rejected and some sort of call back to the thread pool to return the web response back to the clients.
In summary, the ideal solution would be using a synchronization mechanism (like .NET RegisterWaitForSingleObject) where the pooled thread would hook to the waitHandle but be freed up for other thread pool work, and a new non-pooled thread would be created or used to carry on the execution. Ideally from a list of pre-created and idle non-pooled threads.
Seems async-await only works with Tasks and threads from the .NET thread pool, not with other threads. Also most techniques to create non-pooled threads do not allow the pooled thread to be free and return to the pool.
Any ideas? I'm using .NET Core and latest versions of tools and frameworks.
Thank you for the comments provided. The suggestion to check TaskCompletionSource was fundamental. So my goal was to have potentially hundreds or thousands of API requests on ASP.NET Core and being able to serve only a portion of them at a given time frame (due to backend constraints), choosing which ones should be served first and hold the others until backends are free or reject them later. Doing all this with thread pool threads is bad: blocking/holding and having to accept thousands in short time (thread pool size growing).
The design goal was the request jobs to move their processing from the ASP.NET threads to non pooled threads. I plan to to have these pre-created in reasonable numbers to avoid the overhead of creating them all the time. These threads implement a generic request processing engine and can be reused for subsequent requests. Blocking these threads to manage request prioritization is not a problem (using synchronization), most of them will not use CPU at all time and the memory footprint is manageable. The most important is that the thread pool threads will only be used on the very start of the request and released right away, to be only be used once the request is completed and return a response to the remote clients.
The solution is to have a TaskCompletionSource object created and passed to an available non-pooled thread to process the request. This can be done by queuing the request data together with the TaskCompletetionSource object on the right queue depending the type of service and priority of the client, or just passing it to a newly created thread if none available. The ASP.NET controller action will await on the TaskCompletionSouce.Task and once the main processing thread sets the result on this object, the rest of the code from the controller action will be executed by a pooled thread and return the response to the client. Meanwhile, the main processing thread can either be terminated or go get more request jobs from the queues.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
namespace MyApi.Controllers
{
[Route("api/[controller]")]
public class ValuesController : Controller
{
public static readonly object locker = new object();
public static DateTime time;
public static volatile TaskCompletionSource<string> tcs;
// GET api/values
[HttpGet]
public async Task<string> Get()
{
time = DateTime.Now;
ShowThreads("Starting Get Action...");
// Using await will free the pooled thread until a Task result is available, basically
// returns a Task to the ASP.NET, which is a "promise" to have a result in the future.
string result = await CreateTaskCompletionSource();
// This code is only executed once a Task result is available: the non-pooled thread
// completes processing and signals (TrySetResult) the TaskCompletionSource object
ShowThreads($"Signaled... Result: {result}");
Thread.Sleep(2_000);
ShowThreads("End Get Action!");
return result;
}
public static Task<string> CreateTaskCompletionSource()
{
ShowThreads($"Start Task Completion...");
string data = "Data";
tcs = new TaskCompletionSource<string>();
// Create a non-pooled thread (LongRunning), alternatively place the job data into a queue
// or similar and not create a thread because these would already have been pre-created and
// waiting for jobs from queues. The point is that is not mandatory to create a thread here.
Task.Factory.StartNew(s => Workload(data), tcs,
CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default);
ShowThreads($"Task Completion created...");
return tcs.Task;
}
public static void Workload(object data)
{
// I have put this Sleep here to give some time to show that the ASP.NET pooled
// thread was freed and gone back to the pool when the workload starts.
Thread.Sleep(100);
ShowThreads($"Started Workload... Data is: {(string)data}");
Thread.Sleep(10_000);
ShowThreads($"Going to signal...");
// Signal the TaskCompletionSource that work has finished, wich will force a pooled thread
// to be scheduled to execute the final part of the APS.NET controller action and finish.
// tcs.TrySetResult("Done!");
Task.Run((() => tcs.TrySetResult("Done!")));
// The only reason I show the TrySetResult into a task is to free this non-pooled thread
// imediately, otherwise the following line would only be executed after ASP.NET have
// finished processing the response. This briefly activates a pooled thread just execute
// the TrySetResult. If there is no problem to wait for ASP.NET to complete the response,
// we do it synchronosly and avoi using another pooled thread.
Thread.Sleep(1_000);
ShowThreads("End Workload");
}
public static void ShowThreads(string message = null)
{
int maxWorkers, maxIos, minWorkers, minIos, freeWorkers, freeIos;
lock (locker)
{
double elapsed = DateTime.Now.Subtract(time).TotalSeconds;
ThreadPool.GetMaxThreads(out maxWorkers, out maxIos);
ThreadPool.GetMinThreads(out minWorkers, out minIos);
ThreadPool.GetAvailableThreads(out freeWorkers, out freeIos);
Console.WriteLine($"Used WT: {maxWorkers - freeWorkers}, Used IoT: {maxIos - freeIos} - "+
$"+{elapsed.ToString("0.000 s")} : {message}");
}
}
}
}
I have placed the whole sample code so anyone can easily create as ASP.NET Core API project and test it without any changes. Here is the resulting output:
MyApi> Now listening on: http://localhost:23145
MyApi> Application started. Press Ctrl+C to shut down.
MyApi> Used WT: 1, Used IoT: 0 - +0.012 s : Starting Get Action...
MyApi> Used WT: 1, Used IoT: 0 - +0.015 s : Start Task Completion...
MyApi> Used WT: 1, Used IoT: 0 - +0.035 s : Task Completion created...
MyApi> Used WT: 0, Used IoT: 0 - +0.135 s : Started Workload... Data is: Data
MyApi> Used WT: 0, Used IoT: 0 - +10.135 s : Going to signal...
MyApi> Used WT: 2, Used IoT: 0 - +10.136 s : Signaled... Result: Done!
MyApi> Used WT: 1, Used IoT: 0 - +11.142 s : End Workload
MyApi> Used WT: 1, Used IoT: 0 - +12.136 s : End Get Action!
As you can see the pooled thread runs until the await on the TaskCompletionSource creation, and by the time the Workload starts to process the request on the non-pooled thread there is ZERO ThreadPool threads being used and remains using no pooled threads for the entire duration of the processing. When the Run.Task executes the TrySetResult fires a pooled thread for a brief moment to trigger the rest of the controller action code, reason the Worker thread count is 2 for a moment, then a fresh pooled thread runs the rest of the ASP.NET controller action to finish with the response.