Hello all (and most probably #mookid8000),
I am using Rebus C# library with Azure Storage queue.
Everything works fine, just have one issue with logging.
Rebus is making several GetMessage calls to queue per second.
That is extremely problematic, because it generates GB of data per day if "queue" operations are being logged to Log Analytics, which cost a lot money.
Is there a way to tell rebus to check the queue only each 10s, or each minute if last response return no new messages? My app doesn't need fast message handling, so slower refresh period between each call to queue is preferable anyway.
Maybe it can be configured somewhere here?
services.AddRebus((configure, sp) => configure
.Transport(t =>
{
CloudStorageAccount storageAccount = CloudStorageAccount
.Parse(Configuration.GetValue<string>("StorageQueue:ConnectionString"));
t.UseAzureStorageQueues(storageAccount, "queue");
})
.Routing(r =>
{
var routing = r.TypeBased();
routing.Map<TrackedObject>("queue");
})
);
I was checking if I can found some documentation about this, but could not found anything :(
Thank you for any help.
You're looking for how to configure Rebus' back-off times 🙂
With it, you could do something like this
services.AddRebus(
configure => configure.
.(...)
.Options(o => {
o.SetBackoffTimes(
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(10)
);
})
);
and have it wait 10 s between polling for messages after 3 seconds of running idle.
Related
I have some fairly typical SQL calls in an app that look something like this (Dapper typically in the middle), .NET 6:
var connection = new SqlConnection("constring");
using (connection)
{
await connection.OpenAsync();
var command = new SqlCommand("sql");
await command.ExecuteAsync();
await connection.CloseAsync();
connection.Dispose();
}
A request to the app probably generates a half-dozen calls like this, usually returning in <0 to 10ms. I almost never see any SQL usage (it's SQL Azure) beyond a high of 5%.
The problem comes when a bot hits the app with 50+ simultaneous requests, coming all within the same 300 or so milliseconds. This causes the classic error
InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached
I have the following things in place:
I have the connection string set to a max pool size of 250.
I'm running three nodes as an Azure App Service.
The call stacks are all async.
I do have ARR Affinity on because I'm using SignalR, but I assume the load balancer would spread out the requests as the bot likely isn't sending ARR cookies.
The app services and SQL Server do not break a sweat even with these traffic storms.
Here's the question: How do I scale this? I assume human users don't see this and the connection pool exhaustion heals quickly, but it creates a lot of logging noise. The App Service and SQL Server instance are not at all stressed or working beyond their limits, so it appears it's the connection pool mechanics that are a problem. They're kind of a black box abstraction, but a leaky abstraction since I clearly need to know more about them to make them work right.
Here's the question: How do I scale this?
.NET 6 introduced Rate Limiting, which is really the right solution here. Test how many concurrent requests your API app and database can comfortably handle, and stall or reject additional requests.
Take the analogy of an Emergency Room. Do you want to let everyone into the back who walks in the door? No once all the rooms are full, you make them wait in the waiting room, or send them away.
So put in a request throttle like:
builder.Services.AddRateLimiter(options =>
{
options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(httpContext =>
RateLimitPartition.GetFixedWindowLimiter(
partitionKey: httpContext.Request.QueryString.Value!,
factory: partition => new FixedWindowRateLimiterOptions
{
AutoReplenishment = true,
PermitLimit = 50,
QueueLimit = 10,
Window = TimeSpan.FromSeconds(10)
}));
options.OnRejected = (context, cancellationToken) =>
{
context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
return new ValueTask();
};
});
We're using Firebase for our app that needs to process a some data and then send out a series of e-mails after their data has been decided.
Right now I'm triggering a single handler via CRON (which uses pub/sub) that processes the data and then publishes a series of messages to a different pub/sub topic. That topic in turn has a similar trigger function that goes through a few processes and then sends an single email per execution.
// Triggered by CRON task
const cronPublisher = functions.pubsub.topic('queue-emails').onPublish(async () => {
//processing
...
// Publish to other topic
await Promise.all(
emails.map((email) =>
publisher.queueSendOffer(email)
)
);
});
// Triggered by above, at times twice
const sendEmail = functions.pubsub.topic('send-email').onPublish(async () => {
//processing and send email
});
The issue I'm running into is that the 2nd topic trigger at times is executed more than once, sending two identical emails. The main potential cause I've come across by way of Google just involves long execution times resulting in timeouts, and retries. This shouldn't be the case since our acknowledgment timeout is configured to 300 seconds and the execution times never exceed ~12 seconds.
Also, the Firebase interface doesn't seem to give you any control over how this acknowledgment is sent.
This CRON function runs everyday and the issue only occurs every 4-5 days, but then it duplicates every single email.
Any thoughts?
Appreciated.
If 'every single message' is duplicated, perhaps it is your 'cronPublisher' function that is being called twice? Cloud Pubsub offers at least once semantics, so your job should be tolerant to this https://cloud.google.com/pubsub/docs/subscriber#at-least-once-delivery.
If you were to persist some information in a firebase transaction that this cron event had been received, and check that before publishing, you could prevent duplicate publishing to the "send-email" topic.
I read here endpoint spin-up is supposed to be transparent, which I assume means cold start times should not differ from regular execution times. Is this still the case? We are getting extremely slow and unusable cold start times - around 16 seconds - across all endpoints.
Cold start:
Function execution took 16172 ms, finished with status code: 200
After:Function execution took 1002 ms, finished with status code: 304
Is this expected behaviour and what could be causing it?
UPDATE: The cold start times seem to no longer be an issue with node 8, at least for me. I'll leave my answer below for any individuals curious about keeping their functions warm with a cron task via App Engine. However, there is also a new cron method available that may keep them warm more easily. See the firebase blog for more details about cron and Firebase.
My cold start times have been ridiculous, to the point where the browser will timeout waiting for a request. (like if it's waiting for a Firestore API to complete).
Example
A function that creates a new user account (auth.user().onCreate trigger), then sets up a user profile in firestore.
First Start After Deploy: consistently between 30 and 60 seconds, frequently gives me a "connection error" on the first try when cold (this is after waiting several seconds once Firebase CLI says "Deploy Complete!"
Cold Start: 10 - 20 seconds
When Warm: All of this completes in approximately 400ms.
As you can imagine, not many users will sit around waiting more than a few seconds for an account to be setup. I can't just let this happen in the background either, because it's part of an application process that needs a profile setup to store input data.
My solution was to add "ping" function to all of my API's, and create a cron-like scheduler task to ping each of my functions every minute, using app engine.
Ensure the ping function does something, like access a firestore document, or setup a new user account, and not just respond to the http request.
See this tutorial for app engine scheduling: https://cloud.google.com/appengine/docs/flexible/nodejs/scheduling-jobs-with-cron-yaml
Well it is about resource usage of Cloud Functions I guess, I was there too. While your functions are idle, Cloud Functions also releases its resources, at first call it reassignes those resources and at second call you are fine. I cannot say it is good or not, but that is the case.
if you try to return a value from an async function there won't be any variables in the main function definition (functions.https.onCall) and GCP will think that the function has finished and try to remove resources from it.
Previous breaking code (taking 16 + seconds):
functions.https.onCall((data, context) => {
return asyncFunction()
});
After returning a promise in the function definition the function times are a lot faster (milliseconds) as the function waits for the promise to resolve before removing resources.
functions.https.onCall((data, context) => {
return new Promise((resolve) => {
asyncFunction()
.then((message) => {
resolve(message);
}).catch((error) => {
resolve(error);
});
});
});
I have been playing around with Rebus and RabbitMQ, and came across a scenario I cannot seem to get working.
I have a couple of queues; queue1 & queue2 and they take the same class/message type. Now, Rebus seems to prefer different message types per queue, this is not an option for me right now, so i use the advanced routing bus.Advanced.Routing.Send("queue1", Message)
I would like to utilise the bus.defer functionality but am unsure how to combine them both. I know I might need to introduce a waiting queue as an external timeout manager (which I have yet to get working too, but thats for another day)
Has anyone done anything similar?
How to send the message
As you have probably discovered, when you bus.Defer, Rebus will use the endpoint mappings to look up the destination queue from the type of the message being deferred (which is analogous to bus.Send/bus.SendLocal, in that it has an accompanying bus.DeferLocal too, which always sends to the sender's own input queue).
What is missing, is something analogous to bus.Advanced.Routing.Send, but fortunately it is pretty easy to emulate a combination of bus.Defer and an explicitly routed message but setting the rbs2-deferred-recipient header on a message:
var headers = new Dictionary<string, string> {
{Headers.DeferredRecipient, "destination-queue"}
};
var delay = TimeSpan.FromMinutes(5);
await bus.DeferLocal(delay, yourMessage, headers);
How to configure the timeout manager
You can use Rebus' internal timeout manager by configuring some kind of timeout persistence – e.g. by pulling in Rebus.SqlServer and using SQL Server to store timeouts like so:
Configure.With(...)
.(...)
.Timeouts(t => t.StoreInSqlServer(...))
.Start();
Another option is to install a Rebus endpoint as a dedicated timeout manager, which simply uses the same configuration as can be seen above, and then all other endpoints do this:
Configure.With(...)
.(...)
.Timeouts(t => t.UseExternalTimeoutManager("timeouts"))
.Start();
assuming that your timeout manager uses the timeouts queue.
Update relevant from Rebus 5
Rebus 5 (which is currently available as a prerelease package on Nuget.org) has builtin support for deferring messages to an explicitly specified destination queue.
It can be done like this:
var delay = TimeSpan.FromMinutes(2);
await bus.Advanced.Routing.Defer("dest-queue", delay, message);
which will simply carry out the steps mentioned above underneath the covers.
I have a web service that can be broken down into two main sections:
[WebMethod]
MyServiceCall()
{
//Do stuff the client cares about
//Do stuff I care about
}
What I'd like to do is run that 2nd part on another thread, so that the client isn't waiting on it: once the user's logic has completed, send them their information immediately, but continue processing the stuff I care about (logging, etc).
From a web service, what is the recommended way of running that 2nd piece asynchronously, to get the user back their information as quickly as possible? BackgroundWorker? QueueUserWorkItem?
You may want to look into Tasks which are new to .NET 4.0.
It lets you kick off an asynchronous operation, but also gives you an easy way to see if it's done or not later.
var task = Task.Factory.StartNew(() => DoSomeWork());
It'll kick off DoSomeWork() and continue without waiting so you can continue doing your other processing. When you get to the point where you don't want to process anymore until your asynchronous task has finished, you can call:
task.Wait();
Which will wait there until the task has completed. If you want to get a result back from the task, you can do this:
var task = Task.Factory.StartNew(() =>
{
Thread.Sleep(3000);
return "dummy value";
});
Console.WriteLine(task.Result);
A call to task.Result blocks until the result is available.
Here's a tutorial that covers Tasks in greater detail: http://www.codethinked.com/net-40-and-systemthreadingtasks
The easiest way to fire off a new thread is probably:
new Thread(() =>
{
/// do whatever you want here
}).Start();
Be careful, though - if the service is hit very frequently, you could wind up creating a lot of threads that block each other.