I have a ASP.NET 4.0 app that is calling a WCF service. For testing, the closeTimeout, openTimeout, receiveTimeout, and sendTimeout values in the wsHttpBinding binding are all set to 01:00:00.
When I ran a test in which the service took 5 minutes 40 seconds, I could see the correct results of the WCF service in the app event log. However, the app did not process the results.
In subsequent identical tests in which the WCF service took less than 4 minutes, I could see the same correct results in the app event log, but the app processed the results correctly.
I'm thinking there's another timeout setting I don't know about. Any ideas? Thanks.
Try the OperationTimeout property of the WCF service client where it is instantiated:
MyWCFServiceClient client = new MyWCFServiceClient();
client.InnerChannel.OperationTimeout = new TimeSpan(0, 10, 0);
client.Open();
The above will set the timeout to 10 minutes
Related
I have some fairly typical SQL calls in an app that look something like this (Dapper typically in the middle), .NET 6:
var connection = new SqlConnection("constring");
using (connection)
{
await connection.OpenAsync();
var command = new SqlCommand("sql");
await command.ExecuteAsync();
await connection.CloseAsync();
connection.Dispose();
}
A request to the app probably generates a half-dozen calls like this, usually returning in <0 to 10ms. I almost never see any SQL usage (it's SQL Azure) beyond a high of 5%.
The problem comes when a bot hits the app with 50+ simultaneous requests, coming all within the same 300 or so milliseconds. This causes the classic error
InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached
I have the following things in place:
I have the connection string set to a max pool size of 250.
I'm running three nodes as an Azure App Service.
The call stacks are all async.
I do have ARR Affinity on because I'm using SignalR, but I assume the load balancer would spread out the requests as the bot likely isn't sending ARR cookies.
The app services and SQL Server do not break a sweat even with these traffic storms.
Here's the question: How do I scale this? I assume human users don't see this and the connection pool exhaustion heals quickly, but it creates a lot of logging noise. The App Service and SQL Server instance are not at all stressed or working beyond their limits, so it appears it's the connection pool mechanics that are a problem. They're kind of a black box abstraction, but a leaky abstraction since I clearly need to know more about them to make them work right.
Here's the question: How do I scale this?
.NET 6 introduced Rate Limiting, which is really the right solution here. Test how many concurrent requests your API app and database can comfortably handle, and stall or reject additional requests.
Take the analogy of an Emergency Room. Do you want to let everyone into the back who walks in the door? No once all the rooms are full, you make them wait in the waiting room, or send them away.
So put in a request throttle like:
builder.Services.AddRateLimiter(options =>
{
options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(httpContext =>
RateLimitPartition.GetFixedWindowLimiter(
partitionKey: httpContext.Request.QueryString.Value!,
factory: partition => new FixedWindowRateLimiterOptions
{
AutoReplenishment = true,
PermitLimit = 50,
QueueLimit = 10,
Window = TimeSpan.FromSeconds(10)
}));
options.OnRejected = (context, cancellationToken) =>
{
context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
return new ValueTask();
};
});
I am providing a timeout of one second , however when the URL is down it is taking 120+ seconds for the response to come. Is there some variable or something that overrides the timeout in do:url-open ?
Update: I was calling the dp:url-open on request-transformation as well as on response-transformation. So the overriden timeout is 60 sec, adding both side it was becoming 120 sec.
Here's how I am calling this (I am storing the time before and after dp:url-open calls, and then returning them in the response):
Case 1: When the url is reachable I am getting a result like:
Case 2: When url is not reachable:
Update: FIXED: It seems the port that I was using was getting timed-out in the firewall first there it used to spend 1 minute. I was earlier trying to hit an application running on port 8077, later I changed that to 8088, And I started seeing the same timeout that I was passing.
The do:url-open() timeout only affects the operation done in the script but not the service itself. It depends on how you have built the solution but the time-out from the do:url-open() should be honored.
You can check this by setting logs to debug and adding a <xsl:message>Before url-open</xsl:message> and one after to see in the log if it is your url-open call or teh service that waits 120+ sec.
If it is the url-open you have most likely some error in the script and if it is the service that halts the response you need to return from the script (or throw an error depending on your needs) to halt the service.
You can set the time-out for the service itself or set a time-out in the User Agent for the specific URL you are calling as well.
Please note that the time-out will terminate the service after that time if you set it on service level so 1 sec. would not be recommended!
I am creating an ASP.NET Web API using .NET 4.5.2. The API needs to connect to a runspace on startup. I have questions about when this Startup.Configuration method actually runs though. It does not seem to run when I start the website or app pool. It seems to wait until the first time somebody tries to access the website. Is this correct? Also, it seems to run again at random times. I have seen it run after 2 hours, 4 hours, and 16 hours. It doesn't really make any sense. Can somebody clear up for me when these methods should run? Also, if you have a suggestion for a better place to put them given that I want it to be a shared runspace for all connections and that I want it to run before anybody tries to connect to the API. Perhaps a separate service?
Also, would it be worth looking into ASP.NET CORE? I don't need it to run on anything other than IIS, however if there is a benefit to using CORE I am at a point where it will be easy to switch.
public partial class Startup
{
public Cache GlobalCache;
public static PowershellRunspace PSRunspace;
public static ActiveDirectory ADObjects = new ActiveDirectory();
public void Configuration(IAppBuilder app)
{
ConfigureAuth(app);
GlobalCache = new Cache();
AppLog log = new AppLog();
log.InfoLog("Starting PowerShell Runspace in Hangfire...", true);
GlobalConfiguration.Configuration.UseSqlServerStorage("Hangfire");
BackgroundJob.Enqueue(() => log.InfoLog("Hangfire started!", true));
BackgroundJob.Enqueue(() => ADObjects.Startup(true));
BackgroundJob.Enqueue(() => StaticRunspace.Start());
app.UseHangfireDashboard();
app.UseHangfireServer();
}
}
Assuming you're running this application in IIS (and not self-hosting), the following rules apply:
The Configuration method runs once per application start.
The application is started lazily (on the first request to it via HTTP/S).
IIS has a few settings that affect the application:
An idle timeout. If the app isn't accessed via a request in 20 minutes then the application is unloaded / put offline. The next request starts it again.
A regular app pool recycle. It just straight up restarts the application by recycling the app pool every 1740 minutes.
So the behavior you're seeing is likely due to the infrequent access of the application, combined with the IIS defaults. If you'd like to see or configure the settings, you can do so by going into IIS, right clicking on your app pool, and selecting Advanced Settings.
I have to solve the following problem. We got an ASMX web service which is requested every two minutes. If this service is not requested for ten minutes an email should be sent. We want to realize this by using the scheduled tasks. We imagined it like this
1. Creating a scheduled task which will send an email every ten minutes
2. If the service is requested the execution time for the task will be set to ten minutes from now and so the execution time cannot be reached
- If the service is not requested the execution time will be reached and the email is sent
Is there a way to solve this in ASP.NET or are there maybe better solutions?
Thanks for any response.
You may want to take a look at the Revalee open source project.
You can use it to schedule web callbacks at specific times. In your case, you could schedule a web callback (10 minutes in the future) every time your web service is used. When your web service receives the callback, it can determine whether or not the service has been used recently. If the web service has been active, then the callback is ignored; if the web service has been inactive, then it can send out an email message.
For example using Revalee, you might:
Register a future (10 minutes from now) callback when your application launches.
private DateTimeOffet? lastActive = null;
private void ScheduleTenMinuteCallback()
{
// Schedule your callback 10 minutes from now
DateTimeOffset callbackTime = DateTimeOffset.Now.AddMinutes(10.0);
// Your web service's Uri
Uri callbackUrl = new Uri("http://yourwebservice.com/ScheduledCallback/YourActivityMonitor");
// Register the callback request with the Revalee service
RevaleeRegistrar.ScheduleCallback(callbackTime, callbackUrl);
}
Anytime your web service is used, you register another callback and store the date & time that your service was active as a global value.
lastActive = DateTimeOffset.Now;
ScheduleTenMinuteCallback();
Finally, when the web schedule task activates and calls your application back, then you test the value of the global
private void YourActivityMonitor()
{
if (!lastActive.HasValue || lastActive.Value <= DateTimeOffset.Now.AddMinutes(-10.0))
{
// Send your "10 minutes has elapsed" email message
}
}
I hope this helps.
Disclaimer: I was one of the developers involved with the Revalee project. To be clear, however, Revalee is free, open source software. The source code is available on GitHub.
Background:
We have a Python web application which uses SqlAlchemy as ORM. We run this application with Gunicorn(sync worker) currently. This application is only used to respond LONG RUNNING REQUESTS (i.e. serving big files, please don't advise using X-Sendfile/X-Accel-Redirect because the response is generated dynamically from Python app).
With Gunicorn sync workers, when we run 8 workers only 8 request is served simulatenously. Since all of these responses are IO bound, we want to switch to asyncronous worker type to get better throughput.
We have switched the worker type from sync to eventlet in Gunicorn configuration file. Now we can respond all of the requests simultaneously but another mysterious (mysterious to me) problem has occured.
In the application we have a scoped session object in module level. Following code is from our orm.py file:
uri = 'mysql://%s:%s#%s/%s?charset=utf8&use_unicode=1' % (\
config.MYSQL_USER,
config.MYSQL_PASSWD,
config.MYSQL_HOST,
config.MYSQL_DB,
)
engine = create_engine(uri, echo=False)
session = scoped_session(sessionmaker(
autocommit=False,
autoflush=False,
bind=engine,
query_cls=CustomQuery,
expire_on_commit=False
))
Our application uses the session like this:
from putio.models import session
f = session.query(File).first()
f.name = 'asdf'
session.add(f)
session.commit()
While we using sync worker, session was used from 1 request at a time. After we have switched to async eventlet worker, all requests in the same worker share the same session which is not desired. When the session is commited in one request, or an exception is happened, all other requests fail because the session is shared.
In documents of SQLAlchemy, said that scoped_session is used for seperated sessions in threaded environments. AFAIK requests in async workers run in same thread.
Question:
We want seperate sessions for each request in async worker. What is the correct way of using session with async workers in SQLAlchemy?
Use scoped_session's scopefunc argument.