POSTMANCleint--> Apigee OnCloud-->Apigee On Premise---->Backend
Backend is taking 67 sec to respond and i can see the response in Apigee cloud as well however the same response is not sent to client and instead timeout is received .
I have also increased the timeout counts on HTTTargetConnectionProperties but still the issue persists.
Please let us know where to investigate.
There are two levels of timeout in Apigee -- first at the load balancer which has a 60 second timeout, then at the Apigee layer which I believe was 30 seconds but looks like it was increased to 60.
My guess is that the timeout response is coming from the load balancer and that the the timing is just such that Apigee is able to get the response but the load balancer has already dropped the connection.
If this is a paid instance you should be able to get Apigee to adjust the timeouts to make this work (but, man... 67,000ms response times are pretty long...)
Related
I am using a requestor connector in mule 4 for calling an API. That API takes 24 minutes to send back the response to requestor. So when I am running my application in cloudhub, I am getting HTTP:GATEWAY_TIMED_OUT, error_code: 504.
The response timeout I am setting in the requestor is 24 minutes and connection idle timeout is set to default(30000ms)
How can we update the response timeout of cloudhub?
I understand that you are making an HTTP request to a REST API implemented as an application deployed in CloudHub. You probably are using an URL that goes through CloudHub load balancer (example https://myapp.us-e1.cloudhub.io/api/...). The load balancer has a fixed 5 minutes that can not be changed. Note that 24 minutes is a lot of time to keep connections resources open.
Some alternatives could:
Restructure your application to operate in an asynchronous manner. This might require significant effort.
Skip the load balancer tier and connect to the application worker directly using its DNS name (example https://mule-worker-myapp.us-e1.cloudhub.io:8082/api/...). Be aware that going this way you will lose the benefits of using a load balancer.
I have a datapower mpgw service that takes in JSON POST and GET HTTPs requests. Persistent connections are enabled. It sets the backend url using the dp routing-url variable. How do retries work for this? is there some specific retry setting? does it do retries automatically up to a certain point? what if I don't want it to retry?
The backend app is taking about 1.5 minutes to return 500 when it can't connect, but I want it to return more quickly. I have the "backside timeout" set to 30 seconds. I'm wondering if it's because it's retrying a couple times but I can't find info on how retries are working or configured in this case.
I'm open to more answers, but what i found here looks like it says that with persistent connections enabled, DP will retry after the backend timeout duration up until the duration of the persistent connection timeout.
I had the following issue on a system that I supported ~7 years ago. We never got to the bottom of it, and focus shifted onto other issues. I was recently reminded of it, and wondered if anyone would know what was going on. But alas I'll be a little short on details. Sorry.
The Setup
I had a farm of web servers sitting behind a load balancer. The servers were hosting a system that would receive HTTP requests (XML &/or SOAP) from clients, then for each one kick-off a bunch of further HTTP requests to 3rd-party-suppliers, wait for the suppliers' responses, process and combine the results and respond to the client's request.
Think insurance comparison, but as Business-To-Business XML service.
The whole processing would take 5s of seconds, from receiving the initial client request to them sending back a response to that original HTTP request, and the server would be processing 10s or 100s of requests in parallel (i.e. at any given point, a given webserver would have many client Requests that had come in, and been logged, but not yet been responded to.)
We had detailed logging which record the reciept of the requests, including origin IP and which server was processing the request, and record when a response was sent.
All client requests were sent to a single IP address (well, URL), which was the address of the loadbalancer, which would then forward requests to the webservers, which weren't individually accessible to the internet (they didn't have public IP addresses).
Our load balancer would allow us to take individual web-servers out of rotation, for maintenance.
When we did that we could watch the DB logs, and see new requests stop coming in, and the existing request gradually get completed, until there were now outstanding requests and the server was idle.
The problem
We found that sometimes, when we took a server out of rotation ... it wouldn't entirely stop receiving requests. You could see the large bulk of request suddenly stop coming in, but it would still receive a trickle of fresh requests (I don't know ... maybe 0.1% of normal load, maybe less?). I think the longest we left it going was maybe ... 10 minutes?
Notably we realised that all of those requests were coming from a single client/IP address (I don't remember which).
I forget whether other (still-in-rotation) webservers were still receiving requests from this client, but I think they were?
If we rebooted the webserver, no further requests would come in after restarting.
Web stack was Windows, IIS, ASP.NET; pretty old school even at the time. All servers individually owned and configured.
What was happening?
We vaguely waved our hands and asserted that the client's integration with us was "holding an HTTP tunnel open and sending multiple requests through it", rather than sending each request separately, and thus was maintaining that tunnel even after the LB stopped sending new requests to that server. But that was BS-waffle, and since we never needed to actually understand what was going on, we ignored it and moved on with our lives :)
But I'd still like to know what we were seeing, if anyone can diagnose it from that description.
We vaguely waved our hands and asserted that the client's integration with us was "holding an HTTP tunnel open and sending multiple requests through it", rather than sending each request separately, and thus was maintaining that tunnel even after the LB stopped sending new requests to that server.
That sounds like a good explanation.
Normally, a LB will refuse new connections to a removed server, but will allow open connections to live on until they naturally close. This is known as "connection draining" or "graceful shutdown".
If one of your clients had HTTP keepalive on, and was holding a TCP connection open and sending HTTP requests through it for a long time, it would give the symptoms you describe.
Most LBs will have a configuration knob for how long to wait for connections to close before force-closing them during this "connection draining" time. You can set a timeout here to avoid this scenario if it is a problem for you.
The HTTP connection handling behaviour of clients will vary at the client's discretion, to a large extent. Perhaps most of your clients were of one type (say, web browsers) and weren't holding open a single connection for 10 mins, but perhaps one client was different (say, a programmatic HTTP API client)?
Further reading about "connection draining" on AWS Load Balancers here (the exact details will vary by LB vendor): https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-conn-drain.html
Further reading about HTTP keep alive here: https://en.wikipedia.org/wiki/HTTP_persistent_connection
I'm using Bing V7 API and sending an HTTP requests for this endpoint:
https://api.cognitive.microsoft.com/bing/v7.0/search
When I'm define my HttpClient, I need to select the right Timeout value. To short timeout, will makes me loose some answers from the server. Too long timeouts, will make me wait, even if the server is not there.
I looked on Bing documentation and didn't find the right value.
What is the right HTTP request timeout for this calls?
What is the right HTTP request timeout for this calls?
I'm using Bing V7 API and sending an HTTP requests for this endpoint:
Bing provides their API via a HTTP endpoint. This has nothing really to do with the API itself in my opinion as HTTP is just the transport in this situation. HTTP request context is normally handled by eg. reverse proxies such as NGINX (or likely MS IIS here). Hence no documentation in the API docs.
When I'm define my HttpClient, I need to select the right Timeout value. To short timeout, will makes me loose some answers from the server. Too long timeouts, will make me wait, even if the server is not there.
The timeout value in your HttpClient is just ment to eventually recover from a blocking situation. This means that your program won't block indefinitely, but will at some point terminate the HTTP action at hand. This is useful if your HttpClient got into a eg. network split situation, deadlock or similar situation and no reply will ever come.
A timeout value between 45 to 60 seconds is plenty.
Too long timeouts, will make me wait, even if the server is not there.
I would keep the HttpClient timeout value at a fixed eg. 60 seconds and have a second "supervisor" thread doing some more dynamic "Smoke test" to check if connectivity is ok or if there is some other problem at which point you then can terminate HttpClient early.
I am making an http request which ends up taking more than 8 mins. For me, this long running request work fine. I am able to get a response back to my browser without any issues. (I am located on the same network as the server).
However, for some users, the browser never returns any response. (Note: When the same http request executes in 1 min, these users are able to see the response without any problem)
These users happen to be on another network. And there probably is a firewall or two between their location and the server.
I can see on their fiddler that the request is just sitting there waiting for a response.
I am right now assuming that firewall is killing the idle http connection.. but I am not sure.
If you have any idea why the response never gets back, or why the connection never breaks.. it will be really helpful.
Also: Is it possible to fix this issue by writing an Applet that somehow manages to keep the sending dummy signal to the server, even after having sent (flushed) the request to the server?
The users might be behind a connection tracking firewall/NAT gateway. Such gateways tend to drop the TCP connection when nothing has happened for a period of time. In a custom protocol you could send some kind of heartbeat messags to keep the TCP connection alive, but with HTTP you don't have proper control over that connection, nor does HTTP facilitate what's needed to keep a tcp connection "alive".
The usual way to handle long running jobs initated by an HTTP request is to fire off that job in the background, sending a proper response back to the client immediately and have an applet/ajax request poll the status of that job and returning the result when it's done.
If you need a quick fix, see if you can control any timeouts on the gateways between the server and the user.
Have you considered that the users might be using a browser which has a HTTP timeout which causes the browser to stop waiting for a response after a certain amount of time?
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html
http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html
If you are using Linux machine try
# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9
# echo 1500 > /proc/sys/net/ipv4/tcp_keepalive_time
# echo 500 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# echo 20 > /proc/sys/net/ipv4/tcp_keepalive_probes