We have a payment gateway integration that posts data to a third party URL. The user then completes their payment process and when the transaction is complete the gateway posts back to a URL on our server.
That post is failing and the gateway are reporting the following error:
ERROR 13326: Couldn't speak to ServerResultURL [https://foo.com/bar].
Full Error Details: The underlying connection was closed: An unexpected error occurred on a send.
Response object is null
When I post direct to https://foo.com/bar I get a 200 response as I'd expect so I'm not sure where this is falling down.
This is on an Ubuntu box running nginx.
What could be causing that issue and how can I find more detail about it and a way to resolve it?
EDIT:
For brevity the example above is on a URL of /bar but the reality is that I have a rewrite in place (see below). The URL that actually gets posted to is /themes/third_party/cartthrob/lib/extload.php/cardsave_server/result so I'm not sure if the rewrite below is what's causing an issue.
I would still assume not as I do get a 200 response when posting via POSTMAN.
# http://expressionengine.stackexchange.com/questions/19296/404-when-sagepay-attempts-to-contact-cartthrob-notification-url-in-nginx
location /themes/third_party/cartthrob/lib/extload.php {
rewrite ^(.*) /themes/third_party/cartthrob/lib/extload.php?$1 last;
}
Typical causes of this kind of error
I bet your server is responding to the POST to /bar with something that the gateway (PaymentSense, right?) doesn't expect. This might be because:
The gateway can't reach your Ubuntu box over the network, because a firewall or network hardware between the two is blocking it.
Your https cert is bad / expired / self-signed, and the gateway is refusing the connection.
A misconfiguration of NGINX or your web application software (PHP, I imagine? or whatever nginx is serving up) is causing /bar to respond with some odd response, like a 30x, or a 50x error page, or possibly with just the wrong response, such as an HTML page.
Something else is wrong with the response to the POST.
The script/controller running at /bar could be getting unexpected input in the POST request, so you might want to look at the request coming in.
You have a network connectivity issue.
I'll leave the first two items for you to troubleshoot, because I don't think that's what you're asking in this question.
Troubleshooting NGINX Responses
I recommend configuring it to dump its response into an nginx variable using body_filter_by_lua so that you can see what response is coming out. A good example of how to set this up is available here. I think that will lead you understand why /bar is not behaving.
Troubleshooting NGINX Requests
If that isn't revealing the cause of this, try logging the request data. You can do that with something like:
location = /bar {
log_format postdata $request_body;
access_log /var/log/nginx/postdata.log postdata;
fastcgi_pass php_cgi;
}
Review the request headers and body of this POST, and if the error isn't immediately apparent, try to replay the exact same request (using an HTTP client that gives you complete control, such as curl) and debug what is happening with /bar. Is nginx running the script/controller that you think it should be running when you make an identical POST to /bar? Add logging to the /bar script/controller process.
Use interactive debugging if necessary. (This might require remote Xdebug if you're working with PHP, but no matter what you're using on your server, most web application tools offer some form of interactive debugging.)
Network Troubleshooting
If none of this works, it's possible that the gateway simply can't reach the host and port you're running this on, or that you have some other kind of network connectivity issue. I would run tcpdump on your Ubuntu box to capture the network traffic. If you can recreate this on a quiet (network) system, that will be to your advantage. Still, it's TLS (https), so don't expect to see much other than that the connection opens and packets are arriving. If you find that you need to see inside the TLS traffic in order to troubleshoot, you might consider using mitmproxy to do so.
Related
When I try to send web requests (any kind) from Postman, it goes through the network and I can see the response. If I want to do the same from Python (I use spyder IDE), I get a http connection error.
Basically, the requests are timed out.
When I do a tracert to any host (i.e. google.com), after a number of hops the requests are getting timed out.
I'm on company network. We use dynamic proxy file to direct requests.
My question is twofold:
What is the root cause of the issue?
How can I fix it on my end? (Not involving company IT.)
Many thanks
I could solve this issue with the help of company IT. Problem was - if anyone interested - that I wrongly defined the proxy in the request itself, so that it never reached the proxy. Once I changed the proxy settings, the request could go through.
I'm implementing an HTTP proxy. Sometimes when a browser makes a request via my proxy, I get an error such as ECONNRESET, Address not found, and the like. These indicate errors below the HTTP level. I'm not talking about bugs in my program -- but how other servers behave when I send them an HTTP request.
Some servers might simply not exist, others close the socket, and still others not answer at all.
What is the best way to report these errors to the caller? Is there a standard method that, if I use it, browsers will convert my HTTP message to an appropriate error message? (i.e. they get a reply from the proxy that tells them ECONNRESET, and they act as though they received the ECONNRESET themselves).
If not, how should it be handled?
Motivations
I really want my proxy to be totally transparent and for the browser or other client to work exactly as if it wasn't connected to it, so I want to replicate the organic behavior of errors such as ECONNRESET instead of sending an HTTP message with an error code, which would be totally different behavior.
I kind of thought that was the intention when writing an HTTP proxy.
There are several things to keep in mind.
Firstly, if the client is configured to use the proxy (which actually I'd recommend) then fundamentally it will behave differently than if it were directly connecting out over the Internet. This is mostly invisible to the user, but affects things like:
FTP URLs
some caching differences
authentication to the proxy if required
reporting of connection errors etc <= your question.
In the case of reporting errors, a browser will show a connectivity error if it can't connect to the proxy, or open a tunnel via the proxy, but for upstream errors, the proxy will be providing a page (depending on the error, e.g. if a response has already been sent the proxy can't do much but close the connection). This page won't look anything like your browser page would.
If the browser is NOT configured to use a proxy, then you would need to divert or intercept the connection to the proxy. This can cause problems if you decide you want to authenticate your users against the proxy (to identify them / implement user-specific rules etc).
Secondly HTTPS can be a real pain in the neck. This problem is growing as more and more sites move to HTTPS only. There are several issues:
browsers configured to use a proxy, for HTTPS URLS will firstly open a tunnel via the proxy using the CONNECT method. If your proxy wants to prevent this then any information it provides in the block response is ignored by the browser, and instead you get the generic browser connectivity error page.
if you want to provide any other benefits one normally wishes from a proxy (e.g. caching / scanning etc) you need to implement a MitM (Man-in-the-middle) and spoof server SSL certificates etc. In fact you need to do this if you just want to send back a block-page to deny things.
There is a way a browser can act a bit more like it was directly connected via a proxy, and that's using SOCKS. SOCKS has a way to return an error code if there's an upstream connection error. It's not the actual socket error code however.
These are all reasons why we wrote the WinGate Internet Client, which is a LSP-based product for our product WinGate. Client applications then learn the actual upstream error codes etc.
It's not a favoured approach nowadays though, as it requires installation of software on the client computer.
I wouldn't provide them too much info. Report what you need through internal logs in case you have to solve the problem. Return a 400, 403 or 418. Why? Perhaps the're just hacking.
I have a working console app, which sends data to an API. However as soon as I launch fiddler, I get the message:
[Fiddler] ReadResponse() failed: The server did not return a complete response for this request. Server returned 257 bytes.
The first header shown in Fiddler is: HTTP/1.1 504 Fiddler - Receive Failure
which seems to be generated directly by fiddler rather than having come from my API server (.NET).
How can I debug why this is happening, given that fiddler will not show me the raw results from the server? I presume there is an HTTP header error of some sort, which is compatible with my console app but not compatible with Fiddler.
I have been playing with gzip compressed requests, so perhaps one of the headers is incorrect (Content-Length), but with no way to view the raw response, it's very hard to debug this problem.
In the end I got some help from #ErikLaw on this:
Download DebugView https://learn.microsoft.com/en-us/sysinternals/downloads/debugview
In Fiddler's black QuickExec box under the session list, type !spew and hit Enter. Fiddler will begin spewing verbose logging information to DebugView, including all reads and writes to/from the network.
Far more information about the failed request is then shown in DebugView, which led me to the root cause that my web server was closing the connection early, before sending all content.
All credit to Eric.
How can I debug why this is happening, given that fiddler will not show me the raw results from the server?
Use Wireshark to see the actual network traffic. Fiddler's good (it's great), but it's not Wireshark. You'll need to jump through some hoops if your traffic is HTTPS, though.
Wireshark is not as easy to use as Fiddler, but it is significantly more powerful.
Also, if you're on Windows, you need to use your machine's local network IP address (e.g. 192.168.x.y), rather than localhost. See this question.
I am using nginx as reverse proxy for file storage upload with an external provider.
When I am processing a file upload, I need to keep track (in my database) whether an upload was successful before returning the response to the user. I would therefore like to use the ngx.location.capture method provided in the lua-nginx-module to talk to my backend about the outcome of the request. Since I need to wait for the response of the upstream server I can only issue the capture in header_filter_by_lua. Unluckily I cannot issue any outwards communication in header_filter_by_lua. ngx.location.capture, ngx.socket.* and ngx.exec are only available when the response has not yet arrived.
How can I react to an upstream response in nginx?
Other approaches I've thought about:
Have a script watch the access log and then issue a curl request. (Seems like there should be an easier way)
Initially send the file via ngx.location.capture in content_by_lua (I don't think this would handle up to 5 GB filesize)
Help is appreciated :)
use for /upload location:
content_by_lua_file with resty.upload module
I have a server block listening to 80 port requests on a specific server name along with some location directives. How can I make nginx treat any request that doesnt match as if it hadnt received it, that is let it time out? Currently those requests are treated with an 404 error
There is a way to ignore every request and tell nginx to respond nothing:
server {
listen 80 default_server;
return 444;
}
Documented here: http://nginx.org/en/docs/http/ngx_http_rewrite_module.html
The non-standard code 444 closes a connection without sending a response header.
How can I make nginx treat any request that doesnt match as if it hadnt received it, that is let it time out?
You can't unwind the clock. To find out what server name is being requested (from TLS-SNI and/or Host header), nginx must first accept the connection (which will cause the operating system's TCP/IP stack to send a SYN-ACK packet back to the requestor, in order to advance the TCP handshake). Once the connection has been accepted, the other side knows something is listening. So you've already foregone the opportunity for a connection timeout.
Having irrevocably lost that opportunity, the next opportunity you have is to hit the socket receive timeout. Since the client will be waiting for the server to respond with something, you could theoretically trigger its receive timeout by not responding at all. There is no mechanism in standard nginx currently to tell it to sit there and do nothing for a while, but if you are willing to involve another piece of software as well, then you could use the proxy_pass directive or similar to offload that responsibility to another server. Beware, however, that this will keep some amount of operating system resources in use by your server for the life of the connection. Nginx operates on sockets, not raw packets, and the operating system manages those sockets.
Having answered the question as asked, I think it's better to question the premise of the question. Why do you want the client to time out? If your goal is to mess with or "pay back" a malicious attacker, nginx is the wrong tool. This is the space of computer security and you're going to need to use tools that operate at a lower layer on the networking stack. Nginx is designed to be a webserver, not a honeypot.
If your goal is simply to hide the fact that there's a server listening at all, then your goal is impossible to achieve with the way HTTP works currently. Firewalls are only able to achieve ghosting like that by rejecting the connection before it's accepted. Thus, the TCP handshake never proceeds, no SYN-ACK packet gets sent back to the client, and as far as anyone can tell that entire port is a black hole. But it's not possible to do this after determining what server name has been requested. The IP address and port are available from the very first packet; the server name is not.
Someday in the not-so-distant future, HTTP may use UDP instead of TCP, and the request parameters (such as the server name) may be presented up front on the first packet. Unless and until that becomes the norm, however, the situation I describe above will remain.
I'm assuming you are trying to deflect malicious requests. Something like this might work (untested).
Set up a default server (catches any requests that don't match an existing server name), and then redirect the client back to itself:
server {
listen 80 default_server;
rewrite ^ http://127.0.0.1/;
}
You'd have to setup a similar catch-all for invalid locations inside your valid server blocks. Might be more of a headache than you want.
I don't know how useful this would really be in practice. Might be better to consider fail2ban or some other tool that can monitor your logs and ban clients at the firewall.