I am facing a rather tricky issue, where it appears that the varnish is closing the backend connection without waiting for a respones from the backend.
We are using Nginx to serve static content Below is the sequence of messages
Varnish sends POST request to App
App sends back 500 Internal Server Error
Varnish interprets the 500 internal Server Error (to display static error page)
Varnish sends GET request to Nginx server (on the same server) to serve static content
Varnish shows following error message (even though Nginx sends the response successfully within milliseconds)
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 38 boot.staticpages 127.0.0.1 82 127.0.0.1 35064
- BackendStart 127.0.0.1 82
- FetchError backend write error: 0 (Success)
- Timestamp Bereq: 1543420795.016075 5.106813 0.000099
- BackendClose 38 boot.staticpages
- Timestamp Beresp: 1543420795.016497 5.107235 0.000422
- Timestamp Error: 1543420795.016503 5.107241 0.000005
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Wed, 28 Nov 2018 15:59:55 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
Varnish then again goes the same Nginx server to display default content.
Nginx sends response and varnish accepts it and sends it back to the customer
It appears that the backend connection gets closed pretty quickly
Any help in this regard is highly appreciated
Thanks,
We resolve the issue and below is the summary of what the issue was and how we resolved it;
Issue Summary:
Varnish is displaying backend fetch error when original POST request results in 500 Internal Error and backend_response is used to GET staticpage customized 500 Internal Server Error Message
VarnishLog Output (only relevant message):
It can be seen that Backend is being closed as soon as the request is sent.
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 24 boot.staticpages 127.0.0.1 82 127.0.0.1 40696
- BackendStart 127.0.0.1 82
- FetchError backend write error: 0 (Success)
- Timestamp Bereq: 1543416195.877756 5.116981 0.000046
- BackendClose 24 boot.staticpages
- Timestamp Beresp: 1543416195.877888 5.117113 0.000132
- Timestamp Error: 1543416195.877892 5.117117 0.000004
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Wed, 28 Nov 2018 14:43:15 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
Root Cause:
Varnish can't retry because there's no body to send anymore.
Resolution:
Cache the body of the original request by using std.cache_req_body(10KB); https://varnish-cache.org/docs/trunk/reference/vmod_generated.html#func-cache-req-body
Related
I'm running an nsq cluster in Docker containers using the following docker-compose.yaml file:
version: '2'
services:
nsqlookupd:
image: nsqio/nsq
command: /nsqlookupd
ports:
- "4160"
- "4161:4161"
nsqd:
image: nsqio/nsq
command: /nsqd --lookupd-tcp-address=nsqlookupd:4160 --data-path=/data
volumes:
- data:/data
ports:
- "4150:4150"
- "4151:4151"
nsqadmin:
image: nsqio/nsq
command: /nsqadmin --lookupd-http-address=nsqlookupd:4161
ports:
- "4171:4171"
volumes:
data:
Everything runs fine. But, if I call the /nodes endpoint on the nsqdlookup server I get this:
$ http http://localhost:4161/nodes
HTTP/1.1 200 OK
Content-Length: 238
Content-Type: application/json; charset=utf-8
Date: Tue, 24 Jan 2017 08:44:27 GMT
{
"data": {
"producers": [
{
"broadcast_address": "7dd3d550e7f8",
"hostname": "7dd3d550e7f8",
"http_port": 4151,
"remote_address": "172.18.0.4:57156",
"tcp_port": 4150,
"tombstones": [],
"topics": [],
"version": "0.3.8"
}
]
},
"status_code": 200,
"status_txt": "OK"
}
The broadcast address looks like the container's name/hostname. I tried to ping on port 4151 it just in case, but it fails.
> http http://7dd3d550e7f8:4151/ping
http: error: ConnectionError: HTTPConnectionPool(host='7dd3d550e7f8', port=4151): Max retries exceeded with url: /ping (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x000001C397173EF0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',)) while doing GET request to URL: http://7dd3d550e7f8:4151/ping
Same for the remote address:
> http http://172.18.0.4:4151/ping
http: error: ConnectionError: HTTPConnectionPool(host='172.18.0.4', port=4151): Max retries exceeded with url: /ping (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x000001C0D9545F28>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',)) while doing GET request to URL: http://172.18.0.4:4151/ping
Everything works if I use localhost or 127.0.0.1:
> http http://localhost:4151/ping
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Tue, 24 Jan 2017 08:51:30 GMT
OK
But, that's cheating. The whole point of the nsqlookupd servers is that they keep track on the nsqd servers so clients can dynamically get a list of responsive servers.
Is it possible to an accessible URL/IP address for nsqd nodes from nslookupd server when the nsqd nodes are running in Docker containers?
Is there some magic incantation to make it work?
Did someone try maybe using Swarm or Kubernetes?
I found that GKE now supports StatefulSet at 1.5.2
It means your nsqd, nsqlookupd can be spin to as SS instances. Now you can use -broadcast-address=$POD_IP from downward api and your producers will be able to publish to nsq-0.nsq-service-name, nsq-1.nsq-service-name etc., while consumers will get advertised nsqd IP address from nsqlookupd. That works for us. Just managed to make it to work today
I am developing an HTTP proxy in Java. I resend all the data from client to server without touching it, but for some URLs (for example this) server returns the 404 error if I am connecting through my proxy.
The requested URL uses Varnish caching, so it might be the root of problem. I cannot reconfigure it - it is not my.
If I request that URL directly with browser, the server returns 200 and the image is shown correctly.
I am stuck because I even do not know what to read and how to compose a search request.
Thanks a lot.
Fix the Host: header of the re-issued request. The request going out from the proxy either has no Host header or it is broken (or only X-Host exists). Also take note that the proxy application will execute its own DNS lookup and that might yield a different IP address than your local computer (where you issued the original request).
This works:
> curl -s -D - -o /dev/null http://212.25.95.152/w/w-200/1902047-41.jpg -H "Host: msc.wcdn.co.il"
HTTP/1.1 200 OK
Content-Type: image/jpeg
Cache-Control: max-age = 315360000
magicmarker: 1
Content-Length: 27922
Accept-Ranges: bytes
Date: Sun, 05 Jul 2015 00:52:08 GMT
X-Varnish: 2508753650 2474246958
Age: 67952
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
Can't figure this one out!
OS: CentOS 6.6 (Up-To-Date)
I get the following 503 error when using my nagios check_http check (or curl) to query an SSL site served via HAProxy 1.5.
[root#nagios ~]# /usr/local/nagios/libexec/check_http -v -H example.com -S1
GET / HTTP/1.1
User-Agent: check_http/v2.0 (nagios-plugins 2.0)
Connection: close
Host: example.com
https://example.com:443/ is 212 characters
STATUS: HTTP/1.0 503 Service Unavailable
**** HEADER ****
Cache-Control: no-cache
Connection: close
Content-Type: text/html
**** CONTENT ****
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
HTTP CRITICAL: HTTP/1.0 503 Service Unavailable - 212 bytes in 1.076 second response time |time=1.075766s;;;0.000000 size=212B;;;0
[root#nagios ~]# curl -I https://example.com
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
However. I can access the site fine via any browser fine (200 OK), and also curl -I https://example.com from another server:
root#localhost:~# curl -I https://example.com
HTTP/1.1 200 OK
Date: Wed, 18 Feb 2015 14:36:51 GMT
Server: Apache/2.4.6
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
Last-Modified: Wed, 18 Feb 2015 14:36:52 GMT
Content-Type: text/html; charset=UTF-8
Strict-Transport-Security: max-age=31536000;
The HAProxy server is runnning on pfSense 2.2.
I see that HAProxy returns an HTTP/1.0 for nagios and HTTP/1.1 from elsewhere. So is it my check_http' plugin causing this or is itcurl`?
Is my server just not sending the HOST header? If so, how can I resolve this?
What check_http does is it checks whether there exists a index.html-file on the server. This means you might have http running and working, while the check still fails.
Regardless whether or not creating an index.html file on the server resolves the issue, u might not want to create the circumstances such that the check works.
I suppose setting up a check for pinging your example.com and a check via nrpe to see whether your http-service is running will meet your requirements.
check_http has an option called --sni
You need to use that option
I'm making a clojure web app that streams data to clients using chunked HTTP responses. This works great when I run it locally using foreman, but doesn't work properly when I deploy it to Heroku.
A minimal example exhibiting this behaviour can be found on my github here. The frontend (in resources/index.html) performs an AJAX GET request and prints the response chunks as they arrive. The server uses http-kit to send a new chunk to connected clients every second. By design, the HTTP request never completes.
When the same code is deployed to Heroku, the HTTP connection is closed by the server immediately after the first chunk is sent. It seems to be Heroku's routing mesh which is causing this disconnection to occur.
This can also be seen by performing the GET request using curl:
$ curl -v http://arcane-headland-2284.herokuapp.com/stream
* About to connect() to arcane-headland-2284.herokuapp.com port 80 (#0)
* Trying 54.243.166.168...
* Adding handle: conn: 0x6c3be0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x6c3be0) send_pipe: 1, recv_pipe: 0
* Connected to arcane-headland-2284.herokuapp.com (54.243.166.168) port 80 (#0)
> GET /stream HTTP/1.1
> User-Agent: curl/7.31.0
> Host: arcane-headland-2284.herokuapp.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Date: Sat, 17 Aug 2013 16:57:24 GMT
* Server http-kit is not blacklisted
< Server: http-kit
< transfer-encoding: chunked
< Connection: keep-alive
<
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
The time is currently Sat Aug 17 16:57:24 UTC 2013 <-- this is the first chunk
Can anybody suggest why this is happening? HTTP streaming is supposed to be supported in Heroku's Cedar stack. The fact the code runs correctly using foreman suggests it is something in Heroku's routing mesh causing it to break.
Live demo of the failing project: http://arcane-headland-2284.herokuapp.com/
This was due to a bug in http-kit which will be fixed shortly.
https://devcenter.heroku.com/articles/request-timeout may be relevant: "long-polling" requests like yours have to send data every 55 seconds or be terminated.
We have an Windows 2008 server with IIS 7 to test sites we develop for our clients. Each site has a binding on a subdomain:
clienta.example.com
clientb.example.com
clientc.example.com
(* Using example.com to protect the innocent)
For one of these sites we now have to test if it works over https.
So I have created a certificate request with *.example.com as the common name. I have received the certificate (issued by PositiveSSL SA) and completed the request. The certificate is now installed in IIS.
Now I have added an https binding to the second site with the following settings:
type: https
IP address: All Unassigned
Port: 443
Host name: clientb.example.com
SSL certificate: *.example.com
Browsing the site over regular http works fine. When I try to browse the site over https I get the following errors (depending on the browser used):
Chrome
This webpage is not available
Error 102 (net::ERR_CONNECTION_REFUSED): Unknown error.
Firefox
Unable to connect
Firefox can't establish a connection to the server at clientb.example.com
Firebug says Status: Aborted
Internet Explorer
Internet Explorer cannot display the webpage
I have checked Failed Request Tracing, and according to the log the request was completed with status 200.
I have run the SSL Diagnostics Tool with the following result:
System time: Fri, 04 Mar 2011 14:04:35 GMT
Connecting to 192.168.2.95:443
Connected
Handshake: 115 bytes sent
Handshake: 3877 bytes received
Handshake: 326 bytes sent
Handshake: 59 bytes received
Handshake succeeded
Verifying server certificate, it might take a while...
Server certificate name: *.example.com
Server certificate subject: OU=Domain Control Validated, OU=PositiveSSL Wildcard, CN=*.example.com
Server certificate issuer: C=GB, S=Greater Manchester, L=Salford, O=Comodo CA Limited, CN=PositiveSSL CA
Server certificate validity: From 2-3-2011 1:00:00 To 2-3-2012 0:59:59
1:00:00 To 2-3-2012 0:59:59
HTTPS request:
GET / HTTP/1.0
User-Agent: SSLDiag
Accept:*/*
HTTPS: 85 bytes of encrypted data sent
HTTPS: 533 bytes of encrypted data received
Status:
HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Fri, 04 Mar 2011 14:04:35 GMT
Connection: close
Content-Length: 315
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>
HTTPS: server disconnected
Final handshake: 37 bytes sent successfully
Q: What can I do to make this work?
For the IP-binding to work when combined with a host header it's a good idea to first make sure the firewall is forwarding https requests to the webserver (TCP/443).
Doh.