In making a request to my website with curl and HTTP 1.1, I see my keep-alive connection header explicitly:
$ curl https://website.com/ -i
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 2
Connection: keep-alive
But with Chrome's developer tool and HTTP 2 the header is missing:
Content-Type: text/html; charset=utf-8
Content-Length: 2
In HTTP 2 is it normal that the connection header is not sent (and why)?
Yes, this is normal and specified by the HTTP/2 RFC.
Related
I have Nginx 1.16.1 as a reverse proxy for JFrog Artifactory and they are reachable from the external networks via web application firewall. I am trying to get docker client working with this setup. It sends a HEAD request and awaits a Content-Length to check for the existence of a layer. Now I see that Content-Length is not included in the response received by the client. I can examine it by sending the same request using curl that sends docker:
$ curl -H 'User-Agent: docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.128-microsoft-standard os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \(linux\))' \
-H "Authorization: Bearer ${TOKEN}" \
-H 'Connection: close' \
-I \
"https://${ARTIFACTORY_URL}/v2/${IMAGE}/blobs/${DIGEST}"
HTTP/1.1 200 OK
Date: Mon, 09 Nov 2020 14:57:05 GMT
Server: Secure Entry Server
Content-Type: application/octet-stream
Docker-Content-Digest: sha256:[MASKED]
Docker-Distribution-Api-Version: registry/2.0
X-Artifactory-Id: [MASKED]
X-Artifactory-Node-Id: [MASKED]
Set-Cookie: SCDID_S=[MASKED]; path=/; Secure; HttpOnly
Connection: close
However, I see in the access log of Artifactory that it sets this response header. I used tcpdump to see what data is exchanged between Nginx and Artifactory:
HEAD /v2/[MASKED]/blobs/[MASKED] HTTP/1.1
X-JFrog-Override-Base-Url: https://[MASKED]:443
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Host: [MASKED]
X-Forwarded-For: 10.10.40.14
Connection: close
ClientCorrelator: 0rIKeSpqZ9E$
RequestCorrelator: 7f0100-9099-2020.11.09_1457.05.275-001
HSP_CLIENT_ADDR: [MASKED]
Hsp-ListenerUri: https://[MASKED]
HSP_HTTPS_HOST: [MASKED]:443
Accept: */*
Authorization: Bearer [MASKED]
User-Agent: docker/19.03.13 go/go1.13.15 git-commit/4484c46d9d kernel/4.19.128-microsoft-standard os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.13 \(linux\))
HTTPS: on
SSLSessionID: 78ad360e9ea54f5efdb72ea223a63b6cbc7788ae9a1e876620e398040d06182c
SSLSessionTimeLeft: 3600
SSLSessionAge: 0
SSLCipher: ECDHE-RSA-AES128-GCM-SHA256
SSLCipherKeySize: 128
SSLProtocolVersion: TLSv1.2
Via: HTTP/1.1 Secure Entry Server
HTTP/1.1 200 OK
Content-Length: 2529
Content-Type: application/octet-stream
Date: Mon, 09 Nov 2020 14:57:05 GMT
Docker-Content-Digest: [MASKED]
Docker-Distribution-Api-Version: registry/2.0
Server: Artifactory/7.4.1 70401900
X-Artifactory-Id: 5a2dee84b6d80d2f:1f521881:17554c79de4:-8000
X-Artifactory-Node-Id: [MASKED]
Connection: close
The TrafficAnalyzer on the WAF shows that Content-Length in the incoming response from Artifactory is missing. Hence it must be Nginx responsible for removing it.
Now when I connect via VPN to get around the WAF the response looks okay:
Host: [MASKED]
User-Agent: docker/19.03.13 go/go1.13.15 ...
Authorization: Bearer [MASKED]
Connection: close
Date: Fri, 06 Nov 2020 17:13:58 GMT
Content-Type: application/octet-stream
Content-Length: 2529
Docker-Content-Digest: [MASKED]
Docker-Distribution-Api-Version:registry/2.0
Server: Artifactory/7.4.1 70401900
X-Artifactory-Id: 5a2dee84b6d80d2f:1f521881:17554c79de4:-8000
X-Artifactory-Node-Id: [MASKED]
Connection: close
But I also notice, that there are fewer headers set in the request. Is that some additional WAF-header that causes Nginx to remove Content-Length? I don't see anything related to this in Nginx debug log. Any thoughts?
I want to get chunked response from server, but i can't understand - what wrong i did in this terminal log:
telnet www.google.com 80
Trying 172.217.20.36...
Connected to www.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Transfer-Encoding: chunked
HTTP/1.1 302 Found
Location: http://www.google.com/sorry/index?continue=http://www.google.com/&q=EgRVjAHRGPGutfEFIhkA8aeDS1HQdWrbrx7jkGSfPgX8M5Ou6VMLMgFy
Date: Sun, 26 Jan 2020 09:10:10 GMT
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Content-Type: text/html; charset=UTF-8
Server: HTTP server (unknown)
Content-Length: 325
X-XSS-Protection: 0
Connection: close
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
here.
</BODY></HTML>
I choose as option:
Transfer-Encoding: chunked
But i still get Content-Length: 325, and all content is not chunked encoded.
How to make GET request with chunked encoding using telnet?
1) You can't force a server to use chunked encoding in the response.
2) Setting "Transfer-Encoding" on the request means that you are sending using chunked encoding.
When I fill out the form on a site and examine it using the Chrome Dev Tools I get the following information:
Response Headers:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html;charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Status: 200 OK
Request Headers:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 53
Content-Type: application/x-www-form-urlencoded
DNT: 1
Host: www.32x8.com
Origin: http://www.32x8.com
Form Data:
in0: 0
calctype: pos
in1: 1
in2: 1
in3: 0
drawtype: htmlcss
This works perfectly fine in the browser but when I issue the following curl request:
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "in0=1&calctype=pos&in1=1&in2=1&in3=0&drawtype=htmlcss" -v http://www.32x8.com/circuits2
I get this output:
...
* Connected to www.32x8.com (2605:de00:1:1:4a:3c:0:42) port 80 (#0)
> POST /circuits2 HTTP/1.1
> Host: www.32x8.com
< Server: nginx
< Status: 301 Moved Permanently
< Location: http://www.32x8.com/var2.html
...
I get a 301 response. So I tried adding the -L flag to the command to follow any redirects
curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "in0=1&calctype=pos&in1=1&in2=1&in3=0&drawtype=htmlcss" -L -v http://www.32x8.com/circuits2
But that just gets me this output
...
> POST /circuits2 HTTP/1.1
> Host: www.32x8.com
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 53
>
* upload completely sent off: 53 out of 53 bytes
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Wed, 01 May 2019 19:54:49 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Status: 301 Moved Permanently
...
* Switch from POST to GET
...
> POST /var2.html HTTP/1.1
> Host: www.32x8.com
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 405 Not Allowed
< Server: nginx
< Date: Wed, 01 May 2019 19:54:49 GMT
< Content-Type: text/html
< Content-Length: 173
< Connection: keep-alive
...
I get a 405.
What am I doing wrong?
You're not really doing anything wrong, as such, but you're experiencing a quirk of HTTP that dates from years ago and is maintained by cURL for compatibiity reasons.
You're making a POST request to your server, to which the server is responding with a 301 redirect message. When you ask cURL to follow the redirection it does so, but changes the POST to a GET. This is behaviour inherited from generations of browsers past (I don't understand the logic here). Your server then rejects the GET request as Not Allowed.
You have two possible solutions: a 301 message indicates a permanent redirect, so just use the new address (you should probably do this anyway)
Or, in your own code detect the 301 message and follow the redirection yourself so that you can issue the proper POST command. This might mean you need a shell script rather than just cURL.
We are using CDN for bundling in Azure Web APP. We have implemented approach given in below article.But it seems that for every bundle request to CDN, CDN is only forwarding this request to Azure Web-App,instead of delivering it from its own cache making it more time consuming
https://azure.microsoft.com/en-us/documentation/articles/cdn-websites-with-cdn/#integrate-aspnet-bundling-and-minification-with-azure-cdn
On debugging, found that Cache-Control value is set to no-Cache from azure-web-app. In Bundle Request URL we are using build number for Querystring value.
URL looks like - ~/bundles/lib?v=26948
Caveat is ,this works well when deployed on local.Somehow azure web app is sending wrong value for Cache-Control
Below is fiddler trace for bundle request on webapp.
GET /bundles/lib?v=26948 HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Host: mseventsdevtertiary.azurewebsites.net
Connection: Keep-Alive
Accept-Language: en-IN,en;q=0.5
Accept-Encoding: gzip, deflate, peerdist
X-P2P-PeerDist: Version=1.1
X-P2P-PeerDistEx: MinContentInformation=1.0, MaxContentInformation=2.0
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Transfer-Encoding: chunked
Content-Type: text/javascript; charset=utf-8
Content-Encoding: gzip
Expires: -1
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
Access-Control-Allow-Origin: *
Date: Mon, 27 Jun 2016 09:02:28 GMT
Can I specify the HTTP timeout or does the server impose a value?
For example, if I do:
telnet my.server.net 80
Trying X.X.X.X...
Connected to my.server.net.
Escape character is '^]'.
GET /homepage.html HTTP/1.0
Connection: keep-alive
Host: my.server.net
HTTP/1.1 200 OK
Date: Thu, 03 Oct 2013 09:05:28 GMT
Server: Apache
Last-Modified: Wed, 15 Sep 2010 14:45:31 GMT
ETag: "1af210b-7b-4904d6196d8c0"
Accept-Ranges: bytes
Content-Length: 123
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html
[...]
The line:
Keep-Alive: timeout=15, max=100
...specifies that there is a maximum timeout of 100 seconds, right? How can I set such value?
The client cannot specify the timeout, it is the server configuration that determines the maximum timeout value. The extra Keep-Alive header can inform the client how long the server is willing to keep the connection open (timeout=N value) and how many requests you can do over the same connection (max=M) before the server will force a close of the connection.
See also Proper use of KeepAlive in Apache Htaccess
Yes, you can specify timeout but server has no obligation to use that value. If server is configured with a different timeout, it will return its own Keep-Alive header.
The Keep-Alive header is a hop-by-hop header that provides information
about a persistent connection. Both client and server are able to
provide information independently. (Hypertext Transfer Protocol (HTTP) Keep-Alive Header)