There's an HTTP request to the IceCast 2.3.2-kh29 server MP3 stream http://*:*/.mp3 and response (some data are **ed):
GET /*.mp3 HTTP/1.1
Host: ***:*
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
HTTP/1.0 200 OK
Content-Type: audio/mpeg
icy-br: 192
ice-audio-info: bitrate=192;samplerate=44100;channels=2
icy-description: MP3 192 Kbps
icy-genre: *
icy-name: *
icy-pub: 1
icy-url: http://*
Server: Icecast 2.3.2-kh29
Cache-Control: no-cache
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
The Content-Length is not specified in the response. Also keep-alive is missing, despite the request. Is it normal situation? The protocol specifies that Content-Length must be in this case. The stream starts playing. Can be somehow that not all headers are shown in LiveHTTPHeaders this way? Or the stream is a special case, when used some sort of artificial made Content-Length?
HTTP 1.0 does not require that a content length is specified. SHOUTcast/Icecast servers will not specify the content length because the streams are live and there is no pre-determined length.
Related
I'm building a little application that loads fresh data from the OpenStreetMap project. For my app, it's very important to request the freshest data possible, so I don't want to be served cached data. Now, since OpenStreetMap has a lot of data requests to handle, they have some kind of caching on their side too. Simply sending a GET request isn't enough, because this will likely serve a pre-cached tile:
x-served-by: cache-ams21054-AMS
x-cache: HIT
x-cache-hits: 1
x-timer: S1652643221.620976,VS0,VE1
This led me to inspect the requests sent by Firefox on the official website. This is the header that is sent on a normal refresh:
Host: tile.openstreetmap.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
Accept: image/avif,image/webp,*/*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.openstreetmap.org/
Connection: keep-alive
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-site
And this is the request sent on a force-refresh (ctrl+f5):
GET /14/8033/6198.png HTTP/2
Host: tile.openstreetmap.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
Accept: image/avif,image/webp,*/*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.openstreetmap.org/
Connection: keep-alive
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
The only two additional header fields are Cache-Control: no-cache and Pragma: no-cache.
This led me to implement the same in my own application. I added the same header fields to my requests. However, this did not fix my problem and I was still getting served cached content (x-cache: HIT) in the response. After some googling and experimenting, I tried adding max-age=0 to Cache-Control and this did stop the servers from sending me cached content.
Here's my question: how come that my app needs the max-age=0 value, while a browser can get non-cached content reliably with just the no-cache values?
What other fields control that behavior, if all the fields I have are these:
User-Agent: MyApp-v1
Cache-Control: no-cache, max-age=0
Pragma: no-cache
For reference, here are the response headers for a standard refresh and a force-refresh:
Refresh:
HTTP/2 200 OK
server: Apache/2.4.41 (Ubuntu)
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0
etag: "afa1fdc623f5ea600b40ebd27c27f97a"
cache-control: max-age=33407, stale-while-revalidate=604800, stale-if-error=604800
expires: Thu, 12 May 2022 16:47:46 GMT
access-control-allow-origin: *
x-tilerender: odin.openstreetmap.org
content-type: image/png
accept-ranges: bytes
date: Sun, 15 May 2022 20:01:23 GMT
via: 1.1 varnish
age: 304223
x-served-by: cache-ams21051-AMS
x-cache: HIT
x-cache-hits: 1
x-timer: S1652644883.024001,VS0,VE1
content-length: 8290
X-Firefox-Spdy: h2
Forced refresh:
HTTP/2 200 OK
server: Apache/2.4.41 (Ubuntu)
strict-transport-security: max-age=31536000; includeSubDomains; preload
expect-ct: max-age=0
etag: "1e2555621fd6e9da0c35d0ab5440d2e5"
cache-control: max-age=8170, stale-while-revalidate=604800, stale-if-error=604800
expires: Sun, 15 May 2022 22:03:15 GMT
access-control-allow-origin: *
x-tilerender: nidhogg.openstreetmap.org
content-type: image/png
accept-ranges: bytes
date: Sun, 15 May 2022 19:47:05 GMT
via: 1.1 varnish
age: 0
x-served-by: cache-ams21083-AMS
x-cache: MISS
x-cache-hits: 0
x-timer: S1652644026.595670,VS0,VE97
content-length: 14499
X-Firefox-Spdy: h2
A server can emit data however it wishes, including from its own caches. The Cache-Control: no-cache mostly pertain to the browser-side caches.
For a browser to do a 'fresh' request, the most important this is mainly just not sending an ETag or If-Modified-Since header.
There's no standard header/instruction to tell a server to get a 'fresher' version of an item and not use its own internal caches. It's possible that for the API you're using they do have a mechanism to request uncached responses, but it's not standard.
I have a web application that serves both HTML and multiple RDF formats (in the example below, it's RDF/XML).
A page loads as HTML (naturally), and then requests its own URL as RDF/XML.
The problem: it looks like Firefox 74.0 (64-bit) (on Windows) is mixing up ETag values from those two requests, ignoring different Content-Types as well as Vary: Accept being present.
When I reload the page, I can see it uses the ETag: "95e11fbc9e816b56" from the second (RDF/XML) response in the request for HTML, and vice versa:
Request URL: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Host: localhost:4443
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Connection: keep-alive
Cookie: _ga=GA1.1.828629977.1584086266; LinkedDataHub.first-time-message=true
Upgrade-Insecure-Requests: 1
If-None-Match: "95e11fbc9e816b56"
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000;includeSubDomains
ETag: "95e11fbc139f56de"
Cache-Control: max-age=3600, public
Last-Modified: Wed, 12 Feb 2020 23:05:15 GMT
Vary: Accept-Charset,Accept,Accept-Encoding
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Sun, 22 Mar 2020 10:13:43 GMT
Request URL: https://localhost:4443/6a6283d2-2a40-4882-b89d-8073a7c30e17/
Host: localhost:4443
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: application/rdf+xml
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://localhost:4443/d376ee88-ff7d-48ee-81c4-1220c9f482f0/
Connection: keep-alive
Cookie: _ga=GA1.1.828629977.1584086266; LinkedDataHub.first-time-message=true
If-None-Match: "95e11fbc139f56de"
Cache-Control: max-age=0
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Strict-Transport-Security: max-age=31536000;includeSubDomains
ETag: "95e11fbc9e816b56"
Last-Modified: Wed, 12 Feb 2020 23:05:15 GMT
Vary: Accept-Charset,Accept
Content-Type: application/rdf+xml;charset=UTF-8
Transfer-Encoding: chunked
Date: Sun, 22 Mar 2020 10:13:55 GMT
On Chrome, I cannot get it to send If-None-Match headers at all, but this is probably due to the self-signed certificate.
Note that the ETag values are similar, but different: "95e11fbc139f56de" vs. "95e11fbc9e816b56".
This doesn't make any sense to me. Any explanations? Thanks.
The relevant specification is Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests.
The problem, essentially, is that you're relying on behavior that isn't mandated by the HTTP standard, and doesn't happen to be implemented by browsers.
For your scheme to work, browsers would have to store multiple representations of a single resource in their cache. Unfortunately, as discussed in articles like these, they don't do that.
Browsers typically do not implement the capability to store multiple variations per URL. The rationale for this is that the things we typically use Vary for (mainly Accept-Encoding and Accept-Language) do not change frequently within the context of a single user.
So the issue isn't the ETags, it's that the browser is just overwriting the single representation in its cache each time it gets a different representation.
If the browser did store multiple representations, the scheme should work fine. In that case, note that it would be the server, not the client, that selects between multiple ETags. The client would send an If-None-Match header with all the ETags it knows about, and it would be up to the server to decide which one, if any, matched the requested representation.
According to the article above, edge servers (as opposed to browsers) do keep multiple representations in the cache for each resource, so it's still possible that your scheme could generate performance gains.
I'm hosting a large (1.8GB) video file under Xampp. I've created the following HTML file:
<!DOCTYPE html>
<html>
<head>
</head>
<body>
<video controls width="800" src="test01.mp4" />
</body>
</html>
And (good news) the video is not downloaded all at once when the page loads - it is downloaded in pieces as needed to play the video.
However, I'm having trouble understanding the HTTP responses that are sent. When I initially load the page, the first two HTTP requests regarding the video are:
GET http://localhost:90/movies/test01.mp4
Host: localhost:90
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Referer: http://localhost:90/movies/
Range: bytes=0-
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Response headers:
HTTP/1.1 206 Partial Content
Date: Mon, 09 Jul 2018 00:16:32 GMT
Server: Apache/2.4.29 (Win32) OpenSSL/1.1.0g PHP/7.2.3
Last-Modified: Fri, 29 Jun 2018 09:00:33 GMT
ETag: "77ab7f58-56fc414a18560"
Accept-Ranges: bytes
Content-Length: 2007727960
Cache-Control: no-cache, must-revalidate
Content-Range: bytes 0-2007727959/2007727960
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: video/mp4
Second request headers:
Host: localhost:90
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0
Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Referer: http://localhost:90/movies/
Range: bytes=1992327168-
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Second response headers:
HTTP/1.1 206 Partial Content
Date: Mon, 09 Jul 2018 00:16:35 GMT
Server: Apache/2.4.29 (Win32) OpenSSL/1.1.0g PHP/7.2.3
Last-Modified: Fri, 29 Jun 2018 09:00:33 GMT
ETag: "77ab7f58-56fc414a18560"
Accept-Ranges: bytes
Content-Length: 15400792
Cache-Control: no-cache, must-revalidate
Content-Range: bytes 1992327168-2007727959/2007727960
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: video/mp4
The first response has Content-Length: 2007727960 which is 1.8 GB. However, the full video is not transferred - Firefox's network tab reports that the transferred data is only 54.75 MB.
The second request then asks for Range: bytes=1992327168- so somehow the browser could tell the response body length of the first response even with an incorrect Content-Length header. Unlike the first response, the second response has Content-Length set correctly.
In the first request, why is Content-Length not set to the length of the response body as it should be? How then does the client know how long the response body is?
The easiest explanation is that the Content-Length is indeed correct, but Firefox aborts the read operation after ~54MB.
I've read all the caching documentation for HTTP, and this are the headers for my file:
Edit: I am now using amazon S3 to see if it helps and still the same problem! Here go the request and the response> What am I missing? Even thought the response asks for the browser to cache the file, it doesn't! I've checked the Google Chrome cache and the file is not there.
Request URL:https://s3-sa-east-1.amazonaws.com/combo-combat-release/WebPlayer.unity3d
Request Method:GET
Status Code:200 OK
**Request Headers**
GET /combo-combat-release/WebPlayer.unity3d HTTP/1.1
Host: s3-sa-east-1.amazonaws.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: es,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
**Response Headers**
HTTP/1.1 200 OK
x-amz-id-2: xu/RKgR4Xt6G3jQ1qkdO5rzd4TlngDYYERZ+bbGlalitJFMDjAgiLAoP4GXhmkeo
x-amz-request-id: 6A9389D16407D28D
Date: Wed, 16 Jan 2013 22:42:26 GMT
Cache-Control: max-age=2592000, public, must-revalidate
Last-Modified: Wed, 16 Jan 2013 22:29:55 GMT
ETag: "0f8d22257da9fcae61f21fd30b7a1fd4"
Accept-Ranges: bytes
Content-Type: application/octet-stream
Content-Length: 39222189
Server: AmazonS3
But the browser doesn't cache the file. Each time, the request is sent and all the data transferred from scratch (200 OK instead of 304). :(
File size is 37.4 MB
Any tip on what could be wrong?
Thanks,
Manuel
Try HTTP 1.1 instead of 1.0.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html states:
Note that HTTP/1.0 caches might not implement Cache-Control and
might only implement Pragma: no-cache
I'm sending an http request to a server that requests an image but using range header with several ranges, the request text is:
GET /images/nav_logo102.png HTTP/1.1
Host: www.google.com.eg
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Cookie: PREF=ID=8aacc11c670a5a37:U=cba349de64cbf880:FF=0:LD=en:TM=1310392358:LM=1327944471:S=zc8_vfTdF5U3C-XN; NID=56=15iHWUMVXBGpdEkEcMCeu32GUL6GlK3aEB5vmocRT4kdILhwGpe9mn6DqDrnZBYWJzA2g4YzfXP8IP3tH7Hw4CzC6FwDrV3uqgv3XhCqmrWI0TBD52Vs3nbdth5YzkOR; SID=DQAAAMUAAAC5feWAhXsCT9NT6ObU32dOLBTU_KtRzxc7Ug6QWIhQW_bpXNvzQtoddtt-a2BbUefv89ZjwNwgrgCiCse3INYkeCyfR7PVPaoDPEmUWQ_0sHXBgbf5U0JBg0XxT8KZMmY9kLOZfEKGg0UGtfNNk8uJKWwPXWlkCYjGalQFyVinUNwiYqayYHGvnA0vtiftBCgqnpHawkyVWoo9hCk3vfDKrRHSdO-GQZoxbL21VlH2dbsJJrnJBaVlR-y8KTg14eLgYpt2swZJxcVZwHThhaF-; HSID=AJLTtk1qkIE6L4aLe; GZ=Z=1
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:5.0) Gecko/20100101 Firefox/5.0
Range: bytes=2048-4095,4096-6143,6144-8191,8192-10239,10240-12287,12288-14335,14336-16383,16384-18431,18432-20479,20480-22527,22528-24575,24576-26623,26624-26722
but the server is responding with
HTTP/1.1 200 OK
and sending the whole image
if i try the same thing but with only one range, it works and i get only the requested range.
any idea how the request should be made?
You should add \r\n\r\n after last header.
REQUEST:
GET /images/djsBox_2.jpg HTTP/1.1\r\n
Range: bytes=0-2,10-20\r\n
Host: www.djsoft.net\r\n
Accept: */*\r\n
\r\n
I checked it with WFetch tool, and the response is HTTP/1.1 206 Partial Content\r\n
And the server actually returned 2 parts of data:
Accept-Ranges: bytes\r\n
Content-Length: 196\r\n
Content-Type: multipart/byteranges; boundary=4b9e42cd78baa9\r\n
\r\n
\r\n
--4b9e42cd78baa9\r\n
Content-type: image/jpeg\r\n
Content-range: bytes 0-2/4543\r\n
\r\n
ัะจั\r\n
--4b9e42cd78baa9\r\n
Content-type: image/jpeg\r\n
Content-range: bytes 10-20/4543\r\n
\r\n
\x000\x001\x001\x001\x000`\x000`\x000\x000ั\r\n
--4b9e42cd78baa9--\r\n