416 Requested Range Not Satisfiable when range is wider than content - http

I get an Error 416 Range Not Satisfiable response when I make an http request with a byte-range of 0-65536. The byte length of the file requested is only 3356.
Reading the spec on byte-range, it sounds like requesting a range that extends beyond the length of the file is okay, and in this case I would expect the entire file to be fetched. https://httpwg.org/specs/rfc9110.html#byte.ranges
A client can limit the number of bytes requested without knowing the size of the selected representation. If the last-pos value is absent, or if the value is greater than or equal to the current length of the representation data, the byte range is interpreted as the remainder of the representation (i.e., the server replaces the value of last-pos with a value that is one less than the current length of the selected representation).
Am I misreading the spec or is this request truly unsatisfiable? I prefer if my client does not need to know the size of the file before making the request; they could vary widely in size. The request is successful if I limit the byte-range to <= the length of the file.
The file requested is a cloud-optimized-geotiff. It is being served by a vite dev server. And the request is made using OpenLayers and geotiff.js.
Here are the DevTools details of the request and response:
General
---------
Request URL: http://localhost:3000/scenarios/1/1_parcel_fill.tif
Request Method: GET
Status Code: 416 Range Not Satisfiable
Remote Address: [::1]:3000
Referrer Policy: strict-origin-when-cross-origin
Response Headers
--------------------
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 0
Content-Range: bytes */3356
Date: Wed, 21 Sep 2022 21:38:19 GMT
Keep-Alive: timeout=5
Request Headers
-------------------
Accept: */*
Accept-Encoding: identity
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: localhost:3000
Pragma: no-cache
Range: bytes=0-65536
Referer: http://localhost:3000/
sec-ch-ua: "Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36

Related

Cors fails even though set up to be permissive

I'm running GenHTTP server that is setup to to have CORS permissive (so allow all):
var inline = Inline.Create();
inline.Add(CorsPolicy.Permissive());
For some reason the request still fails by CORS. For context http://localhost:55409 is the server and http://localhost:55309 is the client.
Options (preflight):
Request URL: http://localhost:55409/api/v1//metadata
Request Method: OPTIONS
Status Code: 204 No Content
Remote Address: [::1]:55409
Referrer Policy: strict-origin-when-cross-origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: *
Access-Control-Allow-Methods: *
Access-Control-Allow-Origin: http://localhost:55309
Access-Control-Expose-Headers: *
Access-Control-Max-Age: 86400
Connection: Keep-Alive
Content-Length: 0
Date: Mon, 13 Jun 2022 13:05:34 GMT
Server: GenHTTP/6.3.4.0
Vary: Origin
This returns 204 OK.
Get:
Request URL: http://localhost:55409/api/v1//metadata
Referrer Policy: strict-origin-when-cross-origin
Provisional headers are shown
Learn more
Accept: application/json, text/plain, */*
Authorization: Bearer <token>
Referer: http://localhost:55309/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36
Returns CORS error
This also feels weird as similar requests without Authorization header return 200 OK. The options seem to be correct in that it allows everything. Why does it fail then?
Also note that this works with the other server (more proper server running .Net Web Api), so client is not to blame. I think I'm missing some crucial server setup
Turns out I needed to specifically configure it to allow Authorization header, as if * does not include it?
inline.Add(new CorsPolicyBuilder().Default(new OriginPolicy(null, new List<string>() { "authorization" }, null, AllowCredentials: true, 86400u)));
This is the answer, but I'll mark the other answer whoever can explain why * does not include Authorization as OPTIONS has returned this
Access-Control-Allow-Headers: *
so I assume that should allow Authorization header as well

POST command missing BODY parameters in SSL (https:), but not in http:

Working on some code that that I inherited from a non-responsive initial developer. My ASP.NET and web.config are at best "dated", thus I'm turning to the community for some help. One of the first things I had to change is to force this website to operate in SSL (https:) as it deals with sensitive data. The program immediately stopped working and I had to make some undesirable changes to code that "already worked". And it still seems broken, and the changes won't make the client happy.
This is an ASP.NET project that seems hand-rolled.
Sending a POST command with some body text that (I think) is JSon setting additional parameters to the POST command such as: "indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505".
As far as I can tell, this works as intended on non SSL (i.e. http: requests). However, when running in SSL (i.e. https: requests), it appears that the BODY (Json text) isn't getting decoded into the HttpContect.Current.Request parameters (which seems to be happening in http:).
However the post_data that I can read from the input stream has the JSon body text (as clear text?) with the parameters, which my 'fix' adds to the incoming HttpContect.Current.Request parameters as a combined dictionary.
[Here is the RAW command intercepted with Fiddler] POST https://vmdev-xpp/BuilderQC/Services/Data.svc/QueryGridResults?typename=ImportReadyForDownload HTTP/1.1 Host: vmdev-xpp User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0 Accept: application/json, text/javascript, /; q=0.01 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Content-Type: application/x-www-form-urlencoded X-Requested-With: XMLHttpRequest Content-Length: 378 Origin: https://vmdev-xpp Connection: keep-alive Referer: https://vmdev-xpp/BuilderQC/BREDFileManagement.aspx Cookie: ASP.NET_SessionId=ehi2ccsdmekvkgegmfh11n1v; .ASPXLMPTest=2226D5725FBC10FBCCD606108CE5A4E32990EEA8FF8A1864496F69874F116D7E1ABF48A8BFD05EE683FE3F456D4475E88A61B19B299CB557209129BD25E87AC38CECA5303C7E2035E64C1F5A4AD2605D8581181A9C7E48680371F83BC7A93D7A63D8748EA4761A608F424578C20127D01DE0E2FBFBD5F079575E86FD506925D541026B7C8713FDEFE108BCEADBFC1DA0 Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: same-origin
_search=true&nd=1629052554858&rows=40&page=1&sidx=&sord=asc&QC_ProjectID=14&FileReadyForDownload=1&filters=%7B%22fields%22%3A%5B%7B%22field%22%3A%22QC_ProjectID%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%2214%22%7D%2C%7B%22field%22%3A%22FileReadyForDownload%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%221%22%7D%5D%7D&indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505&entity=false
[Here is the post_data I obtained from the input stream, I think that this being in clear-text is suspect] Post_data = _search=true&nd=1629052767566&rows=40&page=1&sidx=&sord=asc&QC_ProjectID=14&FileReadyForDownload=1&filters=%7B%22fields%22%3A%5B%7B%22field%22%3A%22FileName%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%22th%22%7D%2C%7B%22field%22%3A%22QC_ProjectID%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%2214%22%7D%2C%7B%22field%22%3A%22FileReadyForDownload%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%221%22%7D%5D%7D&indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505&entity=false&FileName=th
Here is the incoming HttpContext.Current.Request.Params.AllKeys, Notice the lacking "indexID" among other parameters

Chunked HTTP request

I try to use chunked http request on ODIN-W2 with mbedOS. I have next http request/response:
POST /post HTTP/1.1<CR><LF>
Accept: */*<CR><LF>
Accept-Encoding: gzip<CR><LF>
Accept-Language: uk,en-US;q=0.8,en;q=0.5,ru;q=0.3<CR><LF>
Cache-Control: no-cache<CR><LF>
Connection: Keep-Alive<CR><LF>
Host: httpbin.org<CR><LF>
Transfer-Encoding: chunked<CR><LF>
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like
Gecko<CR><LF>
content-type: application/json<CR><LF>
<CR><LF>
6<CR><LF>
ABCDEF<CR><LF>
0<CR><LF>
<CR><LF>
Status: 411 - Length Required<LF>
<LF>
Headers:<LF>
Date: Tue, 24 Apr 2018 08:01:41 GMT<LF>
Connection: close<LF>
Content-Type: text/html<LF>
Server: meinheld/0.6.1<LF>
Via: 1.1 vegur<LF>
<LF>
Body (92 bytes):<LF>
<html><head><title>Length Required</title></head><body><p>Length Required.
</p></body></html><LF>
I use standard HTTP library from mbedOS.
This is a bug in httpbin, not in the Mbed HTTP library. See https://github.com/requests/httpbin/issues/340. This is also the reason I used reqres.in as the example URL for chunked requests.

Why would a browser make two separate requests for the same file?

I'm debugging a program I wrote and noticed something strange. I set up an HTTP server on port 12345 that servers a simple OGG video file, and attempted to access it from Firefox.
Upon sniffing the network requests, I found these two requests were made:
GET /video.ogv HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
GET /video.ogv HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Range: bytes=8122368-
The video is almost 8 MB in size, so the fact that the second request specifics 8122368 bytes, which is 7932 KB, suggests it is requesting the very end of the file for some reason. Anyone have ideas?
In order to support seeking and playing back regions of the media that aren't yet downloaded, Gecko uses HTTP 1.1 byte-range requests to retrieve the media from the seek target position. So because Ogg files don't contain their duration, the initial download connection is terminated. Then there is a seek to the end of the Ogg file and read a bit of data to extract the time duration of the media. Info from here and here.
Some media format have meta data at the end of the file, and this data is usually required to allow proper seeking of the video.
Its actually requesting 8122368 bytes starting backwards from the end. Which is 7.74MB if I did my calcs correctly.
it might be something in how the buffering for that file type is done.

Http get request packet size in bytes

How many bytes of data does a typical HTTP get request consume.
For instance if I request a page from the server through a browser how many bytes of data would be sent?
Pretty typical request, 430 bytes:
GET /ga.js HTTP/1.1\r\n
Host: www.google-analytics.com\r\n
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)\r\n
Accept: */*\r\n
Accept-Language: en-us,en;q=0.5\r\n
Accept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
Keep-Alive: 300\r\n
Connection: keep-alive\r\n
Referer: http://stackoverflow.com/\r\n
If-Modified-Since: Mon, 31 Aug 2009 17:13:58 GMT\r\n
\r\n
\r\n
Request with a long query string and a small cookie 657 bytes)
GET /pixel;r=978178957;fpan=0;fpa=1241112640-44259546-69321280;ns=0;url=http%3A%2F%2Fstackoverflow.com%2F;ref=;ce=1;je=1;sr=1920x1200x32;dg=E5912-W-MO-5;dst=1;et=1252061014745;tzo=-120;a=p-c1rF4kxgLUzNc HTTP/1.1\r\n
Host: pixel.quantserve.com\r\n
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)\r\n
Accept: image/png,image/*;q=0.8,*/*;q=0.5\r\n
Accept-Language: en-us,en;q=0.5\r\n
Accept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
Keep-Alive: 300\r\n
Connection: keep-alive\r\n
Referer: http://stackoverflow.com/\r\n
Cookie: uid=1274108650-45267447-66848880; mc=1137458542-57565784-88898864\r\n
\r\n
\r\n
Use Fiddler to intercept the request and see for yourself.
It varies, especially when it comes to GET queries or POST requests, but I'd estimate it about 0.5—1k.
Requesting a page from the browser, though, may also result in requesting pictures, stylesheets and other referenced content.
Edit: originally I put in the estimation for request+reply.
I would suggest you use a full packet sniffer like wireshark. You would love it :)
Get it here:
http://www.wireshark.org/

Resources