Thats from wikipedia:
For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always and anyways acceptable, even if not listed in the TE (transfer encoding) request header field
Thats what I get from clients (Mozilla, Opera):
GET http://www.google.com/ HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Apparently there is neither Transfer-Encoding field there, nor I see any chunks (I've checked with HEX editor, no additional symbols).
I open connection as follows (Python)
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Is it lower level handling joins chunks into message? Is so, how can I know where the HTTP message ends so that I can stop reading the request and start handling it?
You should read the specification.
But simply, in this case, since it's a GET, and there's not content, there's not going to be a Content-Length header. So, you stop reading when you get the empty line with just a CR/LF.
Otherwise, you read past that blank line, and read Content-Length bytes.
Related
I get an Error 416 Range Not Satisfiable response when I make an http request with a byte-range of 0-65536. The byte length of the file requested is only 3356.
Reading the spec on byte-range, it sounds like requesting a range that extends beyond the length of the file is okay, and in this case I would expect the entire file to be fetched. https://httpwg.org/specs/rfc9110.html#byte.ranges
A client can limit the number of bytes requested without knowing the size of the selected representation. If the last-pos value is absent, or if the value is greater than or equal to the current length of the representation data, the byte range is interpreted as the remainder of the representation (i.e., the server replaces the value of last-pos with a value that is one less than the current length of the selected representation).
Am I misreading the spec or is this request truly unsatisfiable? I prefer if my client does not need to know the size of the file before making the request; they could vary widely in size. The request is successful if I limit the byte-range to <= the length of the file.
The file requested is a cloud-optimized-geotiff. It is being served by a vite dev server. And the request is made using OpenLayers and geotiff.js.
Here are the DevTools details of the request and response:
General
---------
Request URL: http://localhost:3000/scenarios/1/1_parcel_fill.tif
Request Method: GET
Status Code: 416 Range Not Satisfiable
Remote Address: [::1]:3000
Referrer Policy: strict-origin-when-cross-origin
Response Headers
--------------------
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 0
Content-Range: bytes */3356
Date: Wed, 21 Sep 2022 21:38:19 GMT
Keep-Alive: timeout=5
Request Headers
-------------------
Accept: */*
Accept-Encoding: identity
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: localhost:3000
Pragma: no-cache
Range: bytes=0-65536
Referer: http://localhost:3000/
sec-ch-ua: "Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
Working on some code that that I inherited from a non-responsive initial developer. My ASP.NET and web.config are at best "dated", thus I'm turning to the community for some help. One of the first things I had to change is to force this website to operate in SSL (https:) as it deals with sensitive data. The program immediately stopped working and I had to make some undesirable changes to code that "already worked". And it still seems broken, and the changes won't make the client happy.
This is an ASP.NET project that seems hand-rolled.
Sending a POST command with some body text that (I think) is JSon setting additional parameters to the POST command such as: "indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505".
As far as I can tell, this works as intended on non SSL (i.e. http: requests). However, when running in SSL (i.e. https: requests), it appears that the BODY (Json text) isn't getting decoded into the HttpContect.Current.Request parameters (which seems to be happening in http:).
However the post_data that I can read from the input stream has the JSon body text (as clear text?) with the parameters, which my 'fix' adds to the incoming HttpContect.Current.Request parameters as a combined dictionary.
[Here is the RAW command intercepted with Fiddler] POST https://vmdev-xpp/BuilderQC/Services/Data.svc/QueryGridResults?typename=ImportReadyForDownload HTTP/1.1 Host: vmdev-xpp User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0 Accept: application/json, text/javascript, /; q=0.01 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Content-Type: application/x-www-form-urlencoded X-Requested-With: XMLHttpRequest Content-Length: 378 Origin: https://vmdev-xpp Connection: keep-alive Referer: https://vmdev-xpp/BuilderQC/BREDFileManagement.aspx Cookie: ASP.NET_SessionId=ehi2ccsdmekvkgegmfh11n1v; .ASPXLMPTest=2226D5725FBC10FBCCD606108CE5A4E32990EEA8FF8A1864496F69874F116D7E1ABF48A8BFD05EE683FE3F456D4475E88A61B19B299CB557209129BD25E87AC38CECA5303C7E2035E64C1F5A4AD2605D8581181A9C7E48680371F83BC7A93D7A63D8748EA4761A608F424578C20127D01DE0E2FBFBD5F079575E86FD506925D541026B7C8713FDEFE108BCEADBFC1DA0 Sec-Fetch-Dest: empty Sec-Fetch-Mode: cors Sec-Fetch-Site: same-origin
_search=true&nd=1629052554858&rows=40&page=1&sidx=&sord=asc&QC_ProjectID=14&FileReadyForDownload=1&filters=%7B%22fields%22%3A%5B%7B%22field%22%3A%22QC_ProjectID%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%2214%22%7D%2C%7B%22field%22%3A%22FileReadyForDownload%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%221%22%7D%5D%7D&indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505&entity=false
[Here is the post_data I obtained from the input stream, I think that this being in clear-text is suspect] Post_data = _search=true&nd=1629052767566&rows=40&page=1&sidx=&sord=asc&QC_ProjectID=14&FileReadyForDownload=1&filters=%7B%22fields%22%3A%5B%7B%22field%22%3A%22FileName%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%22th%22%7D%2C%7B%22field%22%3A%22QC_ProjectID%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%2214%22%7D%2C%7B%22field%22%3A%22FileReadyForDownload%22%2C+%22op%22%3A+%22cn%22%2C+%22value%22%3A%221%22%7D%5D%7D&indexID=8379fcd1-5083-4d1c-a6ee-5812f134a505&entity=false&FileName=th
Here is the incoming HttpContext.Current.Request.Params.AllKeys, Notice the lacking "indexID" among other parameters
I need to interact with a remote HTTP server at the lowest possible level (i.e.: at socket level) because my target is a very small embedded system with no support for higher level libraries (it's a bare-metal uController wit no O.S. at all and talking to a GSM modem via serial line; modem has some support for sockets, but nothing above that).
Basic need is to upload a "file" using POST.
I have all needed Header/Body in place and it "usually works".
Problem is I randomly get a "HTTP/1.1 502 Bad gateway error" response and this is more likely to happen as the size of "file" increases.
I understand this means there's some problem between the reverse proxy frontend (nginx, apparently) and the backends, but I have absolutely no control on those (actually I dont't really know how the atual setup besides what can be gleamed from (light) probing).
My current strategy is to open a plain socket and send the folowing sequence (dots represent binary data):
POST /path/to/websend.php HTTP/1.0
Host: host.domain.tld
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:33.0) Gecko/20100101 Firefox/33.0
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Type: multipart/form-data; boundary=AaB03x
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Accept: */*
Content-Length: <full_length>
--AaB03x
Content-Disposition: form-data; name="IV"
Content-Type: application/data
Content-Transfer-Encoding: binary
000102030405060708090A0B0C0D0E0F
--AaB03x
Content-Disposition: form-data; name="S_TXT_FILE"; filename="FILENAME_s.txt"
Content-Type: application/data
Content-Transfer-Encoding: binary
..............................................................
..............................................................
...... several 512byte blocks ................................
..............................................................
..............................................................
--AaB03x--
Is there something I could do to enhance reliability?
I already do multiple retries and this actually works, but sometimes I need to retry six or more times to have a positive answer (200 OK).
Note I send exactly the same sequence on rety and it succeeds... eventually.
I need to send two parts because content is encrypted and first part is the neded "Initialization Vector".
I've inherited a website that contains about 100 audio files. The links to the files are relative links like this:
part 1
Back in the day those usually forced a download. Newer browsers now play the audio in browser. Except....
If the user comes to the site over https they are able to navigate the site and the html pages load, but the links to the audio files generate a 403 Forbidden error. Changing the protocol in the location http allows the mp3 to load and playback in the browser.
Why would the mp3 files be forbidden over https?
Is there a way to force the http protocol without having to make all the links absolute links? I notice the relative links "inherit" the protocol of the page they were loaded on. There isn't anything on any of these pages that need https so I wouldn't mind forcing all the parent pages to load over http....
This is a departmental site within a giant university. So I don't have access to the server, htaccess, or any of those kinds of tricks. All in browser, javascript, html solutions please.
UPDATE
I installed Firebug to view the headers and discovered that the audio plays fine in FireFox (on my mac). In Safari they load and play, but the controls don't show the progress or time, but they do play. And in Chrome they don't play at all.
I had also checked them on my PC at work and they don't play in IE9 (I know! Corporate IT, right?) or Chrome.
Here is what I get for headers in Firefox where the audio plays fine.
HTTP/1.1 200 OK
Date: Sat, 11 Apr 2015 15:39:04 GMT
Server: Apache
WWW: www3
Vary: X-Forwarded-Proto
Last-Modified: Tue, 16 Nov 2010 14:19:25 GMT
Etag: "78e935-d60ac-4952c3e68d540"
Accept-Ranges: bytes
Content-Length: 876716
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
GET /dept/area/language/stories/sounds/file.mp3 HTTP/1.1
Host: example.edu
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0) Gecko/20100101 Firefox/33.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://example.edu/dept/area/language/stories.html
Cookie: _ga=GA1.2.829124232.1405280613; BIGipServerWWW-HTTP=1378527424.20480.0000; _gat=1
Connection: keep-alive
And these are what I get in Chrome.
Remote Address:128.122.119.202:443
Request URL:https://example.edu/dept/area/language/stories/sounds/file.mp3
Request Method:GET
Status Code:206 Partial Content
HTTP/1.1 206 Partial Content
Date: Sat, 11 Apr 2015 15:46:12 GMT
Server: Apache
WWW: www4
Vary: X-Forwarded-Proto
Last-Modified: Tue, 16 Nov 2010 14:19:12 GMT
ETag: "78e939-158dbc-4952c3da27800"
Accept-Ranges: bytes
Content-Length: 1
Content-Range: bytes 382271-382271/1412540
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: audio/mpeg
GET /dept/area/language/stories/sounds/file.mp3 HTTP/1.1
Host: www.nyu.edu
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
DNT: 1
Referer: https://example.edu/dept/area/language/stories.html
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8,es;q=0.6,hi;q=0.4,pt;q=0.2
Cookie: _ap_utmz=57748789.1416681263.3.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); _ap_utma=57748789.722895429.1387124094.1423327171.1425612794.7; __utma=57748789.194555315.1387124094.1423327171.1425612794.7; __utmz=57748789.1416681262.3.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BIGipServerWWW-HTTP=1395304640.20480.0000; _gat=1; _ga=GA1.2.194555315.1387124094
Range: bytes=382271-382271
If-Range: "78e939-158dbc-4952c3da27800"
HTTP Request and Response Headers
Make sure to read about headers, mime types and content encodings.
You could try to utilise the Content-Disposition response header
An opportunity to raise a "File Download" dialogue box for a known MIME type with binary format or suggest a filename for dynamic content. Quotes are necessary with special characters.
Source: Wikipedia
Anyway your issue seems like a http header issue, could be compression as well. Take a look at your headers and whats different and troubleshoot from there. When you understood the problem, you can think of solutions.
Troubleshooting Tools
Use firebug or chrome developer tools to investigate. Fiddler Proxy to simulate different headers, since you have no access to your server.
File Permissions
Could be that SSL runs as another user or config on your server and the mp3 files have the wrong permissions or their parent directory. You need to check those, but since you have no server access you could be out of luck.
However, if SSL is not important to you just link to the files like so:
<a href="http://yourDomain.tld/folder/anotherFolder/file.mp3">
This will enforce the http protocol being used for the links. Most likely this results in the SSL chain being broken due to the mix in of http traffic into your ssl secured traffic. Therefore there's another alternative to achieve what you want:
Meta Refreshes
<meta http-equiv="refresh" content="3; URL=http://www.yourNonSSLDomain.tld/">
This will redirect to your non-SSL website where you can make sure to not mix https and http resources in your html document.
I'm debugging a program I wrote and noticed something strange. I set up an HTTP server on port 12345 that servers a simple OGG video file, and attempted to access it from Firefox.
Upon sniffing the network requests, I found these two requests were made:
GET /video.ogv HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
GET /video.ogv HTTP/1.1
Host: 127.0.0.1:12345
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Range: bytes=8122368-
The video is almost 8 MB in size, so the fact that the second request specifics 8122368 bytes, which is 7932 KB, suggests it is requesting the very end of the file for some reason. Anyone have ideas?
In order to support seeking and playing back regions of the media that aren't yet downloaded, Gecko uses HTTP 1.1 byte-range requests to retrieve the media from the seek target position. So because Ogg files don't contain their duration, the initial download connection is terminated. Then there is a seek to the end of the Ogg file and read a bit of data to extract the time duration of the media. Info from here and here.
Some media format have meta data at the end of the file, and this data is usually required to allow proper seeking of the video.
Its actually requesting 8122368 bytes starting backwards from the end. Which is 7.74MB if I did my calcs correctly.
it might be something in how the buffering for that file type is done.