I found that a get message header looks like:
:method: GET
:scheme: https
:authority: server.net
:path: /config
accept: */*
accept-encoding: gzip,deflate
What a connect message header should look like?
This example is from the RFC of http2:
GET /resource HTTP/1.1 HEADERS
Host: example.org ==> + END_STREAM
Accept: image/jpeg + END_HEADERS
:method = GET
:scheme = https
:path = /resource
host = example.org
accept = image/jpeg
I want to know the equivalent of the connect header in http2.
In Http1 is:
CONNECT example.org:443 HTTP/1.1
Host: example.org:443
The format of the CONNECT method in HTTP/2 is specified in section 8.3.
With the formatting you used above looks like:
:method: CONNECT
:authority: proxy.net:8080
As specified, :scheme and :path must be omitted.
The HTTP/2 CONNECT method can also be used for bootstrapping other protocols (see for example WebSocket over HTTP/2), so that, additionally, the :protocol pseudo-header may also be present.
Remember however that this is only a textual representation of HTTP/2; the bytes that actually travel over the network are different since you must encode them using HPACK.
Unless you are actually writing an HTTP/2 implementation, it is better that you use existing libraries (available in virtually any programming language) to send HTTP/2 requests (of any kind): the libraries will take care of converting your CONNECT request into the proper bytes to send over the network.
Related
Thats from wikipedia:
For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always and anyways acceptable, even if not listed in the TE (transfer encoding) request header field
Thats what I get from clients (Mozilla, Opera):
GET http://www.google.com/ HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Apparently there is neither Transfer-Encoding field there, nor I see any chunks (I've checked with HEX editor, no additional symbols).
I open connection as follows (Python)
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Is it lower level handling joins chunks into message? Is so, how can I know where the HTTP message ends so that I can stop reading the request and start handling it?
You should read the specification.
But simply, in this case, since it's a GET, and there's not content, there's not going to be a Content-Length header. So, you stop reading when you get the empty line with just a CR/LF.
Otherwise, you read past that blank line, and read Content-Length bytes.
I am developing an HTTP proxy in Java. I resend all the data from client to server without touching it, but for some URLs (for example this) server returns the 404 error if I am connecting through my proxy.
The requested URL uses Varnish caching, so it might be the root of problem. I cannot reconfigure it - it is not my.
If I request that URL directly with browser, the server returns 200 and the image is shown correctly.
I am stuck because I even do not know what to read and how to compose a search request.
Thanks a lot.
Fix the Host: header of the re-issued request. The request going out from the proxy either has no Host header or it is broken (or only X-Host exists). Also take note that the proxy application will execute its own DNS lookup and that might yield a different IP address than your local computer (where you issued the original request).
This works:
> curl -s -D - -o /dev/null http://212.25.95.152/w/w-200/1902047-41.jpg -H "Host: msc.wcdn.co.il"
HTTP/1.1 200 OK
Content-Type: image/jpeg
Cache-Control: max-age = 315360000
magicmarker: 1
Content-Length: 27922
Accept-Ranges: bytes
Date: Sun, 05 Jul 2015 00:52:08 GMT
X-Varnish: 2508753650 2474246958
Age: 67952
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT
I have a program that activate an chip for racing results. (Its just a piece of hardware).
I watch with Fiddler (Sniffing program) the in and outgoing traffic from my pc when I connect the chip with my computer.
The program sends the following HTTP Request:
POST http://example.com/index.php HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 185
Content-Type: application/x-www-form-urlencoded
Host: example.com
Pragma: no-cache
User-Agent: SomeProgram 1.2.3
Data==%0D%0AAjlFNEEw-SOMELONGSECRETKEY-RGAw%3D%3D%0D%0A
I receive the following response:
<?xml version="1.0" encoding="utf-8"?>
<message type="3" result="1" txid="someid" activationdate="" availablecredits="732" firstname="John" lastname="Doe" email="JohnDoe#outlook.com" phonenumber="00123445" notification_email="1" notification_text="1"/>
Is it possible to edit the response so that when the programs check for the availablecredits variable, he get the value 9999 instead of 732.
Im working on a Windows 8 laptop.
Definitely - Fiddler allows you to modify requests and responses by adding rules to FiddlerScript. Citing Fiddler documentation:
To make custom changes to web requests and responses, use
FiddlerScript to add rules to Fiddler's OnBeforeRequest or
OnBeforeResponse function. Which function is appropriate depends on
the objects your code uses: OnBeforeRequest is called before each
request, and OnBeforeResponse is called before each response.
So, all you have to do is to add to OnBeforeResponse the logic for replacing the availablecredits attribute value with any value you desire.
I can upload a file to my Apache web server using Curl just fine:
echo "[$(date)] file contents." | curl -T - http://WEB-SERVER/upload/sample.put
However, if I put a Squid proxy server in between, then I am not able to:
echo "[$(date)] file contents." | curl -x http://SQUID-PROXY:3128 -T - http://WEB-SERVER/upload/sample.put
Curl reports the following error:
Note: This error response was in HTML format, but I've removed the tags for ease of reading.
ERROR: The requested URL could not be retrieved
ERROR
The requested URL could not be retrieved
While trying to retrieve the URL:
http://WEB-SERVER/upload/sample.put
The following error was encountered:
Unsupported Request Method and Protocol
Squid does not support all request methods for all access protocols.
For example, you can not POST a Gopher request.
Your cache administrator is root.
My squid.conf doesn't seem to be having any ACL/rule that should disallow based on the src or dst IP addresses, or the protocol, or the HTTP method... as I can do an HTTP POST just fine between the same client and the web server, with the same proxy sitting in between.
In case of the failing HTTP PUT case, to see the request and response traffic that was actually occurring, I placed a netcat process in between Curl and Squid, and this is what I saw:
Request:
PUT http://WEB-SERVER/upload/sample.put HTTP/1.1
User-Agent: curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Host: WEB-SERVER
Pragma: no-cache
Accept: */*
Proxy-Connection: Keep-Alive
Transfer-Encoding: chunked
Expect: 100-continue
Response:
HTTP/1.0 501 Not Implemented
Server: squid/2.6.STABLE21
Date: Sun, 13 May 2012 02:11:39 GMT
Content-Type: text/html
Content-Length: 1078
Expires: Sun, 13 May 2012 02:11:39 GMT
X-Squid-Error: ERR_UNSUP_REQ 0
X-Cache: MISS from SQUID-PROXY-FQDN
X-Cache-Lookup: NONE from SQUID-PROXY-FQDN:3128
Via: 1.0 SQUID-PROXY-FQDN:3128 (squid/2.6.STABLE21)
Proxy-Connection: close
<SNIPPED the HTML error response already shown earlier above>
Note: I have anonymized the IP addresses and server names throughout for readability reasons.
Thanks to Amos Jeffries for answering this on squid-users forum. The issue is basically that Squid before version 3.1 does not implement HTTP 1.1 and thus rejects the chunked transfer encoding.
Given this snippet:
(defroutes main-routes
(POST "/input/:controller" request
(let [buff (ByteArrayOutputStream.)]
(copy (request :body) buff)
;; --- snip
The value of buff will be a non-empty byte array iff there's the Content-Type header in the request. The value can be nonsencial, the header just has to be there.
However, I need to dump the body (hm... that came out wrong) if the request came without a content type, so that the client can track down the offending upload. (The uploading software is not under my control and its maintainers won't provide anything extra in the headers.)
Thank you for any ideas on how to solve or work around this!
EDIT:
Here are the headers I get from the client:
{
"content-length" "159",
"accept" "*/*",
"host" (snip),
"user-agent" (snip)
}
Plus, I discovered that Ring, using an instance of Java's ServletRequest, fills in the content type with the standard default, x-www-form-urlencoded. I'm now guessing that HTTPParser, which supplies the body through HTTPParser#Input, can't parse it correctly.
I face the same issue. It's definitely one of the middleware not being able to parse the body correctly and transforming :body. The main issue is that the Content-Type suggest the body should be parsable.
Using ngrep, I found out how curl confuses the middleware. The following, while intuitive (or rather sexy) on the command line sends a wrong Content-Type which confuses the middleware:
curl -nd "Unknown error" http://localhost:3000/event/error
T 127.0.0.1:44440 -> 127.0.0.1:3000 [AP]
POST /event/error HTTP/1.1.
Authorization: Basic SzM5Mjg6ODc2NXJkZmdoam5idmNkOQ==.
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3.
Host: localhost:3000.
Accept: */*.
Content-Length: 13.
Content-Type: application/x-www-form-urlencoded.
.
Unknown error
The following however forces the Content-Type to being opaque and the middleware will not interfere with the :body.
curl -nd "Unknown error" -H "Content-Type: application/data" http://localhost:3000/event/error
T 127.0.0.1:44441 -> 127.0.0.1:3000 [AP]
POST /event/error HTTP/1.1.
Authorization: Basic SzM5Mjg6ODc2NXJkZmdoam5idmNkOQ==.
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3.
Host: localhost:3000.
Accept: */*.
Content-Type: application/data.
Content-Length: 13.
.
Unknown error
I'm considering replacing the middleware with a more liberal one because even though the request is wrong, I'd still like to be able to decide what to do with the body myself. It's a really weird choice to zero the request body when the request doesn't make sense. I actually think a more correct behavior would be to pass it to an error handler which by default would return a 400 Bad Request or 406 Not Acceptable.
Any thoughts on that? In my case I might propose a patch to Compojure.
According to:
http://mmcgrana.github.com/ring/ring.middleware.content-type-api.html
the default content type is application/octet-stream. Unless you actively support that content type, can't you just check if the content type matches that one, and then dump whatever you need based on that?