I've been dabbling with the underlying client-server communications (cURL, browsers, and server responses, among a variety of others). I think at this point I understand the basic structure of a request and response:
Request:
(Method) (File) (Protocol)
(Headers)
(Empty)
(Body?)
Response:
(Protocol) (Code) (Meaning)
(Headers)
(Empty)
(Body?)
This was working fine until I sent compressed information. More precisely, I used gzip to send HTML in the response.
First, I'm using an updated version of Chrome to test -- and if I use a prebuilt HTTP server solution, gzip (In exactly the way I'm using it now) works fine.
In the 'Accept-Encoding' header sent from Chrome, it states that it will accept gzip. My guess is there's something specific I haven't run across yet (And haven't been able to find through searching).
All that said, here's the response:
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/html
Content-Encoding: gzip
Content-Length: 205
(Zipped data...)
If I remove gzip and leave everything else exactly the same, this works fine. The Content-Length is the length of the zipped data, not the raw data, though I commented that out to make sure it wasn't doing something I wasn't expecting. Chrome doesn't 'require' Content-Length to figure out what it wants to do.
At one stage, I also had Content-Type: text/html; charset=utf-8 which didn't make a difference (Underneath the zip, the string is converted to utf-8 on the server).
Originally I zipped the headers and the body for consistency (And lowest-length). I figured it failed because the headers had to be visible to the browser to figure out the body would be zipped.
In both cases, however, Chrome displays "ERR_CONTENT_DECODING_FAILED" which is classic "We don't know how to decode the information."
Prior to trying to embed this into the same response, I sent the headers first and the body second, thinking that may be the best approach -- two different responses.
Chrome took this to mean the server was pushing downloadable content and entered an infinite acceptance loop, so clearly that wasn't the solution. In addition, the content it received wasn't used.
Technically, a normal web server would end the transmission after sending the second response, but I'm experimenting without that to better understand the connections.
Is there something I'm missing, or are there any thoughts on what might be going wrong?
Thanks!
Edit
The code is scattered out across various functions (I doubt anyone wants to look at ~300 lines), but I picked out the relevant pieces. Right before gzip, self.written is html.
const zipper = require('zlib');
push(value, callback)
{
this.socket.write(value, callback);
}
let self = this;
zipper.gzip(self.written, function(error, zipped)
{
self.setHeaders(
{
'Content-Encoding': 'gzip',
'Content-Length': zipped.length
});
self.getHeaderString(function(error, headers)
{
self.push(headers + zipped, callback);
});
});
When I console.log the final response:
Related
I'm trying to send AT commands to my ESP32* module and am not getting any response back.
I need to perform a POST request that contains the username and password and other requests later on. I am not structuring these correctly and there is not a lot of good documentation for this.
NOTE: because I cannot share my complete url due to privacy I will use something with the same length ********connected.com:443
Send login information to ********connected.com/login (POST) body{"email":"myemail.ca", "password":"xxxxx"}
once I get the token I will make other requests.
get information regarding user profile ********connected.com/getRoutine ( GET) query param username="bob"
I really want to understand how these requests are structured so if someone can explain it to me elegantly that would be great!
Here is what I have tried..
AT
OK
AT+CIPSTART="TCP","********connected.com",443
CONNECT
OK
AT+CIPSEND=48
> "GET ********connected.com:443/getUsersOnline"
OK
>
Recv 48 bytes
SEND OK
CLOSED
REQUESTED POST REQUEST I HAVE USED
AT+CIPSEND=177 “POST \r Host: ********connected.com\r\n Accept: application/json\r\n Content-Length: 224r\n Content-Type: application/jsonr\n { "email":"myemail.com", "password":"myPassword" } “
There are actually several parts of your system that might be the cause of the malfunctioning:
The AT commands sent (it is not clear how you check for server responses. Responses could proviede clues about what's wrong)
The server side app seems to be a custom implementation that might have bugs as well
The POST request might be malformed
Let's focus on the last one.
POST are described in RFC 7231, and though it is an obscure description without examples, it makes one thing clear: there's not actually a well defined standard... because it is strictly application dependant!
I also quote the relevant part of this brilliant answer to an old SO question:
When receiving a POST request, you should always expect a "payload", or, in HTTP terms: a message body. The message body in itself is pretty useless, as there is no standard.
For this reason, all we can do is to build a POST request as accurate as possible and then to debug the system as a whole thing making sure that the request matches what expected by the server side application.
In order to do this, let's check another external link I found: POST request examples. We found this request:
POST /test HTTP/1.1
Host: foo.example
Content-Type: application/x-www-form-urlencoded
Content-Length: 27
field1=value1&field2=value2
Now let's compare this example to your request:
POST
Host: ********connected.com
Accept: application/json
Content-Length: 224
Content-Type: application/jsonr
{ "email":"myemail.com", "password":"myPassword" }
You are saying to the server that you want to pass a resource to an unspecified application (no path), and that this resource is 224 bytes long (wrong! Message body is shorter).
For these reasons, at least these things can be improved:
POST /path/invic18app.php HTTP/1.1 //The path to the application and HTTP version are missing
Content-Length: 48 //This must be the length of the message body, without the header. I also would write it as the last option, just before message body
Write TWO empty lines before message body, otherwise the server will interpret it as further (wrong) options
I hope this helps you, even if it is a tentative answer (I cannot try this request myself). But, again, you definitely need to sniff packets a TCP levels, in order to avoid debugging the server if you are not sure that data is actually received! If you cannot install Wireshark, also tcpdump will be ok.
I've an HTTP client sending many POST requests to a server. The server responds to all requests with 201 Created and a response body. For my purposes, the response header is enough, as I'm only interested in the Location header. I'd like to avoid that the server produces a response body in order to significantly decrease network traffic.
According to RFC 7231, ...
[...] if one or more resources has been created on the origin server as a
result of successfully processing a POST request, the origin server
SHOULD send a 201 (Created) response containing a Location header [...]
..., thus, I assume, the server COULD also respond e.g. with 204 No Content, omiting the body.
Therefore my question: Is it possible to construct a POST request which makes the server respond with 204 No Content or to omit the response body in another way?
Update 1: The server side is a Spring Data REST project and I'm free to configure it. I know that I could set RepositoryRestConfiguration#setReturnBodyOnCreate to false, but that would be overdone as it affects all incoming requests. Therefore, I'd prefer to make the decision on the client side.
There's no real lever you can pull from the client side to control if the server will respond with a body or not, unless the service you work with has a specific feature that allows this.
A header that a server might use is Prefer: return=minimal but if the service doesn't explicitly document support for this, chances are low that this will work.
Really the only think you can do one the client is to:
Kill the TCP connection as soon as you got the response headers
Kill the HTTP/2 stream when you recieved the headers.
This is a pretty 'drastic' thing but clients do use this mechanism for some cases and it does work. However, if the POST response body was somewhat small there's a chance that it's not really making a ton of difference because the response might already have been sent.
There is no way to do it client side only as it is not natively implemented in Spring REST server.
Anyway, any client demand can be transformed as an extra custom header or a query parameter in the request.
A way could be to override default response handlers and detect custom header (implements Prefer: return=minimal as suggested before for instance) and/or query param presence to trigger an empty response with a 204 status. This post may help you to figure it out.
Can you try changing your client such that you
a) Query the server with HTTP HEAD requests instead of POST requests
b) Analyze the response headers. There is no response body for HEAD requests as the purpose of HEAD requests is very similar to your requirement
c) Perform the necessary POST requests only when required
I understand that you may have difficulties at the client end to apply these changes. But, in the longer run, I believe this would be worth it.
Based on Evert's and Bertrand's answers plus a bit of googling, I finally implemented the following interceptor in the Spring Data REST server:
#Configuration
class RepositoryConfiguration {
#Bean
public MappedInterceptor preferReturnMinimalMappedInterceptor() {
return new MappedInterceptor(new String[]{"/**"}, new HandlerInterceptor() {
#Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
if ("return=minimal".equals(request.getHeader("prefer"))) {
response.setContentLength(0);
response.addHeader("Preference-Applied", "return=minimal"");
}
return true;
}
});
}
}
It produces the following communication, which is good enough for my purposes:
> POST /versions HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Type: application/json
> Prefer: return=minimal
> Content-Length: 123
>
> [123 bytes data]
...
< HTTP/1.1 201
< Preference-Applied: return=minimal
< ETag: "0"
< Last-Modified: Fri, 30 Nov 2018 12:37:57 GMT
< Location: http://localhost:8080/versions/1
< Content-Type: application/hal+json;charset=UTF-8
< Content-Length: 0
< Date: Fri, 30 Nov 2018 12:37:57 GMT
I would like to share the bounty evenly, but this is not possible. It goes to Bertrand, as he came with an answer which guided me to the very implementation. Thanks for your help.
This question was motivated by the answers here:
What to do with errors when streaming the body of an Http request
In this case, I have already written a HTTP 200 OK header, then I need to amend this if there is an error, by writing a trail header that says there was an error after writing a success header.
I have this Node.js code:
const writeResponse = function(file: string, socket: Socket){
socket.write([
'HTTP/1.1 200 OK',
'Content-Type: text/javascript; charset=UTF-8',
'Content-Encoding: UTF-8',
'Accept-Ranges: bytes',
'Connection: keep-alive',
].join('\n') + '\n\n');
getStream(file)
.pipe(socket)
.once('error', function (e: any) {
// there was an error
// how can I write trail headers here ?
s.write('some bad shit happened\n')
});
}
how do I write a useful trail header to the response that can be displayed well by the browser?
I think this is the relevant spec for trail headers:
https://www.rfc-editor.org/rfc/rfc2616#section-14.40
I think they should be called "trailing headers", but whatever.
Firstly:
I think this is the relevant spec for trail headers: https://www.rfc-editor.org/rfc/rfc2616#section-14.40
RFC 2616 has been obsoleted by RFC 7230. The current spec for trailers is RFC 7230 § 4.1.2.
Secondly:
].join('\n') + '\n\n'
Lines in HTTP message framing are terminated with \r\n, not \n.
Thirdly:
Content-Encoding: UTF-8
Content-Encoding is for content codings (like gzip), not charsets (like UTF-8). You probably don’t need to indicate charset separately from Content-Type.
And lastly:
how do I write a useful trail header to the response that can be displayed well by the browser?
You don’t. Mainstream Web browsers do not care about trailers.
See also (by the same user?): How to write malformed HTTP response to “guarantee” something akin to HTTP 500
I need to test some client application code I've written to test its' handling of various status codes returned in an HTTP response from a web server.
I have Fiddler 2 (Web Debugging Proxy) installed and I believe there's a way to modify responses using this application, but I'm struggling to find out how. This would be the most convenient way, as it would allow me to leave both client and server code unmodified.
Can anyone assist as I'd like to intercept the HTTP response being sent from server to client and modify the status code before it reaches the client?
Any advice would be much appreciated.
Ok, so I assume that you're already able to monitor your client/server traffic. What you want to do is set a breakpoint on the response then fiddle with it before sending it on to the client.
Here are a couple of different ways to do that:
Rules > Automatic Breakpoints > After Responses
In the quickexec box (the black box at the bottom) type "bpafter yourpage.svc". Now Fiddler will stop at a breakpoint before all requests to any URL that contains "yourpage.svc". Type "bpafter" with no parameters to clear the breakpoint.
Programmatically tamper with the response using FiddlerScript. The best documentation for FiddlerScript is on the official site: http://www.fiddler2.com/Fiddler/dev/
Once you've got a response stopped at the breakpoint, just double click it to open it in the inspectors. You've got a couple of options now:
Right next to the green Run to Completion button (which you click to send the response) there's a dropdown that lets you choose some default response types.
Or, on the Headers inspector, change the response code & message in the textbox at the top.
Or, click the "Raw" inspector and mess with the raw response to do arbitrary things to it. Also a good way to see what your client does when it gets a malformed response, which you'll probably test accidentally :)
Another alternative is to use Fiddler's AutoResponder tab (on the right-hand panel). This allows you to catch a request to any URI that matches a string and serve a "canned" response from a file. The file can contain both headers and payload. The advantage of this approach is that you don't have to write FiddlerScript and you don't have to handle each request manually via a breakpoint.
You would set the rule up in Fiddler like shown below (ensure you enable unmatched requests passthrough otherwise all other http requests will fail).
In this example, any request whose URI includes "fooBar" will get the canned response. The format of the file will vary depending on your APIs (you can use your browser to intercept a "real" response and base it on that) but mine looked like the following:
HTTP/1.1 409 Conflict
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.5; JBoss-5.0/JBossWeb-2.1
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT, PATCH, OPTIONS
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Max-Age: 86400
Content-Type: application/vnd.api+json
Content-Length: 149
Date: Tue, 28 Mar 2017 10:03:29 GMT
{"errors":[{"code":"OutOfStock","detail":"Item not in stock","source":{"lineId":{"type":"Order line Number","id":"1"}},"meta":{"availableStock":0}}]}
I found that it needed a carriage return at the end of the last line (i.e. after the json), and that the Content-Length header had to match the number of characters in the json, otherwise the webapp would hang. Your mileage may vary.
Create a FiddlerScript rule. Here's what I used in order to generate a local copy of a website that was intentionally using 403 on every page to thwart HTTrack/WGET.
https://gist.github.com/JamoCA/22db8d68a9a2fb20cb04a85360185333
/* 20180615 Fiddler rule to ignore all 403 HTTP Status errors so WGET or HTTrack can generate local copy of remote website */
SCENARIO: Changing the user agent or setting a delay isn't enough and the entire remote server is configured to respond w/403.
CONFIGURE: Add below rule to FiddlerScript OnBeforeReponse() section. Configure HTTrack/WGET/CRON to use proxy 127.0.0.01:8888 */
static function OnBeforeResponse(oSession: Session) {
if (oSession.HostnameIs("TARGETHOSTNAME_FILTER.com") && oSession.responseCode == 403) {
oSession.responseCode = 200;
oSession.oResponse.headers.HTTPResponseCode = 200;
oSession.oResponse.headers.HTTPResponseStatus = "200 OK";
}
}
Do web browsers send the file size in the http header when uploading a file to the server? And if that is the case, then, is it possible to refuse the file just by reading the header and not wait for the whole upload process to finish?
http://www.faqs.org/rfcs/rfc1867.html
HTTP clients are
encouraged to supply content-length for overall file input so that a
busy server could detect if the proposed file data is too large to be
processed reasonably
But the content-length is not required, so you cannot rely on it. Also, an attacker can forge a wrong content-length.
To read the file content is the only reliable way. Having said that, if the content-lenght is present and is too big, to close the connection would be a reasonable thing to do.
Also, the content is sent as multipart, so most of the modern frameworks decode it first. That means you won't get the file byte stream until the framework is done, which could mean "until the whole file is uploaded".
EDIT : before going too far, you may want to check this other answer relying on apache configuration : Using jQuery, Restricting File Size Before Uploading . the description below is only useful if you really need even more custom feedback.
Yes, you can get some information upfront, before allowing the upload of the whole file.
Here's an example of header coming from a form with the enctype="multipart/form-data" attribute :
POST / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.7,fr-be;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: multipart/form-data; boundary=---------------------------886261531333586100294758961
Content-Length: 135361
-----------------------------886261531333586100294758961
Content-Disposition: form-data; name=""; filename="IMG_1132.jpg"
Content-Type: image/jpeg
(data starts here and ends with -----------------------------886261531333586100294758961 )
You have the Content-Length in the header, and additionally there is the Content-Type in the header of the file part ( each file has its own header, which is the purpose of multipart encoding ). Beware that it's the browser responsibility to set a relevant Content-Type by guessing the file type ; you can't guarantee it, but it should be fairly reliable for early rejection ( yet you'd better check the whole file when it's entirely available ).
Now, there is a gotcha. I used to filter image files like that, not on the size, but on the content-type ; but as you want to stop the request as soon as possible, the same problem arises : the browser only gets your response once the whole request is sent, including form content and thus uploaded files.
If you don't want the provided content and stop the upload, you have no choice but to brutally close the socket. The user will only see a confusing "connection reset by peer" message. And that sucks, but it's by design.
So you only want to use this method in cases of background asynchronous checks ( using a timer that checks the file field ). So I had that hack :
I use jquery to tell me if the file field has changed
When a new file is chosen, disable all other file fields on the same form to get only that one.
Send the file asynchronously ( jQuery can do it for you, it uses a hidden frame )
Server-side, check the header ( content-length, content-type, ... ), cut the connection as soon as you got what you need.
Set a session variable telling if that file was OK or not.
Client-side, as the file is uploaded to a frame you don't even get any kind of feedback if the connection is closed. Your only alternative is a timer.
Client-side, a timer polls the server to get a status for the uploaded file. Server side, you have that session variable set, send it back to the brower.
The client has the status code ; render it to your form : error message, green checkmark/red X, whatever. Reset the file field or disable the form, you decide. Don't forget to re-enable other file fields.
Quite messy, eh ? If any of you has a better alternative, I'm all ears.
I'm not sure, but you should not really trust anything sent in the header, as it could be faked by the user.
It depends on how the server works. For example in PHP your script will not run until the file upload is complete, so this wouldn't be possible.