I'm trying to read the Content-Encoding in a header_filter_by_lua block. I test using chrome's developer tools while requesting an url which respond with Content-Encoding: gzip. I use these checks:
local test1 = ngx.var.http_content_encoding
local test2 = ngx.header.content_encoding
local test3 = ngx.resp.get_headers()["Content-Encoding"]
and all of them give empty/nil value. Getting User-Agent in same way is successful so what's the problem with Content-Encoding?
ngx.var.http_content_encoding - would return request's (not response's) header
API below work for read access in context of header_filter_by_lua_block and later phases:
ngx.header.content_encoding works for me always and is the right way.
If it doesn't work - check https://github.com/openresty/lua-nginx-module#lua_transform_underscores_in_response_headers
ngx.resp.get_headers()["Content-Encoding"] also work, but not efficient to obtain single header.
To get the value from request use following
ngx.req.get_headers()["content_encoding"]
Related
I'm trying to download a firmware.bin file that is produced in a private Github repository. I have the code that is finding the right asset url to download the file and per Github instructions the accept header needs to be set to accept: application/octet-stream in order to get the binary file. I'm only getting JSON in response. If I run the same request through postman I'm getting a binary file as the body. I've tried downloading it using HTTPClient and I get the same JSON request. It seems the headers aren't being set as requested to tell Github to send the binary content as I'm just getting JSON. As for the ArduinoOTA abstraction, I can't see how to even try to set headers and in digging into the esp_https_ota functions and http_client functions there doesn't appear to be a way to set headers for any of these higher level abstractions because the http_config object has no place for headers as far as I can tell. I might file a feature request to allow for this, but am new to this programming area and want to check to see if I'm missing something first.
Code returns JSON, not binary. URL is github rest api url to the asset (works in postman)
HTTPClient http2;
http2.setAuthorization(githubname,githubpass);
http2.addHeader("Authorization","token MYTOKEN");
http2.addHeader("accept","application/octet-stream");
http2.begin( firmwareURL, GHAPI_CERT); //Specify the URL and certificate
With the ESP IDF HTTP client you can add headers to an initialized HTTP client using function esp_http_client_set_header().
esp_http_client_handle_t client = esp_http_client_init(&config);
esp_http_client_set_header(client, "HeaderKey", "HeaderValue");
err = esp_http_client_perform(client);
If using the HTTPS OTA API, you can register for a callback which gives you a handle to the underlying HTTP client. You can then do the exact same as in above example.
In my Google App Engine app (Standard Environment, written in Java + Scala) I want some of my requests to the server to be gzipped. After a bit of experimenting I got it mostly working, but there are few points which I am uncertain about. I did not find much documentation about correct client-side gzip usage, most documentation and examples seem to be concerned about server encoding its responses, therefore I am unsure if I am doing everything as I should.
I send the request this way (using akka.http in the client application):
val uploadReq = Http().singleRequest(
HttpRequest(
uri = "https://xxx.appspot.com/upload-a-file",
method = HttpMethods.POST,
headers = List(headers.`Content-Encoding`(HttpEncodings.gzip))
entity = HttpEntity(ContentTypes.`text/plain(UTF-8)`, Gzip.encode(ByteString(bytes)))
)
)
On a production GAE server, I get the gzipped request body already decoded, with the encoding header still present. On a development server this is different, the header is also present, but the request body is still gzipped.
The code for decoding the request input stream is not a problem, but I did not find a clean way how to check in my server code if I should decode the request body or not. My current workaround is that if the client knows it is communicating with the development server, it does not use gzip encoding at all, and I never attempt to decode the request body, as I rely upon the Google App Engine to do this for me.
should I encode the request body differently on the client?
is there some other way to recognize on the server if the incoming request body needs decoding or not?
may I assume Google App Engine production servers will decode the body for me?
For the record: the solution I have ended up with is that I check the request body and if it looks like gzipped, I unzip it, ignoring the header completely. This works both on prod (where App Engine does the unzipping and the code does no harm) and dev (where the code unzips). Scala code follows:
def decompressStream(input: InputStream): InputStream = {
val pushbackInputStream = new PushbackInputStream(input, 2)
val signature = new Array[Byte](2)
pushbackInputStream.read(signature)
pushbackInputStream.unread(signature)
if (signature(0) == 0x1f.toByte && signature(1) == 0x8b.toByte) {
new GZIPInputStream(pushbackInputStream)
} else pushbackInputStream
}
The theoretical drawback is someone could send a request which contains 0x1f/0x8b header just by chance. This cannot happen in my case, therefore I am fine with it.
I encountered a very weird error today within phantomjs, that I'm turning to the community for. The expected JSON response of an API I was invoking with phantomJS was not being returned, but rather the result was a status of failure.
onResourceReceived() showed a 200 http status code for the resource in question
onLoadFinished() shows a status of fail
After debugging this for sometime, I noticed that the site was returning a non standard content-type header on the response. Rather than content-type of "application/json" the header being returned was "application/servicename-1.0+json".
To verify this, we spun up a local webserver that served a similar header, and sure enough phantom js cannot load the page. Setting the response header to "application/json" and phantomjs correctly renders the page and sets the page objects page.plainText variable. I've included the testing script below.
Has anyone encountered anything like this before?
Any suggestions on how to handle this issue within phantomjs?
The simplest and quickest solution I can think of (not involving editing PhantomJS' source code and compiling it) is to set up a simple local proxy server in front of PhantomJS that would rewrite the incorrect header.
It could be something like fiddler2, Charles Proxy or a simple node.js script, like this quick-n-dirty example:
// npm install proxy-tamper
var proxy = require('proxy-tamper').start({port: 3000});
proxy.tamper(/api.truelocal.com.au.*$/, function (request) {
request.onResponse(function (response) {
if ('content-type' in response.headers && response.headers['content-type'] == 'application/servicename-1.0+json') {
response.headers['content-type'] = 'application/json';
}
response.complete();
});
});
Then have PhantomJS use it:
phantomjs --proxy=127.0.0.1:3000 script.js
Note that this won't work for secure pages.
Well, after asking this question I did a deeper dive in the phantomjs source. Looks like there is some fairly strict detection on application/json here: https://github.com/Vitallium/qtwebkit/blob/phantomjs/Source/WebCore/dom/DOMImplementation.cpp#L368
I would like to set the default http hrader in Tcl http package to empty and then selecetively put some header on my own using a dictionary. I need to do this becasue I see many items (like sock, binary, -binary, -strict, queryoffset etc) in my Tcl http request header which are not present in the header specified by other web browser like firefox. I get correct response in broswer so I want exactly those heater which are send by the brower. For this I need to set the default http header in Tcl http package to empty, and mannually set the headers (which I can do). How do I empty the default headers?
I'm not quite sure which header you've got a problem with, but the majority of headers can be set quite easily via the -headers option to http::geturl:
# Easiest to build the option as a dictionary
dict set opts X-Example-Header "fruitbats are mammals and this is nonsense"
dict set opts DNT "1 (Do Not Track Enabled)"
http::geturl $theurl -headers $opts
Almost everything can be set or overridden this way, and the things that can't are typically related to the management of the network connection itself (such as keep-alive management, chunking, compression) and are probably best left to the library in the first place, as HTTP/1.1 is a pretty complex protocol for something supposedly stateless.
Note that options to http::geturl do not directly translate into request options. It's a higher-level interface…
I need to test some client application code I've written to test its' handling of various status codes returned in an HTTP response from a web server.
I have Fiddler 2 (Web Debugging Proxy) installed and I believe there's a way to modify responses using this application, but I'm struggling to find out how. This would be the most convenient way, as it would allow me to leave both client and server code unmodified.
Can anyone assist as I'd like to intercept the HTTP response being sent from server to client and modify the status code before it reaches the client?
Any advice would be much appreciated.
Ok, so I assume that you're already able to monitor your client/server traffic. What you want to do is set a breakpoint on the response then fiddle with it before sending it on to the client.
Here are a couple of different ways to do that:
Rules > Automatic Breakpoints > After Responses
In the quickexec box (the black box at the bottom) type "bpafter yourpage.svc". Now Fiddler will stop at a breakpoint before all requests to any URL that contains "yourpage.svc". Type "bpafter" with no parameters to clear the breakpoint.
Programmatically tamper with the response using FiddlerScript. The best documentation for FiddlerScript is on the official site: http://www.fiddler2.com/Fiddler/dev/
Once you've got a response stopped at the breakpoint, just double click it to open it in the inspectors. You've got a couple of options now:
Right next to the green Run to Completion button (which you click to send the response) there's a dropdown that lets you choose some default response types.
Or, on the Headers inspector, change the response code & message in the textbox at the top.
Or, click the "Raw" inspector and mess with the raw response to do arbitrary things to it. Also a good way to see what your client does when it gets a malformed response, which you'll probably test accidentally :)
Another alternative is to use Fiddler's AutoResponder tab (on the right-hand panel). This allows you to catch a request to any URI that matches a string and serve a "canned" response from a file. The file can contain both headers and payload. The advantage of this approach is that you don't have to write FiddlerScript and you don't have to handle each request manually via a breakpoint.
You would set the rule up in Fiddler like shown below (ensure you enable unmatched requests passthrough otherwise all other http requests will fail).
In this example, any request whose URI includes "fooBar" will get the canned response. The format of the file will vary depending on your APIs (you can use your browser to intercept a "real" response and base it on that) but mine looked like the following:
HTTP/1.1 409 Conflict
Server: Apache-Coyote/1.1
X-Powered-By: Servlet 2.5; JBoss-5.0/JBossWeb-2.1
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT, PATCH, OPTIONS
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Max-Age: 86400
Content-Type: application/vnd.api+json
Content-Length: 149
Date: Tue, 28 Mar 2017 10:03:29 GMT
{"errors":[{"code":"OutOfStock","detail":"Item not in stock","source":{"lineId":{"type":"Order line Number","id":"1"}},"meta":{"availableStock":0}}]}
I found that it needed a carriage return at the end of the last line (i.e. after the json), and that the Content-Length header had to match the number of characters in the json, otherwise the webapp would hang. Your mileage may vary.
Create a FiddlerScript rule. Here's what I used in order to generate a local copy of a website that was intentionally using 403 on every page to thwart HTTrack/WGET.
https://gist.github.com/JamoCA/22db8d68a9a2fb20cb04a85360185333
/* 20180615 Fiddler rule to ignore all 403 HTTP Status errors so WGET or HTTrack can generate local copy of remote website */
SCENARIO: Changing the user agent or setting a delay isn't enough and the entire remote server is configured to respond w/403.
CONFIGURE: Add below rule to FiddlerScript OnBeforeReponse() section. Configure HTTrack/WGET/CRON to use proxy 127.0.0.01:8888 */
static function OnBeforeResponse(oSession: Session) {
if (oSession.HostnameIs("TARGETHOSTNAME_FILTER.com") && oSession.responseCode == 403) {
oSession.responseCode = 200;
oSession.oResponse.headers.HTTPResponseCode = 200;
oSession.oResponse.headers.HTTPResponseStatus = "200 OK";
}
}