In my Google App Engine app (Standard Environment, written in Java + Scala) I want some of my requests to the server to be gzipped. After a bit of experimenting I got it mostly working, but there are few points which I am uncertain about. I did not find much documentation about correct client-side gzip usage, most documentation and examples seem to be concerned about server encoding its responses, therefore I am unsure if I am doing everything as I should.
I send the request this way (using akka.http in the client application):
val uploadReq = Http().singleRequest(
HttpRequest(
uri = "https://xxx.appspot.com/upload-a-file",
method = HttpMethods.POST,
headers = List(headers.`Content-Encoding`(HttpEncodings.gzip))
entity = HttpEntity(ContentTypes.`text/plain(UTF-8)`, Gzip.encode(ByteString(bytes)))
)
)
On a production GAE server, I get the gzipped request body already decoded, with the encoding header still present. On a development server this is different, the header is also present, but the request body is still gzipped.
The code for decoding the request input stream is not a problem, but I did not find a clean way how to check in my server code if I should decode the request body or not. My current workaround is that if the client knows it is communicating with the development server, it does not use gzip encoding at all, and I never attempt to decode the request body, as I rely upon the Google App Engine to do this for me.
should I encode the request body differently on the client?
is there some other way to recognize on the server if the incoming request body needs decoding or not?
may I assume Google App Engine production servers will decode the body for me?
For the record: the solution I have ended up with is that I check the request body and if it looks like gzipped, I unzip it, ignoring the header completely. This works both on prod (where App Engine does the unzipping and the code does no harm) and dev (where the code unzips). Scala code follows:
def decompressStream(input: InputStream): InputStream = {
val pushbackInputStream = new PushbackInputStream(input, 2)
val signature = new Array[Byte](2)
pushbackInputStream.read(signature)
pushbackInputStream.unread(signature)
if (signature(0) == 0x1f.toByte && signature(1) == 0x8b.toByte) {
new GZIPInputStream(pushbackInputStream)
} else pushbackInputStream
}
The theoretical drawback is someone could send a request which contains 0x1f/0x8b header just by chance. This cannot happen in my case, therefore I am fine with it.
Related
I'm working on a wasm rust app with Yew, and trying to retrieve database entries from Notion to fill the app's body content. I'm using reqwest to do all the http requesting part, but no matter what I do it keeps returning a 401 status code. I've tried the same reqwest both on postman and curl and it worked, so the bearer auth token and url should be correct.
Here's what the code looks like:
let client = reqwest::Client::new();
let request = client
.get(NOTION_URL)
.bearer_auth(NOTION_TOKEN)
.header("Notion-Version", NOTION_VERSION)
.fetch_mode_no_cors()
.build()
.unwrap();
let response = client.execute(request).await.unwrap();
I've tried logging NOTION_URL, NOTION_TOKEN, NOTION_VERSION env variables to be sure that they are correct, and they are.
I've tried different things, like adding additional headers like accept or user-agent but the result is the same. Also tried to set the client's default headers with both the notion version and auth headers without success.
My guess is that maybe the fetch_mode_no_cors() is the culprit, but without it the request doesn't even go through, and I can't find any alternative to this method.
I'm kinda lost tbh, don't know where to find a solution so here we go!
Thanks!
I'm trying to download a firmware.bin file that is produced in a private Github repository. I have the code that is finding the right asset url to download the file and per Github instructions the accept header needs to be set to accept: application/octet-stream in order to get the binary file. I'm only getting JSON in response. If I run the same request through postman I'm getting a binary file as the body. I've tried downloading it using HTTPClient and I get the same JSON request. It seems the headers aren't being set as requested to tell Github to send the binary content as I'm just getting JSON. As for the ArduinoOTA abstraction, I can't see how to even try to set headers and in digging into the esp_https_ota functions and http_client functions there doesn't appear to be a way to set headers for any of these higher level abstractions because the http_config object has no place for headers as far as I can tell. I might file a feature request to allow for this, but am new to this programming area and want to check to see if I'm missing something first.
Code returns JSON, not binary. URL is github rest api url to the asset (works in postman)
HTTPClient http2;
http2.setAuthorization(githubname,githubpass);
http2.addHeader("Authorization","token MYTOKEN");
http2.addHeader("accept","application/octet-stream");
http2.begin( firmwareURL, GHAPI_CERT); //Specify the URL and certificate
With the ESP IDF HTTP client you can add headers to an initialized HTTP client using function esp_http_client_set_header().
esp_http_client_handle_t client = esp_http_client_init(&config);
esp_http_client_set_header(client, "HeaderKey", "HeaderValue");
err = esp_http_client_perform(client);
If using the HTTPS OTA API, you can register for a callback which gives you a handle to the underlying HTTP client. You can then do the exact same as in above example.
I have a local web app with an HTTP POST endpoint.
This endpoint receives emails sent from SendGrid.
I have saved the body of a previous email with attachments which I know works, as requestBody.bin.
I'm trying to replicate this HTTP POST request using JMeter.
I have this Beanshell Preprocessor script:
FileInputStream in = new FileInputStream("C:\\Projects\\centaurjmeter\\src\\InboundEmails\\requestBody.bin");
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buffer = new byte[999024]; //Value of Content-Length from when I sent the actual email. So this should be more than enough just for the HTTP POST body.
for (int i; (i = in.read(buffer)) != -1; ) {
bos.write(buffer, 0, i);
}
in.close();
byte[] requestBody = bos.toByteArray();
bos.close();
vars.put("requestBody", new String(requestBody, "ISO-8859-1"));
and I'm using ${requestBody} in the body of the HTTP POST request:
This is what my JMeter test looks like:
In my HTTP POST action method, I'm saving the attachments of the email to file.
The pdfs seem to be slightly corrupted, only part of the text on each page is showing.
The .png files do not even open.
I've tried using:
vars.put("requestBody", new String(requestBody));
vars.put("requestBody", new String(requestBody, "Windows-1252")); //ANSI
vars.put("requestBody", new String(requestBody, "ISO-8859-1")); //default encoding
But none of them work. "ISO-8859-1" results in the least amount of corruption, in that at least some text appears in the pdfs (but the .png files don't work at all).
I can't use a HTTP Raw Request Sampler because that doesn't work with https.
How can I get the bytes from requestBody.bin and send them in my HTTP POST body correctly?
UPDATE
I read https://stackoverflow.com/a/41892324/2063755 and tried sending requestBody.bin as a file with the request in the "Files Upload" tab, but I get the same result as using the Beanshell Preprocessor script.
UPDATE
The above link actually did help me solve the problem.
I just had to add this extra TE header:
You need to ensure that the same encoding is being applied everywhere, I would recommend sticking to UTF-8
Run JMeter providing file.encoding=UTF-8 property, it can be added to system.properties file (lives in "bin" folder of your JMeter installation)
Since JMeter 3.1 you should be using JSR223 Test Elements and Groovy language for scripting so you can use the following __groovy() function directly in the "Body Data" tab of your HTTP Request sampler:
${__groovy(new File('C:\\Projects\\centaurjmeter\\src\\InboundEmails\\requestBody.bin').getText('UTF-8'),)}
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
Assuming everything goes well your "local web app" should see the identical file, it can be double-checked using a 3rd-party sniffer tool like Wireshark. Make sure to use UTF-8 for parsing the file in your app as well.
What is the best way to implement the following scenario in a React Native app?
Make an HTTP request to the server, get a JSON response and an ETag header.
Save this JSON response in a way that will persist even after the app is restarted by the user.
Whenever this HTTP request is repeated, send an If-None-Match header.
When you get a "Not Modified" response, use the version in the persisted cache.
When you get a "Successful" response (meaning the response has changed), invalidate the persisted cache, save the new response.
Does React Native have a component that does these things out of the box? If not, what is the most common way people use to handle this?
The fetch() API of React native is following the http caching spec and it provides this feature. When you hit a 304 a 200 old response will be found in the cache and be reused.
Details:
https://github.com/heroku/react-refetch/issues/142
As answered at: https://stackoverflow.com/a/51905151
React Native’s fetch API bridges to NSURLSession on iOS and okhttp3 on Android. Both of these libraries strictly follow the HTTP caching spec. The caching behavior will depend primarily on the Cache-Control and Expires headers in the HTTP response. Each of these libraries have their own configuration you can adjust, for example to control the cache size or to disable caching.
And this: How to use NSURLSession to determine if resource has changed?
The caching provided by NSURLSession via NSURLCache is transparent, meaning when you request a previously cached resource NSURLSession will call the completion handlers/delegates as if a 200 response occurred.
If the cached response has expired then NSURLSession will send a new request to the origin server, but will include the If-Modified-Since and If-None-Match headers using the Last-Modified and Etag entity headers in the cached (though expired) result; this behavior is built in, you don't have to do anything besides enable caching. If the origin server returns a 304 (Not Modified), then NSURLSession will transform this to a 200 response the application (making it look like you fetched a new copy of the resource, even though it was still served from the cache).
Oof. Been over a year. I assume you know this is a resounding "no," right? You'll have to parse the response headers to grab the ETag and store that on the device (you're not using the browser) and then add the header to the subsequent requests after retrieving it from your storage mechanism of choice.
I just found this because I was looking to see if anybody had done this in React, let alone React Native, and I'm not seeing anything.
Whelp, time to roll up my sleeves and invent this thing...
okay, this is my current solution, not production tested yet. would love your feed back googlers.
i use Axios, but if you dont you still implement this around what ever wrapper you have around fetch -unless u use native fetch !-
import api from 'your api wrapper.js'
api.etags = new Set;
api.cache = new Set;
api.addRequestTransform(request => {
const etag = api.etags.get(request.url);
if (etag) {
request.headers['HTTP_IF_NONE_MATCH'] = etag;
}
})
// or whatever you use to wrap ur HTT
api.addResponseTransform(response =>{
if (
response.status === 304 &&
response.headers &&
response.headers.etag &&
api.cache.has(response.headers.etag)
) {
console.log('%cOVERRIDING 304', 'color:red;font-size:22px;');
response.status = 200;
response.data = api.cache.get(response.headers.etag);
} else if (response.ok && response.headers && response.headers.etag) {
api.cache.set(response.headers.etag, response.data);
api.etags.set(response.config.url, response.headers.etag);
}
});
what we are doing here is saving response result into api.cache, and saving the etags into api.etag, then we send etag with request every time.
we can upgrade this to also remember the correct status code, or save etags to disk, duno. what do you think :) ?
Hello everybody I'm trying to do a simple HTTP call to a Tomcat Server running on my server from my Android App. The server will then execute a certain command to my website. I created a button that when I click it runs the HTTP call from the App.
If I use the approach below, it opens the browser on my phone to run this HTTP. Is it possible to do something similar but not have my app open the browser???
Intent browserIntent = new Intent(Intent.ACTION_VIEW, Uri.parse("http://" + IP + ":8080/server/run.jsp"));
startActivity(browserIntent);
thank you so much in advance :D
Of course it starts your browser. Your code is explicitly asking Android to launch an app that can "view" the URL.
If you want your app to access the URL directly, use HttpURLConnection instead:
1.Obtain a new HttpURLConnection by calling URL.openConnection() and casting the result to HttpURLConnection.
2.Prepare the request. The primary property of a request is its URI. Request headers may also include metadata such as credentials, preferred content types, and session cookies.
3.Optionally upload a request body. Instances must be configured with setDoOutput(true) if they include a request body. Transmit data by writing to the stream returned by getOutputStream().
4.Read the response. Response headers typically include metadata such as the response body's content type and length, modified dates and session cookies. The response body may be read from the stream returned by getInputStream(). If the response has no body, that method returns an empty stream.
5.Disconnect. Once the response body has been read, the HttpURLConnection should be closed by calling disconnect(). Disconnecting releases the resources held by a connection so they may be closed or reused.