Issue mentioned and followed up at:
https://github.com/r-lib/httr/issues/626
https://community.rstudio.com/t/possible-to-drop-authentication-header-on-redirects-in-httr/44358
This issue happens when:
httr sends a GET call to a URL which requires an "Authorization" header
the API constructs a redirect to a pre-signed URL which is returned as the response to the original GET request, for which no authentication is required
the httr client sends the same "Authorization" header on the redirect URL as it did on the URL of the original GET call
a 401 is returned because the bearer token is not valid for the redirect URL
It seems that "Authorization" headers should not be sent when a redirect is followed.
But this is a common issue even in e.g. browsers (so this might be broader than httr):
Authorization header when following redirects
The following example works (http redirect to https), but requires writing a function separate from httr.
Can httr handle this type of use case directly or is it best practice to write custom wrappers for this ?
Example (adapted from https://github.com/r-lib/httr/issues/626) :
library(httr)
# Ensure curl version is higher than 7.58: https://curl.se/docs/CVE-2018-1000007.html
curl::curl_version()$version
# Call URL without following to redirect
resp <- GET(
"http://github.com",
config(followlocation = FALSE),
authenticate("user", "pwd", type = "basic")
)
# Check response of URL, which contains the redirect within the location field
resp
redirected_url <- headers(resp)[["location"]]
# Call redirect URL without authentication
resp_redirect <- GET(redirected_url)
Console outputs:
> curl::curl_version()$version
[1] "7.60.0"
> resp
Response [http://github.com/]
Date: 2022-08-24 09:11
Status: 301
Content-Type: <unknown>
<EMPTY BODY>
> redirected_url
[1] "https://github.com/"
> resp_redirect
Response [https://github.com/]
Date: 2022-08-24 09:12
Status: 200
Content-Type: text/html; charset=utf-8
Size: 301 kB
We are deploying the flask server using uwsgi, This works fine if we call the service directly but started getting 502 Bad Gateway with no response when calling it through NGINX and getting status code 502.
Looking at NGINX logs we got:
upstream sent invalid chunked response while reading upstream
we already tried this solution but this didn't work NGINX Error: upstream sent invalid chunked response while reading upstream
I did grep " 500 " on the request log and got this, I understand that HTTP code is 200 but what is this 501?
POST /api/v1/url HTTP/1.1" 200 501
I suppose your server is Apache and logs are in CLF (Common Log Format).
Then 501 is the size of the server response (in bytes and in total).
I've written a REST API service that requires that all responses be JSON. However, when the Go HTTP request parser encounters an error, it returns 400 as a plain text response without ever calling my handlers. Example:
> curl -i -H 'Authorization: Basic hi
there' 'http://localhost:8080/test' -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Authorization: Basic hi
> there
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Connection: close
Connection: close
<
* Closing connection 0
Note the invalid Authorization header. Of course 400 is the proper response, but it's text/plain, of course. Is there some way to configure the Go http parser to use custom error response media types and bodies?
You can't. You can find this in net/http source, it only happens if the request was malformed:
https://github.com/golang/go/blob/master/src/net/http/server.go#L1744
I think your problem might be a new line in the header you're adding in curl?
401, 403, 404, 500 errors you'll be able to respond with json, but bad requests or bad headers (too long, malformed) are handled within server.go.
There is at present no way to intercept such errors though it is under consideration, so your only solution in go would be to patch the stdlib source (I don't recommend this). However, since this error only presents if the client has made a mistake and the request is malformed, it's probably not a huge problem. The reason for the text response is so that a browser or similar client (like curl without -v) doesn't just see an empty response. You could put a proxy like nginx in front of your app but then you'd never see the request either as it is a bad request, your proxy would handle it.
Possibly you'd be able to do it with a proxy like nginx in front though if you set a specific static error page for it to serve for 400 errors and serve a 400.json file that you specify? That's the only solution I can think of. A directive something like this might work for nginx:
error_page 400 /400.json;
If you'd like to be able to customise these errors, perhaps add a comment to the issue linked to let them know you had this specific problem.
If you are using the standard net/http library you can use the following code. Take a look at this answer Showing custom 404 error page with standard http package #Mostafa to which I got this example from
func homeHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
errorHandler(w, r, http.StatusNotFound)
return
}
fmt.Fprint(w, "welcome home")
}
func errorHandler(w http.ResponseWriter, r *http.Request, status int) {
w.WriteHeader(status)
if status == http.StatusNotFound {
// JSON Out here
}
}
In Nginx, I want to customize the status code message right after the status code:
such as this is the default:
HTTP/1.1 429
and I want to do this:
HTTP/1.1 429 Too many requests
How can I get it done please? Note this is the header not the body at all.