Nginx return statement not accepting "text" - nginx

Following config is working for me:
server {
listen 80;
root /app/web;
index index.json;
location / {
return 409;
}
}
If I hit the website the 409 page will be presented. However following is not working:
server {
listen 80;
root /app/web;
index index.json;
location / {
return 409 "foobar";
}
}
The page is unreachable. But according to the docs http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return
return 409 "foobar";
should work. Any ideas whats wrong? There are no logs in nginx/error.log.

The thing is, Nginx does exactly what you ask it to do. You can verify this by calling curl -v http://localhost (or whatever hostname you use). The result will look somewhat like this:
* Rebuilt URL to: http://localhost/
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost
> Accept: */*
>
< HTTP/1.1 409 Conflict
* Server nginx/1.4.6 (Ubuntu) is not blacklisted
< Server: nginx/1.4.6 (Ubuntu)
< Date: Fri, 08 May 2015 19:43:12 GMT
< Content-Type: application/octet-stream
< Content-Length: 6
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
foobar
As you can see, Nginx returns both 409 and foobar, as you ordered.
So the real question here is why your browser shows the pretty formatted error page when there is no custom text after the return code, and the gray "unreachable" one, when such text is present.
And the answer is: because of the Content-Type header value.
The HTTP standard states that some response codes should or must come with the response body. To comply with the standard, Nginx does this: whenever you return a special response code without the required body, the web server sends its own hardcoded HTML response to the client. And a part of this response is the header Content-Type: text/html. This is why you see that pretty white error page, when you do return 409 without the text part — because of this header your browser knows that the returned data is HTML and it renders it as HTML.
On the other hand, when you do specify the text part, there is no need for Nginx to send its own version of the body. So it just sends back to the client your text, the response code and the value of Content-Type that matches the requested file (see /etc/nginx/mime.types).
When there is no file, like when you request a folder or a site root, the default MIME type is used instead. And this MIME type is application/octet-stream, which defines some binary data. Since most browsers have no idea how to render random binary data, they do the best they can, that is, they show their own hardcoded error pages.
And this is why you get what you get.
Now if you want to make your browser to show your foobar, you need to send a suitable Content-Type. Something like text/plain or text/html. Usually, this can be done with add_header, but not in your case, for this directive works only with a limited list of response codes (200, 201, 204, 206, 301, 302, 303, 304, or 307).
The only other option I see is to rewrite your original request to something familiar to Nginx, so that it could use a value from /etc/nginx/mime.types for Content-Type:
server {
listen 80;
root /app/web;
index index.json;
location / {
rewrite ^.*$ /index.html;
return 409 "foobar";
}
}
This might seem somewhat counter-intuitive but this will work.
EDIT:
It appears that the Content-Type can be set with the default_type directive. So you can (and should) use default_type text/plain; instead of the rewrite line.

Updating #ivan-tsirulev 's answer:
By now you can set headers even for page with status codes for errors using always.
location #custom_error_page {
return 409 "foobar";
add_header Content-Type text/plain always;
}
But if you set default_type, the response headers will have two Content-Type headers: default, then added. Nevertheless, it works fine.

Related

Specific location path is not refreshing

We have a OpenResty, Nginx and Redis stack. I have a location block in my nginx.conf which doesn't reponse to any changes I do.
I have a Lua module function that is invoked by location in nginx.conf. The function is like:
function _M.myFunction()
ngx.header["Content-Type"] = "text/html"
ngx.say("debug")
end
I have a location like this - this location was set to another function that returned text/plain Content-Type with an empty string (ngx.say(" ") and I now pointed it to my debug function, it seems that it is "stuck" on previous response:
location /.faulty/foo/{
content_by_lua_block {
myLuaModule:myFunction()
}
}
responding with:
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive
with no body in response.
Testing by adding a different locations seems to working great for the same function and getting me the response I want:
location /.test/foo/{
content_by_lua_block {
myLuaModule:myFunction()
}
}
responding with:
HTTP/1.1 200 OK
Content-Type: text/html
Connection: keep-alive
debug
I checked Redis for any key that might hold this location, and couldn't find anything relevant.
so, for future generations -
there was a LB server in front of that server with the exact same function and location.
once changed there the change was refreshed.

When can you have relative URLs on the request line of GET http requests?

I tried the following request:
GET index.htm HTTP/1.1
connection: close
host: example.com
content-length: 0
But earned a 400 Bad Request. Why? The server should be able to piece together the absolute URL: http://example.com/index.htm . Why does it complain? Do I need a referer header to use relative URLs on the request line?
Short answer
You can't use relative path because HTTP specification (RFC7230) requires the use of absolute path.
Explanation
If you just refer to HTTP specification, it easy to find out why your request got a 400 Bad Request response: it violates the specification.
RFC7230 defines that in your scenario, the request target must use what is called the origin-form that requires absolute path:
origin-form = absolute-path [ "?" query ]
For instance, the HTTP request for http://example.org/where?q=now would be:
GET /where?q=now HTTP/1.1
Host: example.org
If the path is empty, such as http://example.org the HTTP request would be:
GET / HTTP/1.1
Host: example.org
This is because the absolute path is required as explained in Section 5.3.1 as follows (emphasis mine):
When making a request directly to an origin server, other than a
CONNECT or server-wide OPTIONS request (as detailed below), a client
MUST send only the absolute path and query components of the target
URI as the request-target.
I think on this line GET index.htm HTTP/1.1 is missing an 'l' on index.html.
Hope that helps!
I found the answer: If the URL on the request line isn't absolute it must be an absolute path. This means that you can omit the protocol and host name, but never any part of the path. The following worked:
GET /index.htm HTTP/1.1
connection: close
host: example.com
content-length: 0

How can I customize HTTP 400 responses for parse errors?

I've written a REST API service that requires that all responses be JSON. However, when the Go HTTP request parser encounters an error, it returns 400 as a plain text response without ever calling my handlers. Example:
> curl -i -H 'Authorization: Basic hi
there' 'http://localhost:8080/test' -v
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET /test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.54.0
> Accept: */*
> Authorization: Basic hi
> there
>
< HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
< Connection: close
Connection: close
<
* Closing connection 0
Note the invalid Authorization header. Of course 400 is the proper response, but it's text/plain, of course. Is there some way to configure the Go http parser to use custom error response media types and bodies?
You can't. You can find this in net/http source, it only happens if the request was malformed:
https://github.com/golang/go/blob/master/src/net/http/server.go#L1744
I think your problem might be a new line in the header you're adding in curl?
401, 403, 404, 500 errors you'll be able to respond with json, but bad requests or bad headers (too long, malformed) are handled within server.go.
There is at present no way to intercept such errors though it is under consideration, so your only solution in go would be to patch the stdlib source (I don't recommend this). However, since this error only presents if the client has made a mistake and the request is malformed, it's probably not a huge problem. The reason for the text response is so that a browser or similar client (like curl without -v) doesn't just see an empty response. You could put a proxy like nginx in front of your app but then you'd never see the request either as it is a bad request, your proxy would handle it.
Possibly you'd be able to do it with a proxy like nginx in front though if you set a specific static error page for it to serve for 400 errors and serve a 400.json file that you specify? That's the only solution I can think of. A directive something like this might work for nginx:
error_page 400 /400.json;
If you'd like to be able to customise these errors, perhaps add a comment to the issue linked to let them know you had this specific problem.
If you are using the standard net/http library you can use the following code. Take a look at this answer Showing custom 404 error page with standard http package #Mostafa to which I got this example from
func homeHandler(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
errorHandler(w, r, http.StatusNotFound)
return
}
fmt.Fprint(w, "welcome home")
}
func errorHandler(w http.ResponseWriter, r *http.Request, status int) {
w.WriteHeader(status)
if status == http.StatusNotFound {
// JSON Out here
}
}

Access-Control-Allow-Origin server mods are not working

I am having a terrible time trying to get my server to accept requests from another server (local, but given a domain name in my hosts file) without triggering the dreaded
XMLHttpRequest cannot load https://dev.mydomain.org/api/user?uid=1. Origin http://home.domain.org is not allowed by Access-Control-Allow-Origin.
my dev server (internet) is running nginx, my home server(local) is running apache.
I have tried several solutions found on the internet, to no avail. I have tried modifying the headers in the nginx configs to allow my home.mydomain.org server, I have also added htaccess rules locally to allow all origins (*).
My nginx server block has these lines currently:
add_header Access-Control-Allow-Origin http://home.mydomain.org;
add_header Access-Control-Allow-Headers Authorization;
Adding just the first one did change my response slightly (from simple Origin not allowed by Access-Control-Allow-Origin to Request header field Authorization is not allowed by Access-Control-Allow-Headers.) but adding the second line just reverted the error to the original one and I am still blocked.
At this point, I am not sure what else to try.
UPDATES:
Launching Chrome with flag --disable-web-security allows me to test, and my site and code is working fine in Chrome.
However, this revealed another strange problem, which is that if I try adding the add_header lines to a location directive, both my no-web-security Chrome and my unmodified Safari fail to load info from my api. So now I am not sure if my add_header directives in the server block are working correctly at all.
If it helps any, here is my client code (including things I have tried/commented out):
var xhr = new XMLHttpRequest();
var self = this;
xhr.open('GET', apiURL + self.currentIssue);
xhr.setRequestHeader('Access-Control-Allow-Origin','http://home.mydomain.org');
//xhr.setRequestHeader('Access-Control-Allow-Credentials', 'true');
xhr.withCredentials = true;
//xhr.setRequestHeader('Access-Control-Request-Method','*');
xhr.setRequestHeader('Authorization','Bearer longstringoflettersandnumbers');
xhr.onload = function () {
self.posts = JSON.parse(xhr.responseText);
};
xhr.send();
ANOTHER UPDATE AFTER TRYING SUGGESTION BELOW:
After a bunch of trial and error on both client and server, I still am stuck. Here is my latest response from the server using curl (although I have toggled on and off various options client and server for things like Credentials and changing origin to exactly mine or * to no avail):
HTTP/1.1 204 No Content
Server: nginx
Date: Sun, 06 Aug 2017 10:11:57 GMT
Connection: keep-alive
Access-Control-Allow-Origin: http://home.mydomain.org
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range
Access-Control-Max-Age: 1728000
Content-Type: text/plain; charset=utf-8
Content-Length: 0
and here are my console errors (Safari):
[Error] Origin http://home.mydomain.org is not allowed by Access-Control-Allow-Origin.
[Error] Failed to load resource: Origin http://home.mydomain.org is not allowed by Access-Control-Allow-Origin. (actions, line 0)
[Error] XMLHttpRequest cannot load https://dev.mydomain.org/api/user?uid=1 due to access control checks.
And here is my console error for Firefox:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://dev.mydomain.org/api/user?uid=1. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
Also in Firefox, here are the results from the network panel for OPTIONS and GET:
Request URL: https://dev.mydomain.org/api/user?uid=1
Request method: OPTIONS
Status code: 204 No Content
Version: HTTP/2.0
Response headers (511 B)
Server "nginx"
Date "Sun, 06 Aug 2017 10:44:22 GMT"
Access-Control-Allow-Origin "http://home.mydomain.org"
access-control-allow-credentials "true"
Access-Control-Allow-Methods "GET, POST, OPTIONS"
Access-Control-Allow-Headers "Authorization,DNT,X-CustomHea…ent-Type,Content-Range,Range"
Access-Control-Max-Age "1728000"
Content-Type "text/plain; charset=utf-8"
Content-Length "0"
X-Firefox-Spdy "h2"
Request headers (501 B)
Host "dev.mydomain.org"
User-Agent "Mozilla/5.0 (Macintosh; Intel… Gecko/20100101 Firefox/54.0"
Accept "text/html,application/xhtml+x…lication/xml;q=0.9,*/*;q=0.8"
Accept-Language "en-US,en;q=0.5"
Accept-Encoding "gzip, deflate, br"
Access-Control-Request-Method "GET"
Access-Control-Request-Headers "authorization"
Origin "http://home.mydomain.org"
Connection "keep-alive"
Cache-Control "max-age=0"
Request URL: https://dev.mydomain.org/api/user?uid=1
Request method: GET
Status code: 404 Not Found
Version: HTTP/2.0
Response headers (170 B)
Server "nginx"
Date "Sun, 06 Aug 2017 10:44:22 GMT"
Content-Type "text/html"
Vary "Accept-Encoding"
Content-Encoding "gzip"
X-Firefox-Spdy "h2"
Request headers (723 B)
Host "dev.mydomain.org"
User-Agent "Mozilla/5.0 (Macintosh; Intel… Gecko/20100101 Firefox/54.0"
Accept "*/*"
Accept-Language "en-US,en;q=0.5"
Accept-Encoding "gzip, deflate, br"
Referer "http://home.mydomain.org/"
Authorization "Bearer eyJ0eXAG…BRHmX9VmtYHQOvH7k-Y32wwyeCdk"
Origin "http://home.mydomain.org"
Connection "keep-alive"
Cache-Control "max-age=0"
UPDATE WITH PARTIAL SUCESS:
I think I found the problem (partially): changing my location directive in nginx from location /api to location = /api/* gets it working! But only for Safari and Chrome, FF is now not even trying the GET request, there is NO entry for it in network panel.
UPDATE WITH CRYING AND GNASHING OF TEETH AND PULLING OF HAIR
Safari and Chrome intermittently fail with original error about Origin not allowed, even though they were working fine and no changes have been made to server config. I will be drinking heavily tonight...
Wow, was that ever convoluted. Posting answer here in case some other WP user finds their way here. I kept getting inconsistent results (sometimes working, sometimes not mysteriously) and finally tracked down my problem to headers being set in the PHP code on the server, independently of the nginx settings and sometimes contradicting them (although never in a predictable way that I could see). So the things I needed to resolve were:
Removed all my cors declarations in my nginx configs
I also have code on my server that validates a token in the auth header, and it was failing on OPTIONS preflight (which it should never check) so I had to add an if statement before to have it ignore an OPTIONS call (!$_SERVER['REQUEST_METHOD'] === "OPTIONS")
Since I had cloned this site from another of mine using UpdraftPlus plugin, I had to go in to delete my migrate keys since their existence prevented api calls from working too. Once they were deleted my calls started working again.
Removed and re-added the built in WP filter rest_pre_serve_request
My filter code is here:
add_action('rest_api_init', function() {
/* unhook default function */
remove_filter('rest_pre_serve_request', 'rest_send_cors_headers');
/* then add your own filter */
add_filter('rest_pre_serve_request', function( $value ) {
$origin = get_http_origin();
$my_sites = array( $origin ); // add array of accepted sites if you prefer
if ( in_array( $origin, $my_sites ) ) {
header( 'Access-Control-Allow-Origin: ' . esc_url_raw( $origin ) );
} else {
header( 'Access-Control-Allow-Origin: ' . esc_url_raw( site_url() ) );
}
header( 'Access-Control-Allow-Methods: OPTIONS, GET, POST, PUT, PATCH, DELETE' );
header( 'Access-Control-Allow-Credentials: true' );
header('Access-Control-Allow-Headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Origin,Content-Type,X-Auth-Token,Content-Range,Range');
header('Access-Control-Expose-Headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Origin,Content-Type,X-Auth-Token,Content-Range,Range');
header( 'Vary: Origin' );
return $value;
});
}, 15);
Now finally, everything works everywhere (in every browser and in curl too)!

External links URL encoding leads to '%3F' and '%3D' on Nginx server

I got a problem with my server. I got four inbound links to different sites of my dynamic webpage which look something like this:
myurl.com/default/Site%3Fid%3D13
They should look like this:
myurl.com/default/Site?id=13
I do know that those %3F is an escape sequence for the ? sign and the %3D is an escape sequence for the equal sign. But I do get an error 400 when I use those links. What can I do about that?
The four links are for different sites, and I imagine over time there will be more links like that. So one fix for all would be perfect.
An exact same question was actually asked on nginx-ru mailing list about a year ago:
http://mailman.nginx.org/pipermail/nginx-ru/2013-February/050200.html
The most helpful response, by an Nginx, Inc, employee/developer, Валентин Бартенев:
http://mailman.nginx.org/pipermail/nginx-ru/2013-February/050209.html
Если запрос приходит в таком виде, то это уже не параметры, а имя запрошенного
файла. Другое дело, что location ищется по уже раскодированному адресу, о чем в
документации написано.
Translation:
If the request comes in such a form, then these are no longer the args, but the name of the requested file. Another thing is that, as documented, the location matching is performed against a normalised URI.
His suggested solution, translated to the sample example from the question here at SO, would then be:
location /default/Site? {
rewrite \?(.*)$ /default/Site?$1? last;
}
location = /default/Site {
[...]
}
The following sample would redirect all wrongly-looking requests (defined as having ? in the requested filename — encoded as %3F in the request) into less wrongly-looking ones, regardless of URL.
(Please note that, as rightly advised elsewhere, you should not be getting these wrongly-formed links in the first place, so, use it as a last resort — only when you cannot correct the wrongly formed links otherwise, and you do know that such requests are attempted by valid agents.)
server {
listen [::]:80;
server_name localhost;
rewrite ^/([^?]*)\?(.*)$ /$1?$2? permanent;
location / {
return 200 "id is $arg_id\n";
}
}
This is example of how it would work — when a wrongly looking request is encountered, a correction attempt is made with a 301 Moved Permanently response with a supposedly correct Location response header, which would make the browser automatically re-issue the request to the newly provided location:
opti# curl -6v "http://localhost/default/Site%3Fid%3D13"
* About to connect() to localhost port 80 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 80 (#0)
> GET /default/Site%3Fid%3D13 HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.4.1
< Date: Wed, 15 Jan 2014 17:09:25 GMT
< Content-Type: text/html
< Content-Length: 184
< Location: http://localhost/default/Site?id=13
< Connection: keep-alive
<
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.4.1</center>
</body>
</html>
* Connection #0 to host localhost left intact
* Closing connection #0
Note that no correction attempts are made on proper-looking requests:
opti# curl -6v "http://localhost/default/Site?id=13"
* About to connect() to localhost port 80 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 80 (#0)
> GET /default/Site?id=13 HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.4.1
< Date: Wed, 15 Jan 2014 17:09:30 GMT
< Content-Type: application/octet-stream
< Content-Length: 9
< Connection: keep-alive
<
id is 13
* Connection #0 to host localhost left intact
* Closing connection #0
The URL is perfectly valid. The escaped characters it contains are just that, escaped. Which is perfectly fine.
The purpose is that you can actually have a request name (in most cases corresponding to the filename on the disk) that is Site?id=13 and not Site and the rest as the query string.
I would consider it bad practice to have characters in a filename that makes this necessary. However, in URL arguments it may very well be necessary.
Nevertheless, the request URL is valid, and probably not what you want it to be. Which consequently suggest that you should correct the error wherever anybody has picked up the wrong URL in the first place.
I do not really understand why you get an error 400; you should rather get an error 404. But that depends on your setup.
There are also cases, especially with nginx, that mostly involve passing on whole URLs and URL parts along multiple levels (for example reverse proxies, matching regular expressions from the URL and using them as variables, etc.) where such an error may occur. But to verify this and fix it we would need to know more about your setup.

Resources