How to use nginx custom log with logstash properly - nginx

I've found the good article about using of nginx custom log format for logstash. In one on topic comment there is:
Be careful: Between two (referrer, user agent) and four (request method, remote_user) can be set by the end-user and thus can cause the JSON to be invalid (i.e. set the user-agent to "}).
As long as nginx doesn't have explicit JSON support, I would recommend against manually >building JSON and stick to combined log which is well-supported by logstack.
How should I use custom log format to prevent this issue?

Create grok with content
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}
And filter
filter {
grok {
patterns_dir => ["Grok_DIR"]
match => { "message" => "%{NGINXACCESS}" }
}
}

It is not bulletproof in any version of nginx to hand craft the JSON - \xXX is not valid JSON encoding, it should be \u00XX. Any proper JSON parser will reject the JSON string if nginx escapes with \xXX.
If you do this with LogStash you will mostly be OK but expect json parse failures from time to time.

I think it's fine in recent versions of nginx:
Changes with nginx 1.1.6 17 Oct 2011
*) Change: now the 0x7F-0x1F characters are escaped as \xXX in an
access_log.
Changes with nginx 0.7.0 19 May 2008
*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
in an access_log.
Thanks to Maxim Dounin.

Related

Nginx variable for physical server name

I'm trying to setup response headers on my separate webservers that outputs the physical name of the machine that nginx is running on, so that I can tell which servers are serving the responses to our web clients.
Is there a variable that exists to do this already? Or do I just have to hardcode it per-server :(
You're after the $hostname common variable. Common variables are listed in the variable index.
The nginx access log documentation only shows variables that are specific to the access log:
The log format can contain common variables, and variables that exist
only at the time of a log write.
I guess you're looking for $hostname variable.
At first I thought the answer was to use the ENV variable and pull out the hostname from there https://docs.apitools.com/blog/2014/07/02/using-environment-variables-in-nginx-conf.html. But I couldn't get it to work for some reason.
However, this works like a charm:
perl_set $server_int 'sub { use Sys::Hostname; return hostname; }';
And example usage:
add_header 'Server-Int' "$server_int";
Just have to make sure your nginx is compiled with --with-http_perl_module - just run nginx -V to make sure. And that you have Sys::Hostname installed.
Warning: I at first used hostname to return the hostname in the Perl script, but while that did return the name, it for some reason aborted the rest of the output. I don't know if it's a bug with perl_set but you've been warned - using backticks in perl_set may be deadly.

Nginx Rewrite rule to replace '?' with %3f

Nginx is serving only static files, yet, some of file names contains '?'. Yes, the question mark.
All URLs that contains '?' yield 404 even though file actually exists. e.g.
> GET /foo?lang=ar.html HTTP/1.1
...
...
< HTTP/1.1 404 Not Found
While a file named foo?lang=ar.html does exists in the expected location.
> GET /foo%3flang=ar.html HTTP/1.1
...
...
< HTTP/1.1 200 OK
How do I write a rewrite directive so all '?' will be redirected to %3f?
You should url-encode your query string to escape special characters such as ? and =
Specifically, the name of your file you have to request for, once encoded, is this:
foo%3Flang%3Dar.html
In Javascript you can url-encode the filename with encodeURIComponent() function, in PHP you have urlencode().
You MUST encode the ? as %3F before the http call to nginx.
The reason is that the url rfc reserves the ? character as a special character (specifcally see section 3.3). Consequently nginx will, correctly, interpret an unescaped ? character as the end of the path part of the url

HTTP testing on the command line, is there something better than cURL?

Is there a command line utility where you can simply set up an HTTP request and have the trace simply output back to the console?
Also specifying the method simply would be a great feature instead of the method being a side effect.
I can get all the information I need with cURL but I can't figure out a way to just display it without dumping everything to files.
I'd like the output to show the sent headers the received headers and the body of the message.
There must be something out there but I haven't been able to google for it. Figured I should ask before going off and writing it myself.
I dislike answering my own question but c-smile's answer lead me down the right track:
Short answer shell script over cURL:
curl --dump-header - "$#"
The - [dash] meaning stdout is a convention I was unaware of but also works for wget and a number of other unix utilities. It is apparently not part of the shell but built into each utility. The wget equivalent is:
wget --save-headers -qO - "$#"
Did you try wget:
http://www.gnu.org/software/wget/manual/wget.html#Wgetrc-Commands ?
Like wget --save-headers ...
To include the HTTP headers in the output (as well as the server response), just use curl’s -i/--include option. For example:
curl -i "http://www.google.com/"
Here’s what man curl says about this setting:
-i/--include
(HTTP) Include the HTTP-header in the output. The HTTP-header
includes things like server-name, date of the document, HTTP-
version and more...
If this option is used twice, the second will again disable
header include.
Try http, e.g.
http -v example.org
Further into at https://httpie.org
It even includes a page to try online:
https://httpie.org/run
Telnet has for long been a well-known (though now forgotten, I guess) tool for looking at a web page. The general idea is to telnet to the http port, to type an http 1.1 GET command, and then to see the served page on the screen.
A good detailed explanation is http://support.microsoft.com/kb/279466
A Google search yields a whole bunch more.
Use telnet on port 80
For example:
telnet telehack.com 80
GET / HTTP/1.1
host: telehack.com
<CR>
<CR>
<CR> means Enter

Receive an HTTP 400 error if %2F is part of the GET URL in JBOSS

Whenever a URL that has %2F which is the hex code for / is posted to my JBOSS Server, I get an error:
HTTP 400 Bad Request error message.
Here is the URL:
http://localhost:8080/application/**abc%2Fhi**?msg=hello"
If I remove the %2F from the URL the link works fine.
This %2F has to be part of the URL and cannot be a request parameter.
Finally figured out the cause of this (both for JBoss and Apache). Both applications intentionally reject URIs with an encoded slash (%2F for / and %5C for \) to prevent possible security vulnerabilities.
Links:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0450
http://securitytracker.com/id/1018110 (Look at section 4. Solution)
And here are the instructions they provide for enabling this behavior in JBoss:
Note: In response to CVE-2007-0450, JBoss AS considers encoded slashes and backslashes in URLs invalid and its usage will result in HTTP 400 error. It is possible to allow encoded slashes and backslashes by following the steps outlined below, however doing so will expose you to CVE-2007-0450 related attacks:
a) If you use the /var/lib/jbossas/bin/run.sh setup, please edit /etc/jbossas/run.conf and append
- -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
- -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true to the string assigned to JAVA_OPTS
b) If you use the init script setup to run multiple JBoss AS services and you wish to allow encoding by default on all services, please edit /etc/jbossas/jbossas.conf and add the line JAVA_OPTS="${JAVA_OPTS}
- -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
- -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true"
c) If you use the init script setup to run multiple JBoss AS services and want to allow encoding of slashes and backslashes for a particular service, please edit /etc/sysconfig/${NAME} (where NAME is the name of your service) and add the line JAVA_OPTS="${JAVA_OPTS}
- -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
- -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true"
For Apache, it's as simple as setting "AllowEncodedSlashes NoDecode" somewhere in your apache conf or vhost conf (doesn't work in an .htaccess, however).
Apache link: http://httpd.apache.org/docs/current/mod/core.html#allowencodedslashes

How do you change the server header returned by nginx?

There's an option to hide the version so it will display only nginx, but is there a way to hide that too so it will not show anything or change the header?
If you are using nginx to proxy a back-end application and want the back-end to advertise its own Server: header without nginx overwriting it, then you can go inside of your server {…} stanza and set:
proxy_pass_header Server;
That will convince nginx to leave that header alone and not rewrite the value set by the back-end.
The last update was a while ago, so here is what worked for me on Ubuntu:
sudo apt-get update
sudo apt-get install nginx-extras
Then add the following two lines to the http section of nginx.conf, which is usually located at /etc/nginx/nginx.conf:
sudo nano /etc/nginx/nginx.conf
server_tokens off; # removed pound sign
more_set_headers 'Server: Eff_You_Script_Kiddies!';
Also, don't forget to restart nginx with sudo service nginx restart.
Like Apache, this is a quick edit to the source and recompile. From Calomel.org:
The Server: string is the header which
is sent back to the client to tell
them what type of http server you are
running and possibly what version.
This string is used by places like
Alexia and Netcraft to collect
statistics about how many and of what
type of web server are live on the
Internet. To support the author and
statistics for Nginx we recommend
keeping this string as is. But, for
security you may not want people to
know what you are running and you can
change this in the source code. Edit
the source file
src/http/ngx_http_header_filter_module.c
at look at lines 48 and 49. You can
change the String to anything you
want.
## vi src/http/ngx_http_header_filter_module.c (lines 48 and 49)
static char ngx_http_server_string[] = "Server: MyDomain.com" CRLF;
static char ngx_http_server_full_string[] = "Server: MyDomain.com" CRLF;
March 2011 edit: Props to Flavius below for pointing out a new option, replacing Nginx's standard HttpHeadersModule with the forked HttpHeadersMoreModule. Recompiling the standard module is still the quick fix, and makes sense if you want to use the standard module and won't be changing the server string often. But if you want more than that, the HttpHeadersMoreModule is a strong project and lets you do all sorts of runtime black magic with your HTTP headers.
It’s very simple: Add these lines to server section:
server_tokens off;
more_set_headers 'Server: My Very Own Server';
Simple, edit /etc/nginx/nginx.conf and remove comment from
#server_tokens off;
Search for http section.
Install Nginx Extras
sudo apt-get update
sudo apt-get install nginx-extras
Server details can be removed from response by adding following two lines in the nginx.conf (under http section)
more_clear_headers Server;
server_tokens off;
There is a special module: http://wiki.nginx.org/NginxHttpHeadersMoreModule
This module allows you to add, set, or clear any output or input header that you specify.
This is an enhanced version of the standard headers module because it provides more utilities like resetting or clearing "builtin headers" like Content-Type, Content-Length, and Server.
It also allows you to specify an optional HTTP status code criteria using the -s option and an optional content type criteria using the -t option while modifying the output headers with the more_set_headers and more_clear_headers directives...
If you're okay with just changing the header to another string five letters or fewer, you can simply patch the binary.
sed -i 's/nginx\r/thing\r/' `which nginx`
Which, as a solution, has a few notable advantages. Namely, that you can allow your nginx versioning to be handled by the package manager (so, no compiling from source) even if nginx-extras isn't available for your distro, and you don't need to worry about any of the additional code of something like nginx-extras being vulnerable.
Of course, you'll also want to set the option server_tokens off, to hide the version number, or patch that format string as well.
I say "five letters or fewer" because of course you can always replace:
nginx\r\0
with
bob\r\0\r\0
leaving the last two bytes unchanged.
If you actually want more than five characters, you'll want to leave server_tokens on, and replace the (slightly longer) format string, although again there's an upper limit on that length imposed by the length of the format string - 1 (for the carriage return).
...If none of the above makes sense to you, or you've never patched a binary before, you may want to stay away from this approach, though.
According to nginx documentation it supports custom values or even the exclusion:
Syntax: server_tokens on | off | build | string;
but sadly only with a commercial subscription:
Additionally, as part of our commercial subscription, starting from
version 1.9.13 the signature on error pages and the “Server” response
header field value can be set explicitly using the string with
variables. An empty string disables the emission of the “Server”
field.
After I read Parthian Shot's answer, I dig into /usr/sbin/nginx binary file. Then I found out that the file contains these three lines.
Server: nginx/1.12.2
Server: nginx/1.12.2
Server: nginx
Basically first two of them are meant for server_tokens on; directive (Server version included).
Then I change the search criteria to match those lines within the binary file.
sed -i 's/Server: nginx/Server: thing/' `which nginx`
After I dig farther I found out that the error message produced by nginx is also included in this file.
<hr><center>nginx</center>
There are three of them, one without the version, two of them included the version. So I run the following command to replace nginx string within the error message.
sed -i 's/center>nginx/center>thing/' `which nginx`
The only way is to modify the file src/http/ngx_http_header_filter_module.c . I changed nginx on line 48 to a different string.
What you can do in the nginx config file is to set server_tokens to off. This will prevent nginx from printing the version number.
To check things out, try curl -I http://vurbu.com/ | grep Server
It should return
Server: Hai
I know the post is kinda old, but I have found a solution easy that works on Debian based distribution without compiling nginx from source.
First install nginx-extras package
sudo apt install nginx-extras
Then load the nginx http headers more module by editing nginx.conf and adding the following line inside the server block
load_module modules/ngx_http_headers_more_filter_module.so;
Once it's done you'll have access to both more_set_headers and more_clear_headers directives.
Expanding on Parthian Shot's answer, you can actually replace the whole header and not only the value as long as the total length is the same:
sed -i 's/Server: nginx/My-Header: hi/' `which nginx`
Nginx-extra package is deprecated now.
The following therefore did now work for me as i tried installing various packages
more_set_headers 'Server: My Very Own Server';
You can just do the following and no server or version information will be sent back
server_tokens '';
if you just want to remove the version number this works
server_tokens off;
Are you asking about the Server header value in the response? You can try changing that with an add_header directive, but I'm not sure if it'll work. http://wiki.codemongers.com/NginxHttpHeadersModule

Resources