We use following nginx site configure file in our production env.
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
server {
root /srv/www/web;
server_name *.test.com;
access_log /var/log/nginx/xxx.test.com.access.log main;
Both "http://a.test.com/ping" and "http://b.test.com/ping" http request will be record in file xxx.test.com.access.log.
But there is a problem, nginx don't store "domain name" in xxx.test.com.access.log.
"http://a.test.com/ping" and "http://b.test.com/ping" share the same request "Get /ping".
How can I record "a.test.com" or "b.test.com" in nginx log?
Try adding the $host variable in log_format:
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$host" "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
http://wiki.nginx.org/HttpCoreModule#.24host:
$host
This variable is equal to line Host in the header of request or name
of the server processing the request if the Host header is not
available.
This variable may have a different value from $http_host in such
cases: 1) when the Host input header is absent or has an empty value,
$host equals to the value of server_name directive; 2) when the value
of Host contains port number, $host doesn't include that port number.
$host's value is always lowercase since 0.8.17.
If you want to log the full requested url, then my method is
log_format main '$http_x_forwarded_for - $remote_user [$time_local] '
'"$request_method $scheme://$host$request_uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
So splitting the $request into its constituent parts and popping $host in the middle. Also lets you see if the request is http or https.
Related
I have a custom format of nginx access.log and I want to parse it with logstash.
The format of the access log is the following:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" ';
I was not able to find what kind of grok match pattern to use. Any idea of how can I deal with it?
Try this:
%{IPORHOST:remote_addr} - %{USER:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:api_path}%{URIPARAM:api_param}? %{DATA:http_version}" %{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} "%{DATA:http_referer}" "%{DATA:user_agent}" "%{DATA:http_x_forwarded_for}"
I am seeing the following logged in NGINX error log. This is the NGINX reverse proxy that comes with my AWS Elastic Beanstalk deployment.
2022/06/12 09:52:48 [error] 6155#6155: *112735 testing "/var/app/current/public" existence failed (2: No such file or directory) while logging request, client: xx.xx.xx.xx, server: , request: "GET /api/v1?params=xxx HTTP/1.1", upstream: "http://127.0.0.1:5000/api/v1?params=xxx", host: "my.domain.com"
This is a valid request, so I am not sure why it results in an error statement, or what exactly doesn't exist.
I am not overriding the default log pattern for the servers error output, only the access logging, with the following pattern:
'$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';
I am running nginx with default log format
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
It prints the access log in following format
1.2.3.4 - - [27/Mar/2021:09:39:43 +0000] "GET /text.txt HTTP/1.1" 200 315 "-" "curl/7.55.1" "-"
However I want the logs to print time in UTC +05:30 which is not the local time of the server. I tried adding env "TZ=Asia/Kolkata"; at the top of nginx.conf but it didn't work. Please suggest how to do this.
Regarding setting timezone in Nginx logs: Works fine; just need to set the "TZ" environment variable for nginx daemon (e.g. in the script that launches it). Or if you're using Docker, set it in the Dockerfile (or in the "environment" in docker-compose.yml file if you're using that).
My server is compiled on a docker.
The Nginx container is built from a standard assembly.
I want to read the access.log nginx but I see this kind of content:
172.68.244.173 - - [24/Aug/2018:12:14:04 +0000] "\x16\x03\x01\x00\xEC\x01\x00\x00\xE8\x03\x03\x8A?\xB5\xFA\x17?\x8A\x9B\x04T>yK\x1A\xF6\x8F_\xBE:.\xF9\xED\xF6\xEE\xFCM\xD0\x88Ji\xDD\xF5 \xFF\xBDm\x98#mo:U\xA6\x0E\xB7\x93\x02sm`\xC6\xD1s0vV*\x88y\xDA&\xFCfZ\xF4\x00\x16\x13\x01\x13\x02\x13\x03\xC0+\xC0/\xC0\x13\x00\x9C\x00/\xC0(\x005\x00" 400 173 "-" "-"
How to read such a log? What does this mean?
According to nginx documentation the default access log format is:
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
Applied to your log line:
$remote_addr = 172.68.244.173
(literal string for compatibility reasons) = -
$remote_user (from Auth Header) = -
$time_local = [24/Aug/2018:12:14:04 +0000]
$request = "\x16\x03\x01\x00\xEC\x01\x00\x00\xE8\x03\x03\x8A?\xB5\xFA\x17?\x8A\x9B\x04T>yK\x1A\xF6\x8F_\xBE:.\xF9\xED\xF6\xEE\xFCM\xD0\x88Ji\xDD\xF5 \xFF\xBDm\x98#mo:U\xA6\x0E\xB7\x93\x02sm`\xC6\xD1s0vV*\x88y\xDA&\xFCfZ\xF4\x00\x16\x13\x01\x13\x02\x13\x03\xC0+\xC0/\xC0\x13\x00\x9C\x00/\xC0(\x005\x00"
$status = 400
$body_bytes_sent = 173
$http_referer = "-"
$http_user_agent = "-"
To summarize: Your server received a request from the address 172.68.244.173 with no user agent header sent and the request consisted of mostly non-printable characters. Slight possibility this is a broken client sending a bad request, more likely it's an attempt to discover a vulnerability in your web server or application. This will happen often to any server on the internet.
I am extracting custom header values and logging them in the access log file using a custom log format:
log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$http_my_custom_header'
When logging requests that do not contain the header, ngx_http_log_module inserts a - into the logs. Is it possible to define a default value for missing headers? I need the header to always have a numeric value for later indexing in elasticsearch.
I have a workaround using map which sets the header value to 0 when absent:
map $http_my_custom_header $custom_header {
default $http_my_custom_header;
'' 0;
}
log_format custom '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$custom_header'
I'm hoping for a more elegant solution as I will be adding more headers in the future.