logstash nginx pattern thows the result in _grokparsefailure - nginx

Question

I have an nginx patteer which was successfully tested in grokcontructor but when adding it to logstash 1.5.3 the logs do end up with _grokparsefailure
Here is a sample of my access.log:
207.46.13.34 - - [14/Aug/2015:18:33:50 -0400] "GET /tag/dnssec/ HTTP/1.1" 200 1961 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"
and here is the nignx pattern:
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} \[%{HTTPDATE:timestamp}] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:answer} %{NUMBER:byte} "%{URI:referrer}" %{QS:referee} %{QS:agent}
my logstash.conf look like this:
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/z0z0.tk.crt"
ssl_key => "/etc/pki/tls/private/z0z0.tk.key"
}
}
filter {
if [type] == "nginx-access" {
grok {
match => { "message" => "${NGINXACCESS}" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
host => "172.17.0.5"
cluster => "clustername"
flush_size => 2000
}
}

Answer 1

You're trying to match "-" into the field referrer using the pattern URI. Unfortunately, "-" is not a valid character in the URI pattern, which is expecting something like "http://..."
There are pattern examples that match a string or a hyphen (like part of the built-in COMMONAPACHELOG):
(?:%{NUMBER:bytes}|-)
which you could adjust to your pattern.

Answer 2

Thanks Alain for your suggestion I have recreated the pattern but having it in /opt/logstash/pattern/nginx did not work so I moved it to the logstash.conf which works and it looks like this:
if [type] == "nginx-access" {
grok {
match => { 'message' => '%{IPORHOST:clientip} %{NGUSER:indent} %{NGUSER:agent} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|)\" %{NUMBER:answer} (?:%{NUMBER:byte}|-) (?:\"(?:%{URI:referrer}|-))\" (?:%{QS:referree}) %{QS:agent}' }
}
}