Change nginx access log data in logstash or elasticsearch - nginx

Question

In my project I provide api for a mobile app , and in every api the front end use session_id to mark user authenticity, and in the server side accept and validate it.
Recently we want to use ELK(elasticsearch, logstash, kibana) to preserve and analyze web server access log to extract some commonly occurred user activities. I encountered some problems, I wanna change session_id in the log to user_id(in program I can get user_id from session_id through query database) but I just don't know how?
Can logstash's filter do this? or should I change data when log was indexed in elasticsearch?

Answer 1

Alright, I try to give you an answer assuming that you have some kind of interface from which you can retrieve the user_id. Actually you need to do two things:
Split your log line into separate fields to have a field which contains your session_id
Get the corresponding user_id using some kind of api
Split your log line
You need to split your input into separate fields. This could be done with filters like grok and/or kv. Take a look at some SO questions to find a matching grok pattern or use the grok debugger. Please provide a few log lines if you need help with that.
EDIT: For your given examples your configuration should look something like this:
filter {
grok {
match => [ 'message', '"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor}' ]
}
kv {
field_split => "&?"
}
}
Please try it and adjust it yourself to get the session_id.
Once you have a field called session_id you can go on with step 2.
Get the user_id
As you have already mentioned you need a filter plugin because the session_id must be available. There are several official plugins but I think none of them suits your purpose. Since the session_id is assigned dynamically you cannot use a static translate filter or something like that.
It depends on your api but one possible approach is to get the corresponding user_id via http requests. For that purpose you could use a community plugin. For example logstash-filter-rest with a config like this:
filter {
rest {
url => "http://yourserver/getUserBySessionId/"
sprintf => true
method => "post"
params => {
"session_id" => "%{session_id}"
}
response_key => "user_id"
}
}