I am receiving data using Logstash (2.3) and want to mask them or add additional field, all using external web service as a source.
I have a web service available at something like:
someserver:8080/webpage?id=1
I would like to extract a value that I get using this URL, and inject it into data.
my config file looks like:
input {
file {
path => "/opt/logstash/test/*.csv"
start_position => "beginning"
sincedb_path => "/opt/logstash/test/output/test.db"
}
http { url => "http://localhost:8080/webpage"
}
}
filter {
csv {
columns => ["col1", "col2", "col3" ]
separator => ","
skip_empty_columns => true
}
}
output {
stdout { codec => rubydebug }
csv {
fields => ["col1","col2","col3" ]
path => "/opt/logstash/test/output/test.csv"
}
}
what I would like to achieve is to replace each col1 value with a value obtained from such external source.
I found http plugin, but it doesn't look straight forward to me.
tnx
Related
I have a use case where I have to make an external HTTP call to an API endpoint in the logstash by passing the input coming from kafka as the body and read the output from the API call and store it in elasticsearch index. Can we achieve this through HTTP filter plugin? If so, my main question is how do we read the output from the API call and index it into elasticsearch.
I also see something called as http poller plugin, is this something that will be of use to me? Help would be appreciated. Thank you.
That's easy, you can use HTTP Pooler, something like:
input {
# 1. trigger new token requests every hour
http_poller {
urls => {
pipelines => "http://localhost:9600/_node/stats/pipelines?pretty"
}
schedule => { cron => "*/1 * * * *"}
codec => "json"
metadata_target => "http_poller_metadata"
}
}
then use an elasticsearch output like this:
output {
elasticsearch { hosts => ["localhost:9200"] }
}
You can get more info here
I hope this is useful for you.
I've been following this tutorial on how to use ELK stack for nginx logs.
I've created nginx.conf to configure how to get the logs but when i type: bin/logstash -f /etc/logstash/conf.d/nginx.conf
I get this error:
[ERROR] 2020-11-13 14:59:15.254 [Converge
PipelineAction::Create] agent - Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
[A-Za-z0-9_-], [ \t\r\n], "#", "=>" at line 9, column 8 (byte
135) after input{\n\t\n file{\n path =>
["/var/log/nginx/access.log" , "/var/log/nginx/error.log"]\n
type => "nginx"\n }\n filter{\n \n grok",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in
compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in
execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in block
in converge_state'"]}
and here's my nginx.conf file:
input{
file{
path => ["/var/log/nginx/access.log" , "/var/log/nginx/error.log"]
type => "nginx"
}
filter{
grok{
match => ["message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => ["message"]
}
mutate{
convert => ["response","integer"]
convert => ["bytes","integer"]
convert => ["responsetime","float"]
}
geoip{
source => "clientip"
target => "geoip"
add_tag => ["nginx-geoip"]
}
date {
match ⁼> ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
remove_field => ["timestamp"]
}
useragent {
source => "agent"
}
}
output{
elasticsearch {
hosts => ["localhost:9200"]
index => "nginx-%{+yyyy.MM.dd}"
document_type => "nginx_logs"
}
}
}
I found similar question but the answer didn't help.
Is there anyone familiar with logstash syntax and help figure out my error
Thank you
You are missing a } to close the input section. Insert it before the filter keyword.
Also, remove the last } in the file.
I'm new to kibana.I am working with data migration from MySQL to elasticsearch.How can i do this? Is using jdbc input plugin is the only process??
Here is the logstash.conf file where i specified the input and output:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/kibana"
jdbc_user => "xxx"
jdbc_password => "xxxxx"
jdbc_driver_library => "/root/mysql-connector-java-5.1.30/mysql-connector-java-5.1.30-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM datalog"
}
}
output {
elasticsearch {
"hosts" => "localhost:9200"
}
stdout { codec => rubydebug }
}
after running the above file by using ./logstash -f logstash.conf we are getting the below error:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
I want to ship stdout from a running application to logz.io using logstash. Application and logstash are both docker images managed by docker-compose, which does the setup (pull images, network_mode, logging driver etc.). Logstash input is handled via gelf-input-plugin. The shipping to logz.io is handled via tcp-output-plugin.
logstash.conf:
input { gelf {
type => docker
port => 12201 } }
filter { mutate { add_field => { "token" => "${LOGZIOTOKEN}" } } }
output { tcp {
host => "listener.logz.io"
port => 5050
codec => json_lines } }
excerpt from docker-compose.yml:
application:
...
logging:
driver: "gelf"
options:
gelf-address: "udp://0.0.0.0:12201"
This works as expected.
Now there is a TCP proxy server I need to use, to ship the logs from the host (running the logstash instance) to logz.io. Unfortunately I did not find a proxy extension for logstashs tcp-output-plugin. Does anyone has a suggestion for this issue?
The logstash's http output plugin has the attribute proxy. You have to use the logz.io port for shipping with curl: 8070(http)/8071(https).
A working config looks like this:
output { http {
url => "https://listener.logz.io:8071?token=${LOGZIOTOKEN}"
http_method => "post"
format => "json"
content_type => "application/json"
proxy => {
host => "${PROXYHOST}"
port => "${PROXYPORT}"
scheme => 'http'
user => "${PROXYUSER}"
password => "${PROXYPW}"
}}}
You do not need the filter to ship to logz.io like in tcp-output-plugin config. Just add the input and ship it!
recently I deployed ELK and started forwarding logs from nginx through logstash frowarder.
Problem is, that in elasticsearch (1.4.2) / kibana (4) is "bytes" value of request mapped as string.
I uses standard congfiguration found everywhere.
Into logstash patterns added new pattern for nginx logs:
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}
Added these configs for logstash
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/logstash/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/logstash/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
} else if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
}
output {
elasticsearch_http {
host => localhost
}
}
But in elsticsearch I see that as string even if I define "bytes" as long
(?:%{NUMBER:bytes:long}|-)
Does anybody know how to store "bytes" as number type?
Thanks
You're on the right track with (?:%{NUMBER:bytes:long}|-), but "long" isn't a valid data type. Quoting the grok documentation (emphasis mine):
Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the num semantic from a string to an integer. Currently the only supported conversions are int and float.
Note that this doesn't control the data type that's actually used in the indexing on the Elasticsearch side, only the data type of the JSON document that's sent to Elasticsearch (which may or may not affect which mapping ES uses). In the JSON context there's no difference between ints and longs; scalar values are either numbers, bools, or strings.