I'm new to kibana.I am working with data migration from MySQL to elasticsearch.How can i do this? Is using jdbc input plugin is the only process??
Here is the logstash.conf file where i specified the input and output:
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/kibana"
jdbc_user => "xxx"
jdbc_password => "xxxxx"
jdbc_driver_library => "/root/mysql-connector-java-5.1.30/mysql-connector-java-5.1.30-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM datalog"
}
}
output {
elasticsearch {
"hosts" => "localhost:9200"
}
stdout { codec => rubydebug }
}
after running the above file by using ./logstash -f logstash.conf we are getting the below error:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
I want to ship stdout from a running application to logz.io using logstash. Application and logstash are both docker images managed by docker-compose, which does the setup (pull images, network_mode, logging driver etc.). Logstash input is handled via gelf-input-plugin. The shipping to logz.io is handled via tcp-output-plugin.
logstash.conf:
input { gelf {
type => docker
port => 12201 } }
filter { mutate { add_field => { "token" => "${LOGZIOTOKEN}" } } }
output { tcp {
host => "listener.logz.io"
port => 5050
codec => json_lines } }
excerpt from docker-compose.yml:
application:
...
logging:
driver: "gelf"
options:
gelf-address: "udp://0.0.0.0:12201"
This works as expected.
Now there is a TCP proxy server I need to use, to ship the logs from the host (running the logstash instance) to logz.io. Unfortunately I did not find a proxy extension for logstashs tcp-output-plugin. Does anyone has a suggestion for this issue?
The logstash's http output plugin has the attribute proxy. You have to use the logz.io port for shipping with curl: 8070(http)/8071(https).
A working config looks like this:
output { http {
url => "https://listener.logz.io:8071?token=${LOGZIOTOKEN}"
http_method => "post"
format => "json"
content_type => "application/json"
proxy => {
host => "${PROXYHOST}"
port => "${PROXYPORT}"
scheme => 'http'
user => "${PROXYUSER}"
password => "${PROXYPW}"
}}}
You do not need the filter to ship to logz.io like in tcp-output-plugin config. Just add the input and ship it!
I am receiving data using Logstash (2.3) and want to mask them or add additional field, all using external web service as a source.
I have a web service available at something like:
someserver:8080/webpage?id=1
I would like to extract a value that I get using this URL, and inject it into data.
my config file looks like:
input {
file {
path => "/opt/logstash/test/*.csv"
start_position => "beginning"
sincedb_path => "/opt/logstash/test/output/test.db"
}
http { url => "http://localhost:8080/webpage"
}
}
filter {
csv {
columns => ["col1", "col2", "col3" ]
separator => ","
skip_empty_columns => true
}
}
output {
stdout { codec => rubydebug }
csv {
fields => ["col1","col2","col3" ]
path => "/opt/logstash/test/output/test.csv"
}
}
what I would like to achieve is to replace each col1 value with a value obtained from such external source.
I found http plugin, but it doesn't look straight forward to me.
tnx
recently I deployed ELK and started forwarding logs from nginx through logstash frowarder.
Problem is, that in elasticsearch (1.4.2) / kibana (4) is "bytes" value of request mapped as string.
I uses standard congfiguration found everywhere.
Into logstash patterns added new pattern for nginx logs:
NGUSERNAME [a-zA-Z\.\#\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float} %{NUMBER:upstream_time:float}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time:float}
Added these configs for logstash
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/logstash/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/logstash/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
} else if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
}
output {
elasticsearch_http {
host => localhost
}
}
But in elsticsearch I see that as string even if I define "bytes" as long
(?:%{NUMBER:bytes:long}|-)
Does anybody know how to store "bytes" as number type?
Thanks
You're on the right track with (?:%{NUMBER:bytes:long}|-), but "long" isn't a valid data type. Quoting the grok documentation (emphasis mine):
Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the num semantic from a string to an integer. Currently the only supported conversions are int and float.
Note that this doesn't control the data type that's actually used in the indexing on the Elasticsearch side, only the data type of the JSON document that's sent to Elasticsearch (which may or may not affect which mapping ES uses). In the JSON context there's no difference between ints and longs; scalar values are either numbers, bools, or strings.
Trying to use sqlite in development environment. It seems to detect the environment correctly but when I try to migrate to development.sqlite I get exception thrown "database does not exist"
artisan command
php artisan migrate --env=development
bootstrap/start.php
$env = $app->detectEnvironment(array(
'development' => array('localhost'),
));
app/config/development/database.php
<?php
return array(
'default' => 'sqlite',
'connections' => array(
'sqlite' => array(
'driver' => 'sqlite',
'database' => __DIR__.'/../database/development.sqlite',
'prefix' => '',
)
)
);
As far as I know laravel is supposed to create the file if it does not exist but since it didn't I tried manually creating the file and still get the exception thrown.
UPDATE: Maybe something not right with the env because the same thing happens if I try ':memory' for the database.
UPDATE 2: I tried running the sample unit test but add to TestCase.php
/**
* Default preparation for each test
*
*/
public function setUp()
{
parent::setUp(); // Don't forget this!
$this->prepareForTests();
}
/**
* Creates the application.
*
* #return Symfony\Component\HttpKernel\HttpKernelInterface
*/
public function createApplication()
{
$unitTesting = true;
$testEnvironment = 'testing';
return require __DIR__.'/../../bootstrap/start.php';
}
/**
* Migrates the database and set the mailer to 'pretend'.
* This will cause the tests to run quickly.
*
*/
private function prepareForTests()
{
Artisan::call('migrate');
Mail::pretend(true);
}
And this too gives the same exception though the testing env is already shipped with laravel. So I'll see if I can find any new issues on that.
Wow, typos and wrong paths.
Copying the sqlite array from config/database.php into config/development/database.php I forgot to change the path to the development.sqlite file from
__DIR__.'/../database/development.sqlite'
to
__DIR__.'/../../database/development.sqlite'
And for the in memory test it should have been
':memory:'
instead of
':memory'
I noticed that my database.php file had the following
'sqlite' => [
'driver' => 'sqlite',
'database' => env('DB_DATABASE', database_path('database.sqlite')),
'prefix' => '',
],
I changed it to read the following, and it worked just fine.
'sqlite' => [
'driver' => 'sqlite',
'database' => database_path('database.sqlite'),
'prefix' => '',
],
One of the problem which I faced was I use "touch storage/database.sqlite" in terminal, so database is created in Storage folder instead of database folder.
in my config/database.php path is database_path('database.sqlite')
'sqlite' => [
'driver' => 'sqlite',
'database' => database_path('database.sqlite'),
'prefix' => '',
],
than I use command "php artisan migrate" which gave me error "Database (/Applications/MAMP/htdocs/FOLDER_NAME/database/database.sqlite) does
not exist."
so it's obvious database file is not in database folder as It was generated in Storage folder, so copy "database.sqlite" from storage folder or run command "touch database/database.sqlite"
Hope that helps.!!
Well, my answer is kinda outdated, but anyway. I faced the same problem, but with Laravel 5, I am using Windows 7 x64. First I manually created SQLite database called 'db' and placed it into storage directory, then fixed my .env file like this:
APP_ENV=local
APP_DEBUG=true
APP_KEY=oBxQMkpqbENPb07bLccw6Xv7opAiG3Jp
DB_HOST=localhost
DB_DATABASE='db'
DB_USERNAME=''
DB_PASSWORD=''
CACHE_DRIVER=file
SESSION_DRIVER=file
QUEUE_DRIVER=sync
MAIL_DRIVER=smtp
MAIL_HOST=mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null`
I thought it would fix my problems, but the command line keeps telling me that database doesn't exist. And then I just checked the path to db in my database.php file and this is why I put database file into storage directory. But nothing changed. And finally I checked db's extension and it was .db, not .sqlite as default extension you see in your sqlite block in database.php. So this is how I reconfigured sqlite piece:
'sqlite' => [
'driver' => 'sqlite',
'database' => storage_path().'/db.db',
'prefix' => '',
],
And of course don't forget to set sqlite as default database in your database.php file. Good luck!
For me it was that path to database had to be '/var/www/html' + location to the database in your project. In my case database was stored in database/db.sqlite so DB_DATABASE='/var/www/html/database/db.sqlite'
I had the same error while running a GitHub action test workflow.
For me the solution was to define the relative path to the database archive into the workflow file:
on:
...
env:
DB_CONNECTION: sqlite
DB_DATABASE: database/database.sqlite
jobs:
laravel-tests:
...
I think that the previous answers reduce the importance of the config and most likely the developers wanted to get the database file like this:
'sqlite' => [
'driver' => 'sqlite',
'url' => env('DATABASE_URL'),
'database' => database_path(env('DB_DATABASE', 'database').'.sqlite'), // <- like this
'prefix' => '',
'foreign_key_constraints' => env('DB_FOREIGN_KEYS', true),
],
Tested on Laravel 9.x