I am using multiple conf files inside my main conf file. However, the service is constantly taking on the first conf file.
Below is my main conf file code,
#INCLUDE fluent-bit-for-application-info-log.conf
#INCLUDE fluent-bit-for-application.conf
I have tested with different orders but every time it is constantly picking up only the first line.
[SERVICE]
flush 1
log_level debug
parsers_file parsers.conf
plugins_file plugins.conf
storage.metrics on
[INPUT]
name tail
path D:\xxxx\**\*.log
parser iis
Multiline Off
Skip_Empty_Lines On
refresh_interval 10
Tag iis_log
[OUTPUT]
Name s3
Match iis_log
Bucket xxxx
region ap-xx-1
s3_key_format /xxxx/xxxxx/%Y/%m/%d/%H/$UUID.log
Is there any additional config we need to add to process all config files?
Note: All my config files have the same code except path, bucket and tag details.
Please let me know if you need any information.
Thanks in Advance.
Related
We're got a pre-existing rsyslog config file which is working for papertrail e.g.
/etc/rsyslog.d/20-papertrail.conf which has
*.* #logs4.papertrailapp.com:44407
However we've got a couple of NGINX websites on the server so would like to have it also monitor their error logs.
The paths to them are:
/var/log/nginx/www.website-one.com-error.log
/var/log/nginx/www.website-two.com-error.log
/var/log/nginx/www.website-three.com-error.log
However this /var/log/nginx also contains a bunch of .log files which we do not want to monitor e.g.
/var/log/nginx/error.log
/var/log/nginx/access.log
/var/log/nginx/error.log1
/var/log/nginx/nginx.log
In my head we need to add something like...
/var/log/nginx/*-error.log
And make sure they pipe to the papertrail url as well.
However I'm struggling to decipher the rsyslog documentation to figure out how to do this.
Thanks!
In rsyslog documentation it seems that you can use wildcards in files.
File
The file being monitored. So far, this must be an absolute name (no macros or templates). Note that wildcards are supported at the file name level (see WildCards below for more details).
WildCards
Before Version: 8.25.0
Wildcards are only supported in the filename part, not in directory names.
/var/log/*.log works.
/var/log/*/syslog.log does not work.
Since Version: 8.25.0
Wildcards are supported in filename and paths which means these samples will work:
/var/log/*.log works.
/var/log/*/syslog.log works.
/var/log/*/*.log works.
All matching files in all matching subfolders will work. Note that this may decrease performance in imfile depending on how many directories and files are being watched dynamically.
If you want to forward your vhosts logs you can change configuration directly in NGINX vhosts configuration, you should change/add access_log and error_log policies as explained here or use custom facilities to forward your logs (using rsyslog).
HOW TO DO IT USING RSYSLOG?
Create a new custom file in /etc/rsyslog.d/nginx_custom.conf:
module(load="imfile" PollingInterval="1") #needs to be done just once
# File 1
input(type="imfile"
File="/var/log/nginx/www.website-*.com-error.log"
Tag="websites"
Facility="local0")
local0.* #logs4.papertrailapp.com:44407
#Just to test that logs are forwarded, comment the line once you've tested it
local0.* /var/log/test.log
And restart rsyslog service
NOTE: Line local0.* /var/log/test.log is just to test that you can see forwarded logs into your local server, comment this line after you've tested that everything works.
We have tried to migrate some SQL versions in a single database and it went well. When to tried to implement the migrations for multiple databases at the same time by passing multiple config files is failing.
The issue is it takes only the last config file and the migration is performed only for the database mentioned in the last config file, when passed the multiple config files in "-configFiles" parameter.
Below is the screenshot of the same, it took only flywayconfdb.conf file and left other files.
[oracle#localhost flyway-5.1.4]$ ./flyway -configFiles=/home/oracle/flyway/flyway-5.1.4/conf/flyway.conf,/home/oracle/flyway/flyway-5.1.4/conf/flywayjiradb.conf,/home/oracle/flyway/flyway-5.1.4/conf/flywayconfdb.conf info
Flyway Community Edition 5.1.4 by Boxfuse
Database: jdbc:oracle:thin:#//XXXXXXXXX:1521/confdb (Oracle 12.2)
Schema version: << Empty Schema >>
+----------+---------+-------------+------+--------------+-------+
| Category | Version | Description | Type | Installed On | State |
+----------+---------+-------------+------+--------------+-------+
| No migrations found |
+----------+---------+-------------+------+--------------+-------+
Please help us in resolving the same.
Flyway merges the config files. It doesn't do a separate migration for each one.
For each config file, Flyway adds the content to a Properties map. Properties has only one value per key, so if the same key appears in a second config file it would overwrite the previous value. This is why it seems like just the settings from the last config file are used.
It allows you to define some common settings somewhere, for example in ~/flyway.conf, which could be merged with some more specific settings, e.g. in individual projects.
But it doesn't allow you to migrate multiple databases in a single run. You need to run Flyway once per database:
./flyway -configFiles=/home/oracle/flyway/flyway-5.1.4/conf/flywayjiradb.conf info
./flyway -configFiles=/home/oracle/flyway/flyway-5.1.4/conf/flywayconfdb.conf info
The documentation describes the Overriding Order as:
Command-line arguments
Environment variables
Custom config files
<current-dir>/flyway.conf
<user-home>/flyway.conf
<install-dir>/conf/flyway.conf
Flyway command-line defaults
With settings defined higher up the list having greater precedence.
The documentation gives the following example:
The means that if for example flyway.url is both present in a config
file and passed as -url= from the command-line, the command-line
argument will take precedence and be used.
The Custom config files (-configFiles) lines could be expanded as:
Command-line arguments
Environment variables
Custom config file n
...
Custom config file 2
Custom config file 1
<current-dir>/flyway.conf
<user-home>/flyway.conf
<install-dir>/conf/flyway.conf
Flyway command-line defaults
And a corresponding example could be:
The means that if for example flyway.url is both present in custom config file 1 and custom config file 2, the custom config file 2 settings will take precedence and be used.
Similarly, if the flyway.url was also in custom config file n that would override that setting from custom config file 2.
I'm trying to send multiple nginx logs to loggly...
Config file: /etc/rsyslog.d/21-nginx.conf
$ModLoad imfile
#$InputFilePollInterval 10
$InputFileMode inotify
$WorkDirectory /var/spool/rsyslog
$PrivDropToGroup adm
# nginx access file:
$InputFileName /var/log/nginx/*access.log
$InputFileTag nginx-access:
$InputFileStateFile stat-nginx-access
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
# other stuff continues......
after restart i get this error in log syslog:
imfile: The to-be-monitored file "/var/log/nginx/*access.log" contains wildcards. This is not supported in polling mode. [v8.16.0 try http://www.rsyslog.com/e/2420 ]
activation of module imfile failed [v8.16.0 try http://www.rsyslog.com/e/-3 ]
did i make something wrong?
Are there other places in your rsyslog configuration where the file mode is changed to pulling or the file poll interval is active? The problem with using this kind of legacy syntax is that all the configuration is loaded globally, so things in other configuration files can interact. You might consider using the new action syntax so that the inotify mode is applied to the specific source. You can see an example of it here http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html
Showing currently applied configuration values
In v2.0+ of Riak there is a new command option: riak config effective
Which I read as it would tell you the current running values of riak.
At any time, you can get a snapshot of currently applied
configurations through the command line. For a listing of all of the
configs currently applied in the node
Config changes applied only on start of each node?
In multiple locations in Riak documentation there is reference like:
Remember that you must stop and then re-start each node when you
change storage backends or modify any other configuration
Problem:
However when I made a change to a setting (I've tested this in both riak.conf and advanced.conf), I see the newest value when running: riak config effective
ie:
Start node: riak start
View current setting for log level: riak config effective | grep log.console.level
log.console.level = info
Change the level to debug (something that will output a lot to console.log)
Re-run: riak config effective | grep log.console.level, we get:
log.console.level = debug
Checking the console log file for debug: cat /var/log/riak/console.log | grep debug give no results (indicating the config change has not been applied)
So the question is, how can I retrieve and verify what config setting each Riak node is running under?
When Riak starts, it creates two files: 'app..config' and 'vm..config'. The default location is in a 'generated.configs' directory under the platform data directory (usually /var/lib/riak).
These files will contain the settings that were in place when Riak was started. The command riak config effective processes the current riak.conf and advanced.config files.
has anyone tried to use a log4j.xml reference within a WinRun4j service configuration. here is a copy of my service.ini file. I have tried many configuration combinations. this is just my latest attempt
service.class=org.boris.winrun4j.MainService
service.id=SimpleBacnetIpDataTransfer
service.name=Simple Backnet IP DataTransfer Service
service.description=This is the service for the Simple Backnet IP DataTransfer.
service.startup=auto
classpath.1=C:\Inbox\DataTransferClient-1.0-SNAPSHOT-jar-with-dependencies.jar
classpath.2=WinRun4J.jar
classpath.3=C:\Inbox\log4j-1.2.16.jar
arg.1=C:\Inbox\DataTransferClient.xml
log=C:\WinRun4J-Service\SimpleBacnetIpDataTransfer\NBP-DT-service.log
log.overwrite=true
log.roll.size=10MB
[MainService]
class=com.shiftenergy.ws.App
vmarg.1=-Xdebug
vmarg.2=-Xnoagent
vmarg.3=-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
vmarg.4=-Dlog4j.configuration=file:C:\Inbox\log4j.xml
within the log4j.xml file, there is reference to a log file for when the application runs. if I run the java -jar -Dlog4j.configuration=file:C:\Inbox\log4j.xml ...., the log file is created accordingly. if I register my service and start the service, the log file does not get created.
has anyone had success using the -D log4j configuration, using winrun4j?
thanks
I think that you provided the vmarg.4 parameter incorrectly. In your case it has to be like:
vmarg.4=-Dlog4j.configurationFile=[Path for log4j.xml]
I am also using the same and in my case, it is working perfectly fine. Please see below example:
vmarg.1=-Dlog4j.configurationFile=.\log4j2.xml
Have you tried setting the path in your code instead:
System.setProperty("log4j.configurationFile", "config/log4j.xml");
I'm using a relative path to a folder named config that contains log4j.xml. An absolute path is not recommended, but may work as well.
Just be sure to set this before making any calls to log4j, including any log4j config settings or static method calls!
System.setProperty("log4j.configurationFile", "config/log4j.xml");
final Logger log = Logger.getLogger(Main.class);
log.info("Starting up");
I didn't specify the log4j path in the ini file, only placed log4j.xml file at the same place the jar was placed.
Also without specify the
System.setProperty("log4j.configurationFile", "config/log4j.xml");
In the Java project it was stored in (src/main/resources) and will be included in the jar, but it will not be that one used if placed outside the jar.