Several tcp connections from same syslog-ng clients - tcp

We have a syslog-ng server with several rsyslog clients. Over time some of them open a lot of client connections to the server instead of just one TCP connection. From the client perspective a netstat shows only one connection but from the server side netstat shows several of them for the same client.
Anyone ever had a similar pb ? What could that be?
Server conf:
#version:3.2
# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
options {
time_reopen (10);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (no);
stats-freq (3600);
};
source s_sys {
file ("/proc/kmsg" program_override("kernel: "));
unix-stream ("/dev/log");
internal();
};
source s_network {
tcp(ip(serverIP) port(601) max-connections(100) log-fetch-limit(100) log-iw-size(10000));
};
#FROM REMOTE CLIENTS
destination d_clients { file("/var/log/messages_${HOST}" perm(0644)); };
template log2db {
template("INSERT INTO logs (host, facility, priority, level, tag, datetime, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n");
template_escape(no);
};
destination go2db {
program( "/usr/bin/mysql -u myusername --password=mypass mybddname -Bs > /dev/null"
template(log2db) log_fifo_size(30000) flush_lines (100));
};
log { source(s_network); destination(d_clients); };
log { source(s_network); destination(go2db); flags(flow-control); };
Client conf:
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad immark # provides --MARK-- message capability
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
$MarkMessagePeriod 3600
$preserveFQDN on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
*.*;cron.none;cron.warning ##serverIP:601
Thanks,

Related

Reference outside text file content from Nginx configuration file

I am looking at options to add client-side certificate authentication with a fingerprint whitelist to a local site, and have successfully configured nginx to operate in the intended manner. My configuration is as follows:
# Client Certificate Whitelisting
map $ssl_client_fingerprint $reject {
default 1;
<ALLOWED_FINGERPRINT_1> 0;
<ALLOWED_FINGERPRINT_2> 0;
...
<ALLOWED_FINGERPRINT_N> 0;
}
server {
...
ssl_client_certificate /etc/pki/tls/certs/Private-CA-bundle.pem;
ssl_verify_client on;
...
if ($reject) { return 403; }
...
}
However, I would like to store the fingerprint list in a separate text file, rather than manipulating the nginx configuration file directly each time. Is this possible?
As a bonus, it would be great if I could modify the contents of the text file and have them take effect without reloading nginx. It is acceptable for removals to still require a service restart or other manual session teardown procedure.
---- EDIT ----
Based on the accepted answer, I was able to get this working.
The updated configuration file is:
# Client Certificate Whitelisting
map $ssl_client_fingerprint $reject {
default 1;
include /etc/nginx/cert-whitelist;
}
I was able to add a new certificate and apply the changes without a full service restart.
### Attempt connection with client certificate; returns 403 Forbidden
[root]# cat /run/nginx.pid
5606
[root]# echo "${FINGERPRINT} 0;" >> /etc/nginx/cert-whitelist
[root]# kill -1 $(cat /run/nginx.pid)
[root]# cat /run/nginx.pid
5606
### Attempt connection with client certificate; success
The map directive has the ability to source a correctly formatted file. See this document for details.
You can use SIGHUP to re-read the configuration file without restarting Nginx. See this document for details.

Error while trying to send logs with rsyslog without local storage

I'm trying to send logs into datadog using rsyslog. Ideally, I'm trying to do this without having the logs stored on the server hosting rsyslog. I've run into an error in my config that I haven't been able to find out much about. The error occurs on startup of rsyslog.
omfwd: could not get addrinfo for hostname '(null)':'(null)': Name or service not known [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Here's the portion I've added into the default rsyslog.config
module(load="imudp")
input(type="imudp" port="514" ruleset="datadog")
ruleset(name="datadog"){
action(
type="omfwd"
action.resumeRetryCount="-1"
queue.type="linkedList"
queue.saveOnShutdown="on"
queue.maxDiskSpace="1g"
queue.fileName="fwdRule1"
)
$template DatadogFormat,"00000000000000000 <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n "
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.logs.datadoghq.com
*.* ##intake.logs.datadoghq.com:10516;DatadogFormat
}
First things first.
The module imudp enables log reception over udp.
The module omfwd enables log forwarding over (tcp, udp, ...)
So most probably - or atleast as far as i can tell - with rsyslog you just want to log messages locally and then send them to datadog.
I don't know anything about the $ActionSendStreamDriver tags, so I can't help you there. But what is jumping out is, that in your action you haven't defined where the logs should be sent to.
ruleset(name="datadog"){
action(
type="omfwd"
target="10.100.1.1"
port="514"
protocol="udp"
...
)
...
}

Enabling NCSA access log with wsadmin scrpting

How can someone enable NCSA access log by using wsadmin script.
To view the settings page for an HTTP channels in WAS console we follow these steps:
Servers > Server Types > WebSphere application servers > server > Web Container Settings > Web container transport chains > Chain > HTTP inbound channel.
On the console, there is no administrative assistance for this task!!
Thank you
Use below snippet. I have added comments for each step.
Update the first two lines - serverName and chainName as per your environment before running this.
serverName = 'server1'
chainName = 'Chain'
#update this variable to true/false to toggle logging on/off
loggingEnabled = 'true'
#Get the server id
serverId = AdminConfig.getid('/Server:%s' %(serverName))
#Get the list of all Web Container transport chains
wcTransportChains = AdminTask.listChains(AdminConfig.list("TransportChannelService", serverId), '[-acceptorFilter WebContainerInboundChannel]').splitlines()
#Iterate the list and find the chain we are interested in
for chain in wcTransportChains:
if chain.startswith(chainName):
#list all transport channles for this chain
transportChannels = AdminConfig.showAttribute(chain, 'transportChannels').split(" ")
#iterate the list and find HTTPInboundChannel to enable NCSA logging
for channel in transportChannels:
if channel.find('HTTPInboundChannel') != -1:
#Enable logging config
print ('\nEnabling NCSA logging for Transport Channel : %s on server : %s\n' %(AdminConfig.showAttribute(channel, 'name'), serverName))
AdminConfig.modify(channel, [['enableLogging', loggingEnabled]])
#end if
#end for
#end if
#end for
#save the changes
AdminConfig.save()

Logs not received on remote syslog server

I am trying to centralize logs (/var/log/secure and /var/log/messages) from a Linux server (rsyslog) to a Solaris server (syslog).
My rsyslog.conf file:
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
*.info;mail.none;authpriv.none;cron.none #logserver:514
# The authpriv file has restricted access.
authpriv.* /var/log/secure
authpriv.* #logserver:514
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
On Solaris I configured the server to accept logs:
svccfg -s system-log setprop config/log_from_remote = true
I restarted syslog and rsyslog, but it's not working.
UDP is probably firewalled. Use the Solaris snoop command to see if data is coming in.

Flume syslogtcp source using rsyslog

I need to ingest data from remote server using flume to hdfs::
I have used source as syslogtcp. My flume.conf file is as:
Agent.sources = syslog
Agent.channels = MemChannel
Agent.sinks = HDFS
Agent.sources.syslog.type = syslogtcp
Agent.sources.syslog.channels = MemChannel
Agent.sources.syslog.port = 5140
Agent.sources.syslog.host = localhost
Agent.sinks.HDFS.channel = MemChannel
Agent.sinks.HDFS.type = hdfs
Agent.sinks.HDFS.hdfs.path = hdfs://192.168.111.130:8022/user/cloudera/Twitter/apple_data/%y/%m/%d/
Agent.sinks.HDFS.hdfs.fileType = DataStream
Agent.sinks.HDFS.hdfs.writeFormat = Text
Agent.sinks.HDFS.hdfs.batchSize = 1000
Agent.sinks.HDFS.hdfs.rollSize = 0
Agent.sinks.HDFS.hdfs.rollCount = 10000
Agent.channels.MemChannel.type = memory
Agent.channels.MemChannel.capacity = 10000
Agent.channels.MemChannel.transactionCapacity = 100
I have a file in /etc/rsyslog.d/B2B.conf
# rsyslog v5 configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
#$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
#$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ModLoad imfile
$InputFileName /home/cloudera/Desktop/my_logs/my_log.txt
$InputFileTag tag1:
$InputFileStateFile stat-file1
$InputFileFacility local7
$InputRunFileMonitor
$InputFilePollingInterval 10
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* ##192.168.111.130:514
# ### end of the forwarding rule ###
Now when I run the java application to create log, flume and rsyslog:
It halts at
Shutting down system logger:
Starting system logger:

Resources