Syslog-ng multiline input over TCP/Network module - tcp

Question

We are attempting to capture logs to syslog-ng over tcp connection. The logs looks something like this:
1810717353--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--<11>Oct 23 16:03:23 HOSTNAME [443-Multiplexer][0x80000001][xsltmsg][error] xmlfirewall(443-Multiplexer): trans(2607607975)[error][ip] gtid(value):
1810717354--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00-- Default rule caught error code '
1810717355--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00-- 0x00230001
1810717356--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00-- '
1810717357--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--
1810717358--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--<14>Oct 23 16:03:23 HOSTNAME [443-Multiplexer][value][multistep][info] xmlfirewall(443-Multiplexer): trans(2607607975)[error][ip] gtid(2607607975):
Is there any way to strip out the syslog header during the input? If the logs remain multiline, thats fine, we can package them with logstash during read, but we need a way for the logs to now have a header when they are written to file ..
Here is my current syslog-ng config :
#version: 3.6
##include "scl.conf"
options {
flush_lines (0);
frac-digits(8);
use-rcptid (yes);
time_reopen (10);
log_fifo_size (1000);
chain_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
use-rcptid (yes);
threaded(yes);
log_msg_size(32768);
};
source sourcename
{
network(ip("0.0.0.0") port(5521) flags(no-parse));
};
destination destname
{
file("/opt/elk/data/syslogs/datapower/${HOST}.log"
#follow-freq(0) multi-line-mode(regexp) multi-line-prefix("^\<\[0-9]{2}\>[A-z]{3} [0-9]{2} [0-9]{2}\:[0-9]{2}\:[0-9]{2}")
create_dirs(yes) dir_owner("elkuser") dir_group("users") dir_perm(0700)
owner("user") group("group") perm(0600)
template("${RCPTID}--$FACILITY--$PRIORITY--$FULLHOST--$PROGRAM--$ISODATE--${MSGHDR}${MSG}\n")
);
};
Thanks

Answer 1

I'm not sure where you want to split the message, but depending on that, you can do the following:
If it is enough to delete a fix number of characters, you can use the substr template function (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-template-functions.html)
If you want to join the multiline messages, you might want to check the multiline options of the network source: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-network.html
If you need a lot of custom processing for these logs, it might be easier to write a custom python parser ( https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/python-parser.html ). Note that Python parser requires a recent version of syslog-ng.
Also, it seems that you are sending the logs to Elasticsearch somehow (at least I suspect that's what elk in the destination stands for): recent versions of syslog-ng can do that directly: https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-elasticsearch2.html
HTH,
Robert