Messages dropped in Syslog-NG - Buffer configuration? - syslog

Question

i'm using Syslog-NG 3.8 as a syslog server receiving messages from many different sources, about 400 servers (& filtering them and relaying to a splunk server eventually).
however, it looks like many messages are "dropped" even before they are filtered and forwarded to the splunk instance.
i have a configuration where i'm keeping track of "incoming" messages before filtering in a flat file and i can't see some messages in there (see example below), while i can see the tcp trace successfully when running a tcpdump, meaning that Syslog-NG "drops" messages during the "source" mecanism.
i suspect this is due to the large mount of messages incoming on its interface, and i need to do some tuning with buffers & specific options.
here is a concrete example of the bug :
from the source machine, if i do a little loop sending messages every second with an ID that increments (so 20,21,22,23,24 & so on) :
root#sm1u1050vmo /var/log: for ((i=20;i<100;i++)); do logger -p auth.notice "test auth notice $i" ; sleep 1 ; done
if i tail the "incoming.log" flat file on Syslog-NG (receiver), i can see many missing messages :
[root#xm1p1034vmo 20]# tail -f incoming.log | grep sm1u1050vmo
Sep 20 12:27:32 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 28
Sep 20 12:27:34 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 30
Sep 20 12:27:37 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 33
Sep 20 12:27:42 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 38
Sep 20 12:27:43 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 39
Sep 20 12:27:52 sm1u1050vmo root: [ID 702911 auth.notice] test auth notice 48
we can clearly see many messages missing.
here is a small part of my config, with options & sources :
options {
chain_hostnames(no);
log_msg_size(8192);
time_reopen(2);
create_dirs(yes);
use_dns(yes);
keep_hostname(yes);
stats_freq(3600);
flush_lines(1);
log_fifo_size(1000);
};
the source being used :
source s_EXTERNAL {
network(transport("udp") log-fetch-limit(500));
};
the "local copy" destination being used to track these incoming messages before they are filtered :
destination d_INCOMING_ALL
{
file("/app/syslog-ng/logs/${YEAR}/${MONTH}/${DAY}/incoming.log" create-dirs(yes));
};
note, i am currently trying to play with options such as so-rcvbuf() in the source (& adapting the kernel parameter rmem_max at the same time). it was 16Mb and i increased it to 64M but it didn't change anything for now. i can still see RcvbufErrors & packet receive errors increasing when running a netstat -us
any hints please ?
thanks

Answer 1

I'd tell you a UDP joke, but you probably wouldn't get it. Seriously, UDP is not guaranteed to arrive. Change the protocol to TCP on each end, and you should be set. Protocol aside, if your system is dropping messages at the rate of 1 per second, you likely have configuration issues on your network or it's simply overloaded.