I have an udp input in fluentd :
<source> # mylog
#type udp
port 41234
tag json
<parse>
#type none
</parse>
</source>
I'd like to pass it through a tcp ouput,because I go through haproxys and then in a logstash input. Historic reason.
I got it worked with #type forward for output, So I assume it forward udp, according to the doc.
But then in logstash side I got some extra weird characters around the payload. logstash conf :
input {
tcp {
host => "127.0.0.1"
port => 41234
codec => "json"
}
}
How would you do that ? Is that even possible?
Related
We have a centralized rsyslog infrastructure capturing events from TCP sent by devices around the world using imtcp module.
The idea is to read from syslog (TCP) and store the events to disk, one line per event. The events are later processed by other consumers.
As far as we can see, some events are splitted in multiple events once they are stored on the disk breaking the rest of our process.
Capturing one single package with tcpdump, we confirmed that the source syslog is sending us the whole event containing multiple lines (typical java exceptions).
[root#xx xx.xx.xx.xx]# tcpdump -i bond0 tcp port 50520 -A -c 1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:12:26.062110 IP xx.xx.xx.xx.com.41444 > xx.xx.xx.com.50520: Flags [P.], seq 3270590174:3270590613, ack 2646946316, win 27, options [nop,nop,TS val 3937801207 ecr 2623497312], length 439
E....`#.<.ML..A....N...X..>...2......q.....
....._d`<13> xxx #2.0.#2021 02 10 12:19:50:898#+00#Info#com.xx.xx.xx.xx.xx#
##JavaEE/xx#xx#xx#JavaEE/xx#com.xx.xx.xx.xx.APIServiceHandler#xx#xx##xx#xx##0#Thread[HTTP Worker [#xx],5,Dedicated_Application_Thread]#Plain##
Is the user getting thru SSO? xx:true#
1 packet captured
44 packets received by filter
2 packets dropped by kernel
As this is a global system, we cannot request the device owners to modify the format, all the actions should take place on our side.
This is our rsyslog.conf file
$MaxMessageSize 128k
# Global configuration/modules
module(load="imtcp" MaxListeners="100")
module(load="imfile" mode="inotify")
module(load="impstats" interval="10" resetCounters="on" format="cee" ruleset="monitoring")
module(load="mmjsonparse")
module(load="mmsequence")
module(load="omelasticsearch")
module(load="omudpspoof")
# Include all conf files
$IncludeConfig /etc/rsyslog.d/*.conf
And this is our template that reads from tcp and writes to file (etc/rsyslog.d/template.conf)
template(name="outjsonfmt_device" type="list") {
constant(value="{")
property(outname="device_ip" name="fromhost-ip" format="jsonf")
constant(value=",")
property(outname="time_collect" name="timegenerated" dateFormat="rfc3339" format="jsonf")
constant(value=",")
constant(value="\"device_type\":\"device\"")
constant(value=",")
property(outname="collector_id" name="$myhostname" format="jsonf")
constant(value=",")
property(outname="msg" name="rawmsg-after-pri" format="jsonf" )
constant(value="}\n")
}
template(name="device-out-filename" type="string" string="/data1/input/device/%fromhost-ip%/device_%$now-utc%_%$hour-utc%.log")
ruleset(name="writeRemoteDataToFile_device") {
action(type="omfile" dynaFileCacheSize="10000" dirCreateMode="0700" FileCreateMode="0644" dirOwner="user" dirGroup="logstash" fileOwner="user" fileGroup="user" dynafile="device-out-filename" template="outjsonfmt_device")
}
input(type="imtcp" port="50520" ruleset="writeRemoteDataToFile_device")
How can we configure rsyslog to escape line breaks in the middle of an event, prior to write the event to disk? We already tried $EscapeControlCharactersOnReceive with no success and other similar parameters
The imtcp has a module parameter DisableLFDelimiter which you could try setting to on to ignore line-feed delimiters, assuming your input has an octet-count header. The page says, This mode is non-standard and will probably come with a lot of problems.
module(load="imtcp" MaxListeners="100" DisableLFDelimiter="on")
I am finding the following problem with WSO2 API Manager (I am working on a WSO2 API Manager 6.2.0 version).
I published an API on the publisher portal. This API expects a POST request over HTTPS protocol and take a JSON payload as the request body.
The problem is that the published API generates a cURL request like this:
curl -k -X POST "https://MY_SERVER:8243/puntualitest/v1.0.0/puntuali" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"header\": { \"msgUid\": \"a36bea3f-6dc6-49d7-9376-XXXXXXX\", \"timestamp\": 1567060509108, \"metadata\": { \"TRACKER_BIZID_REV_CODICE\": \"7175\", \"TRACKER_BIZID_REV_NUMERO\": \"545/2019\" }, \"codApplication\": null, \"codEnte\": null, \"invocationContext\": null, \"caller\": \"SRW\", \"user\": null, \"service\": \"crediti.invioPosizioneCreditoria\" }, \"body\": { \"#dto\": \"binary\", \"content\": \"PD94bWwgdmVyc2lvbj..............+\" }}"
The problem is that, as you can see, it is trying to call the final endpoint on the 8243 port instead of on the expected 443 port that is the standard HTTPS port.
So this call ends into a Connection refused message error because the final endpoint listen on the 443 and not on the 8243.
So I have some questions:
Why WSO2 API Manager try to call the final endpoint on the 8243 and not on the 443 port if the chosen protocol is HTTPS that, in theory, works on the 443?
Is there a way to set the final endpoint port that has to be used is 443?
Thank you
If you take a production case, your gateways are exposed via a load balancer and it is exposed in the port 443. From lb you can load balance to 8243 gateway servers. You can refer [1] for setting up the load balancer.
If you want, you an change the ports 8280 and 8243. For that you can change those in axis2.xml file which resides in repository/conf/axis2 location.
<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
<parameter name="port" locked="false">8280</parameter>
<parameter name="non-blocking" locked="false">true</parameter>
<!--parameter name="bind-address" locked="false">hostname or IP address</parameter-->
<!--parameter name="WSDLEPRPrefix" locked="false">https://apachehost:port/somepath</parameter-->
<parameter name="httpGetProcessor" locked="false">org.wso2.carbon.mediation.transport.handlers.PassThroughNHttpGetProcessor</parameter>
<!--<parameter name="priorityConfigFile" locked="false">location of priority configuration file</parameter>-->
</transportReceiver>
In the port value, you can change this to 80 and a similar definition can be found for https for the port 8243.
[1] - https://docs.wso2.com/display/AM210/Configuring+the+Proxy+Server+and+the+Load+Balancer#ConfiguringtheProxyServerandtheLoadBalancer-Step2-Configuretheloadbalancer/reverseproxyserver
I'm trying to use fluent-bit to ship logs from log files to telegraf which's listening on a port 8094. I'm able to send data to this port via terminal like this
echo "some_log_data" | nc localhost 8094
but when I'm using fluent-bit formward output plugin to send data to the same port, it's giving this error in the fluent-bit logs
fluent-bit_1 | [2019/11/21 11:14:44] [error] [io] TCP connection failed: localhost:8094 (Connection refused)
fluent-bit_1 | [2019/11/21 11:14:44] [error] [out_fw] no upstream connections available
This's my docker-compose file:
version: '3'
services:
# Define a Telegraf service
telegraf:
image: telegraf
volumes:
- ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
ports:
- "8092:8092/udp"
- "8094:8094"
- "8125:8125/udp"
- "9126:9126"
networks:
- mynet
fluent-bit:
image: fluent/fluent-bit:1.3.2-debug
volumes:
- ./fluent-bit:/fluent-bit/etc
- ./access_logs/localhost_access_log:/logs
depends_on:
- telegraf
networks:
- mynet
networks:
mynet:
fluent-bit.conf:
[SERVICE]
Flush 2
Parsers_File parsers.conf
[INPUT]
Name tail
Tag cuic.logs
Path /logs/*.log
Path_Key File_Path
Multiline On
Parser_Firstline start
[OUTPUT]
Name forward
Match *
Host localhost
Port 8094
Tag cuic.logs
telegraf.conf:
[[outputs.file]]
files = ["/tmp/metrics.out"]
data_format = "json"
json_timestamp_units = "1s"
[[inputs.socket_listener]]
service_address = "tcp://:8094"
socket_mode = "777"
data_format = "grok"
grok_patterns = ["%{CUSTOM_LOG}"]
grok_custom_patterns = '''
SOME_GROK_PATTERN
'''
[[aggregators.histogram]]
period = "10s"
drop_original = false
[[aggregators.histogram.config]]
buckets = [0.0, 10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0, 100.0]
measurement_name = "access_log"
fields = ["resp_time"]
Can someone please help me figure out what I did wrong?
I think the problem is when using the hostname "localhost". For the container localhost at a network level will be its own network scope, it won't be able to access the other container TCP Port as desired.
You can read more about the same problem here:
How to share localhost between two different Docker containers?
and... note that Forward output protocol in Fluent Bit uses a binary protocol as opposed to the normal JSON that I suspect you want to use. Use the tcp output plugin instead.
It's definitely the output/input plugins you are using. Telegraf has a FluentD plugin here, and it looks like this:
# Read metrics exposed by fluentd in_monitor plugin
[[inputs.fluentd]]
## This plugin reads information exposed by fluentd (using /api/plugins.json endpoint).
##
## Endpoint:
## - only one URI is allowed
## - https is not supported
endpoint = "http://localhost:24220/api/plugins.json"
## Define which plugins have to be excluded (based on "type" field - e.g. monitor_agent)
exclude = [
"monitor_agent",
"dummy",
]
Your Fluent-Bit http output config would look like this:
[INPUT]
Name cpu
Tag cpu
[OUTPUT]
Name http
Match *
Host 192.168.2.3
Port 80
URI /something
But Fluent-Bit also has an InfluxDB output plugin.
Fluent-bit has an out plugin named forward, it can forward the output according to fluentd protocol. You can set up it according to this doc: https://docs.fluentbit.io/manual/pipeline/inputs/forward
Then, you can find the telegraf has an input plugin named fluentd, set it as input and then gather metrics from the fluentd client endpoint which can fulfill your requirement.
IP Address:
Web Application -> 192.168.x.209
HAProxy -> 192.168.x.211
Logstash Shipper -> 192.168.x.210
With the below configuration, HAProxy is not able to receive logs from Logstash application and thus logstash shipper is not able to receive the logs.
Following are the configurations I did:
WebApplication - logback.xml
<appender name="stash"
class="net.logstash.logback.appender.LogstashAccessTcpSocketAppender">
<destination>192.168.x.211:5001</destination>
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
HAProxy Config
listen logstash
bind :5001
mode tcp
balance leastconn
option tcplog
server logstash01 192.168.x.212:5001 check
server logstash02 192.168.x.210:5001 check
Logstash Shipper - config
input{
tcp{
type => "log1"
port => 5001
codec => "json"
}
tcp{
type => "log2"
port => 5002
codec => "json"
}
output {
stdout{
codec => rubydebug
}}
Is there anything wrong with the configuration? help required.
Found the issue.
This was because firewall rules were active. After shutting it down, things worked as expected.
service iptables stop
setenv 0
I am setting up simple tcp connection routing using HAProxy acl's. The idea is to route connections depending on request content having two flavors: read and write requests.
For testing I made a simple tcp client/server setup using perl. Strangely enough about 10-40% of the ACL's fail and are sent to the default backend.
The ACL's should find the substring 'read' or 'write' and route accordingly, but this is not allways the case.
Sending a read/write request using nc (netcat) has the same effect.
I tested this configuration with mode=http and everything works as expected.
I also tested with reg, sub and bin, to no avail.
The example server setup is as follows:
HAProxy instance, listens on port 8000
Client (creates tcp connection to proxy and sends user input (read/write string) to server through port 8000, after which it closes the connection)
Server1 (write server), listens on port 8001
Server2 (read server), listens on port 8002
Server3 (default server), listens on port 8003
My HAProxy configuration file looks is:
global
log /dev/log local0 debug
#daemon
maxconn 32
defaults
log global
balance roundrobin
mode tcp
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend tcp-in
bind *:8000
tcp-request inspect-delay 3s
acl read req.payload(0,4) -m sub read
acl write req.payload(0,5) -m sub write
use_backend read_servers if read
use_backend write_server if write
default_backend testFault
backend write_server
server server1 127.0.0.1:8001 maxconn 32
backend read_servers
server server2 127.0.0.1:8002 maxconn 32
backend testFault
server server3 127.0.0.1:8003 maxconn 32
The client code (in perl):
use IO::Socket::INET;
# auto-flush on socket
#$| = 1;
print "connecting to the server\n";
while(<STDIN>){
# create a connecting socket
my $socket = new IO::Socket::INET (
PeerHost => 'localhost',
PeerPort => '8000',
Proto => 'tcp',
);
die "cannot connect to the server $!\n" unless $socket;
# data to send to a server
$req = $_;
chomp $req;
$size = $socket->send($req);
print "sent data of length $size\n";
# notify server that request has been sent
shutdown($socket, 1);
# receive a response of up to 1024 characters from server
$response = "";
$socket->recv($response, 1024);
print "received response: $response\n";
$socket->close();
}
The server (perl code):
use IO::Socket::INET;
if(!$ARGV[0]){
die("Usage; specify a port..");
}
# auto-flush on socket
$| = 1;
# creating a listening socket
my $socket = new IO::Socket::INET (
LocalHost => '0.0.0.0',
LocalPort => $ARGV[0],
Proto => 'tcp',
Listen => 5,
Reuse => 0
);
die "cannot create socket $!\n" unless $socket;
print "server waiting for client connection on port $ARGV[0]\n";
while(1){
# waiting for a new client connection
my $client_socket = $socket->accept();
# get information about a newly connected client
my $client_address = $client_socket->peerhost();
my $client_port = $client_socket->peerport();
print "connection from $client_address:$client_port\n";
# read up to 1024 characters from the connected client
my $data = "";
$client_socket->recv($data, 1024);
print "received data: $data\n";
# write response data to the connected client
$data = "ok";
$client_socket->send($data);
# notify client that response has been sent
shutdown($client_socket, 1);
$client_socket->close();
print "Connection closed..\n\n";
}
$socket->close();
Binary data in haproxy is tricky. Probably some bug, but the following worked for me on haproxy 1.7.9.
I am trying to build a thrift proxy server which can route to appropriate backend based on the user_id in the payload.
frontend thriftrouter
bind *:10090
mode tcp
option tcplog
log global
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq captured_user:%[capture.req.hdr(0)] req.len:%[capture.req.hdr(1)]"
tcp-request inspect-delay 100ms
tcp-request content capture req.payload(52,10) len 10
tcp-request content capture req.len len 10
tcp-request content accept if WAIT_END
acl acl_thrift_call req.payload(2,2) -m bin 0001 # Thrift CALL method
acl acl_magic_field_id req.payload(30,2) -m bin 270f # Magic field number 9999
# Define access control list for each user
acl acl_user_u1 req.payload(52,10) -m sub |user1|
acl acl_user_u2 req.payload(52,10) -m sub |user2|
# Route based on the user. No default backend so that one always has to set it
use_backend backend_1 if acl_user_u1 acl_magic_field_id acl_thrift_call
use_backend backend_2 if acl_user_u2 acl_magic_field_id acl_thrift_call
When matching binary data in acl, make sure you're looking at the right number of bytes, for substring to work properly. Or use the hex conversion method and match on hex bytes.
Dont I feel silly. Re-reading the HAProxy documentation I found the following directive (fetch method) that fixes the issue:
tcp-request content accept if WAIT_END
That solved the unexpected behaviour.