I'm trying to send logs into datadog using rsyslog. Ideally, I'm trying to do this without having the logs stored on the server hosting rsyslog. I've run into an error in my config that I haven't been able to find out much about. The error occurs on startup of rsyslog.
omfwd: could not get addrinfo for hostname '(null)':'(null)': Name or service not known [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Here's the portion I've added into the default rsyslog.config
module(load="imudp")
input(type="imudp" port="514" ruleset="datadog")
ruleset(name="datadog"){
action(
type="omfwd"
action.resumeRetryCount="-1"
queue.type="linkedList"
queue.saveOnShutdown="on"
queue.maxDiskSpace="1g"
queue.fileName="fwdRule1"
)
$template DatadogFormat,"00000000000000000 <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n "
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.logs.datadoghq.com
*.* ##intake.logs.datadoghq.com:10516;DatadogFormat
}
First things first.
The module imudp enables log reception over udp.
The module omfwd enables log forwarding over (tcp, udp, ...)
So most probably - or atleast as far as i can tell - with rsyslog you just want to log messages locally and then send them to datadog.
I don't know anything about the $ActionSendStreamDriver tags, so I can't help you there. But what is jumping out is, that in your action you haven't defined where the logs should be sent to.
ruleset(name="datadog"){
action(
type="omfwd"
target="10.100.1.1"
port="514"
protocol="udp"
...
)
...
}
I'm running a flask_restful API service that is being forwarded traffic via an nginx proxy. While the IP address is being forward through the proxy via some variables, flask_restful doesn't seem to be able to see these variables, as indicated by its output which points to 127.0.0.1:
127.0.0.1 - - [25/Oct/2017 21:55:37] "HEAD sne/event/SN2014J/photometry HTTP/1.0" 200 -
While I know I can retrieve the IP address via the request object (nginx forwards X-Forwarded-For and X-Real-IP), I don't know how to make the above output from flask_restful show/use this IP address, which is important if you want to say limit the number of API calls from a given IP address with flask_limiter. Any way to make this happen?
You can use (for older version of werkzeug)
from werkzeug.contrib.fixers import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app)
For newer version of werkzeug (1.0.0+)
from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app)
This will fix the IP using X-Forwarded-For. If you need a enhanced version you case use
class SaferProxyFix(object):
"""This middleware can be applied to add HTTP proxy support to an
application that was not designed with HTTP proxies in mind. It
sets `REMOTE_ADDR`, `HTTP_HOST` from `X-Forwarded` headers.
If you have more than one proxy server in front of your app, set
num_proxy_servers accordingly
Do not use this middleware in non-proxy setups for security reasons.
get_remote_addr will raise an exception if it sees a request that
does not seem to have enough proxy servers behind it so long as
detect_misconfiguration is True.
The original values of `REMOTE_ADDR` and `HTTP_HOST` are stored in
the WSGI environment as `werkzeug.proxy_fix.orig_remote_addr` and
`werkzeug.proxy_fix.orig_http_host`.
:param app: the WSGI application
"""
def __init__(self, app, num_proxy_servers=1, detect_misconfiguration=False):
self.app = app
self.num_proxy_servers = num_proxy_servers
self.detect_misconfiguration = detect_misconfiguration
def get_remote_addr(self, forwarded_for):
"""Selects the new remote addr from the given list of ips in
X-Forwarded-For. By default the last one is picked. Specify
num_proxy_servers=2 to pick the second to last one, and so on.
"""
if self.detect_misconfiguration and not forwarded_for:
raise Exception("SaferProxyFix did not detect a proxy server. Do not use this fixer if you are not behind a proxy.")
if self.detect_misconfiguration and len(forwarded_for) < self.num_proxy_servers:
raise Exception("SaferProxyFix did not detect enough proxy servers. Check your num_proxy_servers setting.")
if forwarded_for and len(forwarded_for) >= self.num_proxy_servers:
return forwarded_for[-1 * self.num_proxy_servers]
def __call__(self, environ, start_response):
getter = environ.get
forwarded_proto = getter('HTTP_X_FORWARDED_PROTO', '')
forwarded_for = getter('HTTP_X_FORWARDED_FOR', '').split(',')
forwarded_host = getter('HTTP_X_FORWARDED_HOST', '')
environ.update({
'werkzeug.proxy_fix.orig_wsgi_url_scheme': getter('wsgi.url_scheme'),
'werkzeug.proxy_fix.orig_remote_addr': getter('REMOTE_ADDR'),
'werkzeug.proxy_fix.orig_http_host': getter('HTTP_HOST')
})
forwarded_for = [x for x in [x.strip() for x in forwarded_for] if x]
remote_addr = self.get_remote_addr(forwarded_for)
if remote_addr is not None:
environ['REMOTE_ADDR'] = remote_addr
if forwarded_host:
environ['HTTP_HOST'] = forwarded_host
if forwarded_proto:
environ['wsgi.url_scheme'] = forwarded_proto
return self.app(environ, start_response)
from saferproxyfix import SaferProxyFix
app.wsgi_app = SaferProxyFix(app.wsgi_app)
PS: Code taken from http://esd.io/blog/flask-apps-heroku-real-ip-spoofing.html
Can we made a network with multiple switches without any controller in mininet .
I mean that we manually control the switches by "dpctl"
I made the topology like this:
s2
/ \
h1---s1/ \s3--h2
\ /
\s4/
and I want to send from h1 to h2.
first of all that I turn on the floodlight, and I extract the switch port number from there. after that I want to add flows manually that I encountered to an error.
mininet#mininet-vm:~$ dpctl add-flow tcp:127.0.0.1:40566 in_port=1,actions=output:2
dpctl: failed to send packet to switch: Connection refused
how can i fix it??
thanks a lot
dpctl add-flow tcp:127.0.0.1:6634in_port=1,actions=output:2
Port number will be 6634.
Controller is at Port number 6633 and from 6634 you have port for dpctl.
My issue is as follows: I want to implement a listen service using scapy to stimulate a honeypot (because honeypot uses a fake ip, so I can't use OS sockets) and I chose scapy.
I implemented a very simple TCP hand-shake procedure, however one thing frustrated me: one byte of the packet I use PSH to send is eaten.
For example I send "abc" out to a client, but the client's socket, for example netcat or wget, only receive "bc". Another example is "HTTP/1.1 200 OK" becomes "TTP/1.1 200 OK". I captured packet and wireshark can correctly recognize my hand-made packet as HTTP, but the client socket just lack 1 byte. I don't know why.
The code is as follows:
192.168.1.100 stands for server(my) ip addr,9999 is the port. For example, I run this python script on 192.168.1.100, then I use "nc 192.168.1.100 9999". I expect to get "abc", but I can only get "bc", but the packet seems no problem in Wireshark. it's so strange.
'''
Created on Jun 2, 2012
#author: root
'''
from scapy import all
from scapy.layers.inet import IP, ICMP, TCP
from scapy.packet import ls, Raw
from scapy.sendrecv import sniff, send
from scipy.signal.signaltools import lfilter
import scapy.all
HOSTADDR = "192.168.1.100"
TCPPORT = 9999 'port to listen for'
SEQ_NUM = 100
ADD_TEST = "abc"
def tcp_monitor_callback(pkt):
global SEQ_NUM
global TCPPORT
if(pkt.payload.payload.flags == 2):
'A syn situation, 2 for SYN'
print("tcp incoming connection")
ACK=TCP(sport=TCPPORT, dport=pkt.payload.payload.sport, flags="SA",ack=pkt.payload.payload.seq + 1,seq=0)
send(IP(src=pkt.payload.dst,dst=pkt.payload.src)/ACK)
if(pkt.payload.payload.flags & 8 !=0):
'accept push from client, 8 for PSH flag'
print("tcp push connection")
pushLen = len(pkt.payload.payload.load)
httpPart=TCP(sport=TCPPORT, dport=pkt.payload.payload.sport, flags="PA", ack=pkt.payload.payload.seq + pushLen)/Raw(load=ADD_TEST)
'PROBLEM HERE!!!! If I send out abc, the client socket only receive bc, one byte disappers!!!But the packet received by client is CORRECT'
send(IP(src=pkt.payload.dst,dst=pkt.payload.src)/httpPart)
if(pkt.payload.payload.flags & 1 !=0):
'accept fin from cilent'
print ("tcp fin connection")
FIN=TCP(sport=TCPPORT, dport=pkt.payload.payload.sport, flags="FA", ack=pkt.payload.payload.seq +1, seq = pkt.payload.payload.ack)
send(IP(src=pkt.payload.dst,dst=pkt.payload.src)/FIN)
def dispatcher_callback(pkt):
print "packet incoming"
global HOSTADDR
global TCPPORT
if(pkt.haslayer(TCP) and (pkt.payload.dst == HOSTADDR) and (pkt.payload.dport == TCPPORT)):
tcp_monitor_callback(pkt)
else:
return
if __name__ == '__main__':
print "HoneyPot listen Module Test"
scapy.all.conf.iface = "eth0"
sniff(filter=("(tcp dst port %s) and dst host %s") % (TCPPORT,HOSTADDR), prn=dispatcher_callback)
Some suggestions:
Sniff may append some payload to the end of the packet, so len(pkt.payload.payload.load) may not be the real payload length. You can use pkt[IP].len-40 (40 is the common header length of IP+TCP). You may also use -len(pkt[IP].options)-len(pkt[TCP].options) for more accurate results.
Usually the application layer above TCP uses line breaks ("\r\n") to separate commands, so you'd better change ADD_TEST to "abc\r\n"
If none of above methods work, you may upgrade to the latest netcat and try again.
I tested your code, you are missing sending proper tcp sequence
httpPart=TCP(sport=TCPPORT, dport=pkt.payload.payload.sport, flags="PA", ack=pkt.payload.payload.seq + pushLen, seq=pkt.payload.payload.ack)/Raw(load=ADD_TEST)
should fix the issue, you may have other packet length issue, but the eaten 1 byte is caused by missing proper tcp sequence
How do you use gen_udp in Erlang to do multicasting? I know its in the code, there is just no documentation behind it. Sending out data is obvious and simple. I was wondering on how to add memberships. Not only adding memberships at start-up, but adding memberships while running would be useful too.
Here is example code on how to listen in on Bonjour / Zeroconf traffic.
-module(zcclient).
-export([open/2,start/0]).
-export([stop/1,receiver/0]).
open(Addr,Port) ->
{ok,S} = gen_udp:open(Port,[{reuseaddr,true}, {ip,Addr}, {multicast_ttl,4}, {multicast_loop,false}, binary]),
inet:setopts(S,[{add_membership,{Addr,{0,0,0,0}}}]),
S.
close(S) -> gen_udp:close(S).
start() ->
S=open({224,0,0,251},5353),
Pid=spawn(?MODULE,receiver,[]),
gen_udp:controlling_process(S,Pid),
{S,Pid}.
stop({S,Pid}) ->
close(S),
Pid ! stop.
receiver() ->
receive
{udp, _Socket, IP, InPortNo, Packet} ->
io:format("~n~nFrom: ~p~nPort: ~p~nData: ~p~n",[IP,InPortNo,inet_dns:decode(Packet)]),
receiver();
stop -> true;
AnythingElse -> io:format("RECEIVED: ~p~n",[AnythingElse]),
receiver()
end.
Multicast sending has been answered, receipt requires subscription to the multicast group.
It (still) seems undocumented, but has been covered on the erlang-questions mailing list before. http://www.erlang.org/pipermail/erlang-questions/2003-March/008071.html
{ok, Socket} = gen_udp:open(Port, [binary, {active, false},
{reuseaddr, true},{ip, Addr},
{add_membership, {Addr, LAddr}}]).
where the Addr is the multicast group, and LAddr is a local interface. (code courtesy of mog)
The same options used above can be passed to inet:setopts including {drop_membership, {Addr, LAddr}} to stop listening to the group.
I try to get this example running on my PC. What could happen, if I get always the message {error,eaddrnotavail} by opening the receive socket?
Example 1: This works:
{ok, Socket} = gen_udp:open(?PORT, [{reuseaddr,true}, {ip,?SERVER_IP},
{multicast_ttl,4}, {multicast_loop,false}, binary]),
Example 2: Getting an runtime Error:
{ok, Socket} = gen_udp:open(?PORT, [{reuseaddr,true}, {ip,?MULTICAST_IP},
{multicast_ttl,4}, {multicast_loop,false}, binary]),
% --> {error,eaddrnotavail}
-define(SERVER_IP, {10,31,123,123}). % The IP of the current computer
-define(PORT, 5353).
-define(MULTICAST_IP, {224,0,0,251}).
Multicast is specified by IP Address
It's the same in erlang as for all languages. The IP addresses 224.0.0.0 through 239.255.255.255 are multicast addresses.
Pick an address in that range, check that you're not overlapping an already assigned address, and you are good to go.
http://www.iana.org/assignments/multicast-addresses