Rsyslog parsing issue - no header in syslog message that is sent to Rsyslog - syslog

I have an applience which sends logs to an Rsyslog server over TCP as JSON Stream. however it doesn't put any header in the syslog message. The sample raw message that is sent is below:
{"query_class":"C_INTERNET","source_ip":"10.1.1.1","query_type":"A","trans_id":502,"err_code":"NXDOMAIN","#type":"dns","dest_ip":"10.10.10.1","dest_port":53,"uid":"CDdCwH1lD2ToFS5y02","epochdate":1595000476.3491,"query":"www.google.com","#host":"host-10-01","rejected":true,"source_port":54764,"proto":"udp"}
Rsyslog forwards this message to another server and puts a header in the format like below(there is no template applied):
<13>Jun 16 10:43:09 host01.example.local
I don't know where this header comes from(maybe a default config is applied when there is no header in the message).
The problem is, the TAG is missing in the header and it causes the raw message be parsed improperly.
When Rsyslog forwards the logs to another machine or writes it to a file, this is how it looks:
<13>Jun 16 10:43:09 host01.example.local {"query_class" "C_INTERNET","source_ip":"10.1.1.1","query_type":"A","trans_id":502,"err_code":"NXDOMAIN","#type":"dns","dest_ip":"10.10.10.1","dest_port":53,"uid":"CDdCwH1lD2ToFS5y02","epochdate":1595000476.3491,"query":"www.google.com","#host":"host-10-01","rejected":true,"source_port":54764,"proto":"udp"}
{"query_class" gets extraced and becomes the TAG (":" is removed). The rest becomes the syslog message.
Unfortunately, it is not possible to make any change at the source to configure a header.
How can I add a header to this raw message in Rsyslog and forward it to another remote machine or write it to a file? Or, how can I change the default header config to add a custom processname tag(if there is default config applied)?

rsyslog is probably using the RSYSLOG_TraditionalForwardFormat default template for omfwd which is:
template(name="RSYSLOG_TraditionalForwardFormat" type="string"
string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32% %msg:::sp-if-no-1st-sp%%msg%")
Define your own template and use it in your forwarding rule, eg:
template(name="myfwd" type="string"
string="<%PRI%>%TIMESTAMP% %HOSTNAME% dummytag %rawmsg%\n")
if $fromhost == 'thehost' then {
action(type="omfwd" target="theserver" protocol="tcp" template="myfwd")
}

Related

How to release request_body->buffs after I read them

I write a Nginx module, the client will post stream data to it. But I found the client will be blocked after send about 64KB data(http body's size). I try to change the Nginx config client_body_buffer_size, after that the client can send more data.
r->request_body_no_buffering = 1; ngx_http_read_client_request_body(r, body_handler);
Do I have to do something in the body_handler function which release the request_body->buffs?
I try to use ngx_free_chain, but it doesn't work.

syslog-ng return original value when mapping does not exists

i'm using syslog-ng for collecting json messages and send alarms to slack
there is parameter in json message which contains IP address of router from which I'm receiving the json message and I want to convert IP address to router hostname
i'm using two parsers 1) parse json 2) replace IP address with router hostname:
parser p_json {
json-parser(prefix(".json."));
};
parser p_acd_router {
add-contextual-data(selector("${.json.router_ip}"), database("host_map.csv"), prefix(".meta_router."));
};
until now everything works fine.
Hi,
I'm having the problem when I receive log message from router and I don't have IP_to_Hostname mapping for it in csv file. in this case i just get blank space in syslog message. there is option to return default value if mapping does not exist, but i would prefer to return original value (IP address)
if mapping exists convert IP address to hostname >> works
if mapping does not exist, return original IP address >> don't know how to set it up
is this even supported?
thanks
You can set the default-selector("UNKNOWN") option for add-contextual-data(), and add a record to your CSV file with the ID UNKNOWN, and use the following value when setting .meta_router.hostname: ${.json.router_ip}.
TLDR: templates are supported inside the CSV file as well.
Note: In case your IPs are reverse-resolvable, you can just use the $(dns-resolve-ip) template function instead of maintaining a complete CSV database:
https://github.com/syslog-ng/syslog-ng/pull/3046

Not able to send data on reply channel using tcp-outbound-gateway with ActiveMQ

My Requirement is to get data from TCP, after getting the data from TCP convert into java object and send on ActiveMQ. Post that after doing some processing need to send the acknowledgement/response code on same channel on TCP.
To fulfill this requirement I am using tcp-outbound-gateway as bidirectional communication is required.
Problem is I am not able to send acknowledgement with ActiveMQ. If I comment out ActiveMQ part and write a dummy string on replyChannel it is visible, but the moment I send the object on Active MQ queue it is giving me a message "null reply received for nothing to send".
I am using a new queue to get the acknowledgement and trying to put the response on reply channel of tcp-outbound-gate, but error message is no output-channel or replyChannel header available.
I got the MessageHeaders details via Incoming message and sending it via queue to use copyHeader. I am able to set the headers and see paylod in Message object, verified the same by applying Interceptos on reply channel, but still getting the same error no output-channel or replyChannel header available.
Code is :
<int:gateway id="gw" default-reply-channel="replyChannel" default-reply-timeout="10000" service-interface= "com.telnet.core.integration.connection.ParseTcpConfiguration$SimpleGateway"
default-request-channel="${server.inboundChannel}"/>
<int:channel id="telnetLandingChannel" />
<ip:tcp-connection-factory id="serverFactory" type="server" host="${server.host}" port="${server.port}" single-use="false"
serializer="${server.serializer}" deserializer="${server.serializer}" task-executor="serverFactoryTaskExecutor"/>
<ip:tcp-inbound-gateway id="serverInboundAdpater" request-channel="telnetLandingChannel" reply-channel="replyChannel"
connection-factory="serverFactory" error-channel="errorChannel" reply-timeout="1000000" auto-startup="false"/>
<int:channel id="replyChannel"></int:channel>
<beans:bean id="acknowledgementHandler" class= "com.telnet.core.integration.AcknowledgementHandler">
</beans:bean>
<int:channel id="incidentDispatchMessageChannel" datatype="${incident.interaction.dispatch.response.datatype}"></int:channel>
<int-jms:message-driven-channel-adapter id="incidentDispatchMessageChannelAdapter" error-channel="errorChannel"
connection-factory="mqConnectionFactory"
destination-name="${incident.processing.tcp.dispatch.response.queues}"
channel="incidentDispatchMessageChannel"/>
<int:transformer id="incidentMessageActivator"
input-channel="incidentDispatchMessageChannel"
output-channel="replyChannel"
ref="acknowledgementHandler" method="incidentAck">
</int:transformer>
public Message incidentAck(final DefaultIncidentAcknowledgeMessage defaultIncidentAcknowledgeMessage){
MessageHeaders ms = (MessageHeaders)defaultIncidentAcknowledgeMessage.getProperties().get("MessageHeader");
Message<String> message = MessageBuilder.withPayload("1").copyHeaders(ms).build();
return message;
}
Need to see your Integration configuration though, but let me guess that you are loosing TemporaryReplyChannel object in the replyChannel header because it isn't Serializable. Consider to use:
<int:header-enricher>
<int:header-channels-to-string/>
</int:header-enricher>
somewhere before sending to the ActveMQ.
See Reference Manual for more information.
UPDATE
Looks like this is a continuation of Receive the acknowledgement from TCP server to our application using spring Integration. And I see you still use the same replyChannel for many places. That isn't going to work properly. The replyChannel header from the gateway can accept only one reply. Even if we figure out what to do with the reply from ActiveMQ, the TemporaryReplyChannel will be fulfilled with the reply from the TCP Outbound Gateway.
If I understand you correctly, alongside with the reply from the TCP you need also get some message from ActiveMQ. And send everything as a reply to the gateway call. For this purpose I suggest you to consider to use Aggregator and figure out some custom correlation strategy to match the reply from TCP to that acknowledge from the ActiveMQ. After aggregation you really can just use the existing replyChannel header to reply to the gateway.

How can I remove the Server header in Payara Micro 172?

I'd like to remove the server header from Payara Micro's output.
For example, it reports this:
HTTP/1.1 200 OK
Server: Payara Micro #badassfish
I'd like to get rid of that Server line.
I see that issue 32 provided the capability in theory to disable this. The associated pull request certainly seems to show that there is some sort of property being consulted to disable this. And I can infer from this that I could presumably go into the administration GUI if this were a full-fledged server and click a checkbox somewhere.
But I'm running Payara Micro which needs to be fully configured from the command line at startup.
What I'm lacking is a simple "put this on the command line"-type of instruction and I can't seem to locate that anywhere. Is there a setting in, say, glassfish-web.xml-as-modified-by-Payara I could use? Or a command line switch? Or a specific asadmin attribute I could set in a pre-boot script?
Ideas?
The change that you referenced linked to a pull request which added a boolean server-header property to the http-listener which would disable the header.
There is no native asadmin command to set this property, so you will need to use an asadmin set command with the correct dotted name for the listener you want to modify. To find out what this is, you can use the get command with a wildcard and grep for the value you want, as shown below with Payara Server:
➜ ~ /opt/payara/server/171.1/bin/asadmin get "*" | grep server-header
configs.config.default-config.network-config.protocols.protocol.http-listener-2.http.server-header=true
configs.config.default-config.network-config.protocols.protocol.http-listener-1.http.server-header=true
configs.config.default-config.network-config.protocols.protocol.admin-listener.http.server-header=true
configs.config.server-config.network-config.protocols.protocol.admin-listener.http.server-header=true
configs.config.default-config.network-config.protocols.protocol.sec-admin-listener.http.server-header=true
configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.server-header=true
configs.config.server-config.network-config.protocols.protocol.http-listener-2.http.server-header=true
Since the default-config is just a template and not used, we want the listeners from server-config. http-listener-1 is for HTTP by default and http-listener-2 is for HTTPS by default. To modify the server-header property in Payara Micro, you would need to create a file with the following command in (note that Payara Micro only has a single listener by default called http-listener):
set configs.config.server-config.network-config.protocols.protocol.http-listener.http.server-header=false
You can then apply these with a prebootcommandfile as follows:
java -jar /opt/payara/micro/173/payara-micro.jar --prebootcommandfile myCommands.txt
You may also wish to disable the xpowered-by property via the same method.

Need to archive received emails

I have a mail receiving setup where in Postfix (2.6.6) is the MTA and then I have amavisd-new (with spamassassin and CLamAV) as content filter.
I have enabled spam report header in my amavisd-new conf file.
I want to archive the files on local storage in maildir format. Is it possible via amavisd-new conf file?
In effect I want my archives to be created in following format or similar to one for each email received:
<BaseArchiveDirectory>/<UserEmailID>/<subjectOfEmail>.eml
Thanks
Ashish Sharma
I used the following setting in my amavisd-new conf file:
$archive_quarantine_method = 'smtp:[127.0.0.1]:10050';
$archive_quarantine_to = 'archive-quarantine';
#archive_quarantine_to_maps = (\$archive_quarantine_to);
$mailfrom_to_quarantine = undef; # undef keeps original sender
via above settings I could redirect a duplicate email stream to my archiving process (that I coded myself) listening on port 10050.
Above settings don't interfere in the basic path of the email content filter flows.

Resources