I have configured ELK-stack (Elasticsearch, Logstash, and Kibana) cluster for centralized logging system with Filebeat. Now I have been asked to reconfigure to EFK (Elasticsearch, FluentD, and Kibana) with Filebeat. I have disabled the Logstash and Installed FluentD, But I'm not able to configure FluentD with Filebeat. I have installed FluentD plugin for Filebeat and modified /etc/td-agent/td-agent.conf, but it seems not working.
td-agent.conf
<source>
#type beats
tag record['#metadata']['beat']
port 5044
bind 0.0.0.0
</source>
<match *.**>
#type copy
<store>
##type file
#type elasticsearch_dynamic
logstash_format true
logstash_prefix ${tag_parts[0]}
type_name ${record['type']}
</store>
<store>
#type file
logstash_format true
logstash_prefix ${tag_parts[0]}
type_name ${record['type']}
path /var/log/td-agent/data_logs.*.log
</store>
</match>
A source is an input not an output in fluentd you would want a match with the corresponding fluentd tags that match to be shipped to filebeats then out to Elastic.
Related
I am oozie on Hue interface where i would like to get the email alerts for any killed/Failed/long runnig jobs.
Is there any way to get this.
Below is the component list:
Component Version
Hue 2.6.1
HDP 2.3.6
Hadoop 2.7.1
Oozie 4.2.0
Ambari 2.6.0
You can use the below action
<action name="ErrorHandler">
<email xmlns="uri:oozie:email-action:0.1">
<to>${notify}</to>
<subject>FAILURE - Automated email notification for process</subject>
<body>The upload process for <Process name> failed.</body>
</email>
<ok to="error"/>
<error to="error"/>
</action>
Few points to remember -
In job.properties, mention the required email address for the notify variable.
Set SMTP hostname and port in oozie-site.xml
I'm setting up a Service Fabric application which contains:
an Nginx instance as frontend (single instance, port 80)
some applications written with Asp.net core (1 website, some API services) (multiple instances, dynamic port)
a Gateway service for address resolution (single instance, port 8081)
For nginx, I'm using a solution available as Nuget package.
The gateway and, in general, the example to run .NET core app have been taken here
It is suggested by the .NET core team itself to host applications behind a real web server liken nginx.
Therefore I'd like to deploy my Service Fabric application with an instance of nginx as entry point, which redirects to the Gateway service, which will do the service resolution for the replicated stateless services.
My question is about the address that I need to use in the nginx.conf to point to the Gateway address. While trying locally, I can use the local address 127.0.0.1 and it works as expected, but what happens if on a real cluster my Nginx and Gateway instances are deployed to different machines?
This is my application manifest:
<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="SFApplicationType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
<Parameters>
<Parameter Name="NginxPoC_InstanceCount" DefaultValue="1" />
<Parameter Name="Gateway_InstanceCount" DefaultValue="1" />
</Parameters>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="NginxPoCPkg" ServiceManifestVersion="1.0.0" />
<Policies>
<RunAsPolicy CodePackageRef="Code" UserRef="Admin" EntryPointType="All" />
</Policies>
</ServiceManifestImport>
<ServiceManifestImport>
<ServiceManifestRef ServiceManifestName="Gateway" ServiceManifestVersion="1.0.0" />
</ServiceManifestImport>
<DefaultServices>
<Service Name="NginxPoC">
<StatelessService ServiceTypeName="NginxPoCType" InstanceCount="[NginxPoC_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
<Service Name="Gateway">
<StatelessService ServiceTypeName="GatewayType" InstanceCount="[Gateway_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
<Principals>
<Users>
<User Name="Admin">
<MemberOf>
<SystemGroup Name="Administrators" />
</MemberOf>
</User>
</Users>
</Principals>
</ApplicationManifest>
and this is my current nginx.conf file:
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://127.0.0.1:8081;
}
}
Update 2016-10-09
As requested in the discussion, I've created a test project here. Every contribute to the project is welcome.
f you deploy the nginx and gateway service to all nodes (InstanceCount = -1) you should be good. If the gateway service is down on one node, you would of course not be able to forward the request from nginx to a gateway service on another node. For this, you need the nginx service to look-up the gateway service.
You can get the service endpoint address for the gateway using a REST call: https://msdn.microsoft.com/en-us/library/azure/dn707638.aspx
How to use nxLog? I installed it on my windows 7 and unix box, but not able to use it.
My Conf File(not sure its correct or not):
define ROOT C:\Program Files\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
<Extension syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_file
File 'D:\dotnet\Analytics\nxLog\association.log'
SavePos TRUE
ReadFromLast TRUE
PollInterval 1
Exec $Message = $raw_event; $SyslogFacilityValue = 22;
</Input>
<Output out1>
Module om_udp
Host 10.1.1.1
Port 514
Exec to_syslog_bsd();
</Output>
<Output out2>
Module om_udp
Host 10.1.1.2
Port 514
Exec to_syslog_bsd();
</Output>
<Route 1>
Path in => out1, out2
</Route>
And not sure what to write in host and port.
nxlog.log should contain the error messages to help you diagnose the problems.
"And not sure what to write in host and port."
The destination where the udp syslog should be sent to.
So your host is the destination IP address or hostname (haven't verified hostname functionality) of your destination. AKA where you want to send your logs to. The port is the port. After you update make sure to go to nxlog/data/nxlog.log to check and see if everything started up OK. If it did you should see no error messages at the bottom. I've only done it with TCP and it says that it's trying to establish a connection and then nothing below it. Not sure what you would see with UDP. I also see a message that says "Info nxlog started"
Good luck
I've got an app which is running on JBoss 7. Its URL is http://localhost:8080/archive/app. How can I make it look http://localhost:8080/app or http://localhost/app?
PS. /archive means archive.war
1) Remove the welcome root. In the standalone.xml set:
<virtual-server name="default-host" enable-welcome-root="false">
2) Set the context root. In your archive.war add a jboss-web.xml file in the WEB-INF folder:
<?xml version="1.0"?>
<jboss-web>
<context-root>/</context-root>
</jboss-web>
3) Set the http port to 80. In the standalone.xml change:
<socket-binding name="http" port="8080"/> to <socket-binding name="http" port="80"/>
Remember that if your server runs on a Linux machine it must be launched by root in order to use ports under 1024. In that case you can redirect from 80 to 8080 by means of iptables without the need to change the standalone.xml socket-binding
I have a program running on a remote host that I need to connect to, handshake, then listen for messages. I have setup the following camel route:
<route>
<from uri="netty:tcp://localhost:50001?decoders=#decoders&sync=false" />
<bean ref="TransformMessage" method="inboundDecoder" />
<to uri="eventadmin:messages/aacus/inbound" />
</route>
<route>
<from uri="eventadmin:messages/aacus/outbound" />
<bean ref="TransformMessage" method="outboundEncoder" />
<to uri="netty:tcp://192.168.0.111:50001?allowDefaultCodec=false&sync=false" />
</route>
My question is how do I make this work? If I establish the route using
<from uri="netty:tcp://192.168.0.111:50001?decoders=#decoders&sync=false" />
it fails with a binding error.
How can I setup the connection to respond on a specific port without modifying the server?
This is not possible with either camel-mina nor camel-netty at this time of writing. A consumer can only bind to a local server. There is a JIRA ticket at Apache to implement such a new feature for the future. https://issues.apache.org/jira/browse/CAMEL-1077
Use the following workaround:
Instead ob 192.168.0.111 use localhost.
Then install "socat" and start it as follows
socat -s -u tcp4:192.168.0.111:50001 tcp4:localhost:50001
This will Tunnel your remote connection to the local service you created with camel/netty.