I have multiple Splunk Universal Forwarders, and I am looking to send them to an ETL tool. I am trying to evaluate whether FluentD is a compatible option for this.
FluentD has built in plug-ins for many specific log collection tools, however the Splunk UF is not listed as one of them. However, FluentD is capable of receiving traffic over HTTPS, and Splunk is capable of sending log data over HTTPS, does this mean the two are compatible?
Splunk's Universal Forwarder (UF) can send data using the Splunk-to-Splunk protocol or Splunk HTTP Event Collector (HEC) format.
Why not use Splunk instead of FluentD? Or use Cribl instead of a UF to send to FluentD?
Related
I am trying to analyze Kafka packages, but Wireshark is not recognizing Kafka protocol for some reason. Instead of something like this:
I get everything as plain TCP:
I made sure that the protocol is enabled and the correct port is set.
I also tried to use Analyze/Decode As... without any success.
I am using Wireshark v3.6.6
How can I make it work?
As far as I can tell, Firestore uses protocol buffers when making a connection from an android/ios app. Out of curiosity I want to see what network traffic is going up and down, but I can't seem to make charles proxy show any real decoded info. I can see the open connection, but I'd like to see what's going over the wire.
Firestores sdks are open source it seems. So it should be possible to use it to help decode the output. https://github.com/firebase/firebase-js-sdk/tree/master/packages/firestore/src/protos
A few Google services (like AdMob: https://developers.google.com/admob/android/charles) have documentation on how to read network traffic with Charles Proxy but I think your question is, if it’s possible with Cloud Firestore since Charles has support for protobufs.
The answer is : it is not possible right now. The firestore requests can be seen, but can't actually read any of the data being sent since it's using protocol buffers. There is no documentation on how to use Charles with Firestore requests, there is an open issue(feature request) on this with the product team which has no ETA. In the meanwhile, you can try with the Protocol Buffers Viewer.
Alternatives for viewing Firestore network traffic could be :
From Firestore documentation,
For all app types, Performance Monitoring automatically collects a
trace for each network request issued by your app, called an HTTP/S
network request trace. These traces collect metrics for the time
between when your app issues a request to a service endpoint and when
the response from that endpoint is complete. For any endpoint to which
your app makes a request, Performance Monitoring captures several
metrics:
Response time — Time between when the request is made and when the response is fully received
Response payload size — Byte size of the network payload downloaded by the app
Request payload size — Byte size of the network payload uploaded by the app
Success rate — Percentage of successful responses compared to total responses (to measure network or server failures)
You can view data from these traces in the Network requests subtab of
the traces table, which is at the bottom of the Performance dashboard
(learn more about using the console later on this page).This
out-of-the-box monitoring includes most network requests for your app.
However, some requests might not be reported or you might use a
different library to make network requests. In these cases, you can
use the Performance Monitoring API to manually instrument custom
network request traces. Firebase displays URL patterns and their
aggregated data in the Network tab in the Performance dashboard of the
Firebase console.
From stackoverflow thread,
The wire protocol for Cloud Firestore is based on gRPC, which is
indeed a lot harder to troubleshoot than the websockets that the
Realtime Database uses. One way is to enable debug logging with:
firebase.firestore.setLogLevel('debug');
Once you do that, the debug output will start getting logged.
Firestore use gRPC as their API, and charles not support gRPC now.
In this case you can use Mediator, Mediator is a Cross-platform GUI gRPC debugging proxy like Charles but design for gRPC.
You can dump all gRPC requests without any configuration.
For decode the gRPC/TLS traffic, you need download and install the Mediator Root Certificate to your device follow the document.
For decode the request/response message, you need download proto files which in your description, then configure the proto root in Mediator follow the document.
the following are my understanding
.net core api with serilog singk to ELK can directly send logs to ELK
Logstash & Fluentd is needed only if we want to send a log file (by massaging the data) to ELK
my question is
why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my api?
If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?
I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.
Appreciate some clear explanation on this.
why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my API?
If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?
Question 2 answers question 1 here. FluentD has a battle-tested buffering mechanism to deal with ELK outages. Moreover, you don't want to use the app thread to deal with a task completely unrelated to an app - log shipping. This complicates your app and decreases portability.
I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.
Correct. FluentD has a buffer https://docs.fluentd.org/configuration/buffer-section. It will send whatever came for the period of time you've set in match (buffer is used to accumulate logs for the time period here). If the log backend (ELK) is down, it will keep storing the unsuccessful log records in the buffer. Depending on the buffer size this can handle pretty severe log backend outages. Once the log backend (ELK) is up again, all the buffers are sent to it and you don't lose anything.
Logstash's persistent queue is a similar mechanism, but they went further and after the in-memory buffer they added external queues like Kafka. FluentD is also capable to use the queue when you use kafka input/output, and you still have a buffer on top of this in case a Kafka goes down.
I have an internal website which is completely based on Geo Map and uses HTTP/HTTPS protocol also wfs and wms service calls.
While recording with LR I am able to launch and record all the contents,
but after that for next transaction when I click on any of the option to see the map It is not recording in LR.
For next transaction type of calls are jquery which gets the server response in form of images (geo locations).
I tried recording with HTTP single and HTTP with Web-services multiple protocols also
My LR version is 12.01 and recording it with Chrome browser
Help me out please !
Both WFS and WMS, from the open geospacial consortium, should be using HTTP as a carrier. Can you provide insight on why such calls to these servers are being ignored
Do you have the servers in any sort of filter as a third party element which should not be tested as you do not have ownership or control of the target?
Are you electing to connect to these services over a protocol other than HTTP?
You note, "Not recording," what is the specific objective evidence that a recording is not occuring? Lack of evidence in the recording logs? Lack of an increase in the events counter? Lack of ?? Please clarify
Have you tried the HTTP Proxy method of recording versus the default sockets model?
I'm beginning to learn to code.
Someone said to me: "cURL is the best http client".
To help me understand this sentence, I have two questions:
what is an HTTP CLIENT; and
what is cURL?
I understand you are asking two things:
What is an HTTP CLIENT?
This is any program/application used make communications on the web using the Hypertext Transfer Protocol (HTTP). A common example is a browser.
What is cURL?
This a particular HTTP CLIENT designed to make HTTP communications on the web but built to be used via the command line of a terminal (command prompt).
If you perform a search for these topics, you will easily be able to find more in depth explanations about each.