the following are my understanding
.net core api with serilog singk to ELK can directly send logs to ELK
Logstash & Fluentd is needed only if we want to send a log file (by massaging the data) to ELK
my question is
why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my api?
If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?
I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.
Appreciate some clear explanation on this.
why do I need logstash | fluentd if I can directly send my logs to ELK using a serilog sink in my API?
If I send using serilog sing to ELK directly what happens if the connection to ELK is down? will it save temporarily and re send?
Question 2 answers question 1 here. FluentD has a battle-tested buffering mechanism to deal with ELK outages. Moreover, you don't want to use the app thread to deal with a task completely unrelated to an app - log shipping. This complicates your app and decreases portability.
I read in article it says FluentD uses persistent queue and Logstash doesn't but why exactly this queue needed? lets say If my app have 1 logfile and it gets updated every second. So logstash sends the whole file to ELK every second? even if it fails it can resend my log file to ELK right? so why a persistent queue needed here for Fluentd/ logstash comparasion.
Correct. FluentD has a buffer https://docs.fluentd.org/configuration/buffer-section. It will send whatever came for the period of time you've set in match (buffer is used to accumulate logs for the time period here). If the log backend (ELK) is down, it will keep storing the unsuccessful log records in the buffer. Depending on the buffer size this can handle pretty severe log backend outages. Once the log backend (ELK) is up again, all the buffers are sent to it and you don't lose anything.
Logstash's persistent queue is a similar mechanism, but they went further and after the in-memory buffer they added external queues like Kafka. FluentD is also capable to use the queue when you use kafka input/output, and you still have a buffer on top of this in case a Kafka goes down.
Related
I'm using gRPC for internal communication b/w 2 java services.
I configured gRPC retry using service config . I am able to get retry count in server using "grpc-previous-rpc-attempts" metadata header. However , I don't find any logs those are getting printed in the client app while retries are happening .
why gRPC is not logging retry attempts which ideally should have
been done when retries are configured
Is there any way to log each
retry attempt in the client app? This is needed for better
observability.
Thanks
At this moment there is no logging support for retries, but it would be a reasonable thing to add.
You can either file a feature request to get that added, or better yet make a pull request for the change. If you decide to make the change, it should be localized to just RetriableStream.java. Feel free to tag me (#temawi) on it and I'll review it for you.
As far as I can tell, Firestore uses protocol buffers when making a connection from an android/ios app. Out of curiosity I want to see what network traffic is going up and down, but I can't seem to make charles proxy show any real decoded info. I can see the open connection, but I'd like to see what's going over the wire.
Firestores sdks are open source it seems. So it should be possible to use it to help decode the output. https://github.com/firebase/firebase-js-sdk/tree/master/packages/firestore/src/protos
A few Google services (like AdMob: https://developers.google.com/admob/android/charles) have documentation on how to read network traffic with Charles Proxy but I think your question is, if it’s possible with Cloud Firestore since Charles has support for protobufs.
The answer is : it is not possible right now. The firestore requests can be seen, but can't actually read any of the data being sent since it's using protocol buffers. There is no documentation on how to use Charles with Firestore requests, there is an open issue(feature request) on this with the product team which has no ETA. In the meanwhile, you can try with the Protocol Buffers Viewer.
Alternatives for viewing Firestore network traffic could be :
From Firestore documentation,
For all app types, Performance Monitoring automatically collects a
trace for each network request issued by your app, called an HTTP/S
network request trace. These traces collect metrics for the time
between when your app issues a request to a service endpoint and when
the response from that endpoint is complete. For any endpoint to which
your app makes a request, Performance Monitoring captures several
metrics:
Response time — Time between when the request is made and when the response is fully received
Response payload size — Byte size of the network payload downloaded by the app
Request payload size — Byte size of the network payload uploaded by the app
Success rate — Percentage of successful responses compared to total responses (to measure network or server failures)
You can view data from these traces in the Network requests subtab of
the traces table, which is at the bottom of the Performance dashboard
(learn more about using the console later on this page).This
out-of-the-box monitoring includes most network requests for your app.
However, some requests might not be reported or you might use a
different library to make network requests. In these cases, you can
use the Performance Monitoring API to manually instrument custom
network request traces. Firebase displays URL patterns and their
aggregated data in the Network tab in the Performance dashboard of the
Firebase console.
From stackoverflow thread,
The wire protocol for Cloud Firestore is based on gRPC, which is
indeed a lot harder to troubleshoot than the websockets that the
Realtime Database uses. One way is to enable debug logging with:
firebase.firestore.setLogLevel('debug');
Once you do that, the debug output will start getting logged.
Firestore use gRPC as their API, and charles not support gRPC now.
In this case you can use Mediator, Mediator is a Cross-platform GUI gRPC debugging proxy like Charles but design for gRPC.
You can dump all gRPC requests without any configuration.
For decode the gRPC/TLS traffic, you need download and install the Mediator Root Certificate to your device follow the document.
For decode the request/response message, you need download proto files which in your description, then configure the proto root in Mediator follow the document.
.. when the http response entity is not consumed, or the client tcp buffer becomes full, or when the rate of client taking from its tcp buffer is lower then the rate of server pushing data to it?
I am looking for a way for to achieve the following:
Let's assume that there is a backpressure-able source of data on the server, such as an Apache Kafka topic.
If I consume this source from a remote location it may be possible that the rate at which that remote location can consume is lower - this is solved if Kafka client or consumer is used.
However let's assume that the client is a browser and that exposing direct Kafka protocol / connectivity is not a possibility.
Further, let's assume that there is a possibility of getting all the value even if jumping over some messages.
For instance in case of compacted topics, getting only the latest values for each key is enough for a client, no need to go through intermediate values.
This would be equivalent to Flowable.onBackpressureLatest() or AkkaStreams.aggregateOnBackpressure or onBackpressureAggregate.
Would it be a way to expose the topic over HTTP REST (e.g. Server Side Events / chunked transfer-encoding) or over web-sockets, that would achieve this effect of skipping over intermediate values for each key?
Please advise, thanks
Akka http supports back pressure based on TCP protocol very well and you can read about using it in combination with streaming here
Kafka consumption and exposure via http with back pressure can be easily achieved in combination of akka-http, akka-stream and alpakka-kafka.
Kafka consumers need to do polling and alpakka covers back pressure with reduction of polling requests.
I don't see the necessity of skipping over the messages when back pressure is fully supported. Kafka will keep track of the offset consumed by a consumer group (the one you pick for your service or http connection) and this will guarantee eventual consumption of all messages. Of course, if you produce messages way faster in a topic, the consumer will never catch up. Let me know if this is your case.
As a final note, you may check out Confluent REST Proxy API, which allows you to read Kafka messages in a restful manner.
I am currently using PHP / CURL on the back-end to update values in Firebase. We use Firebase primarily as a JavaScript layer so we can show browser and app clients real time status progression of jobs we process.
We've gotten to the point where we're doing quite a bit of status updating using CURL from our back-end and I feel we are close to the threshold where establishing a persistent connection between Firebase and our server would be more efficient than opening and closing dozens of HTTP requests per minute.
Is there anyway to do this with Firebase right now?
Firebase has server-side SDKs for Java and Node.js. If you can't use those, the REST API is your only alternative.
If you'd like to listen for data over REST, you can use Firebase's REST Streaming API, which uses a long-lived HTTP connection to return a stream of events. It is similar to the Firebase SDKs, but it can only attach a single listener per connection, and you'll still need separate requests for write operations.
That last part seems to the crux of your problem. So I'm afraid there really aren't any alternatives from using the SDKs as I mentioned. In my testing using HTTP requests for frequent small (although in my case admittedly read) operations was quite fast.
I am trying to implement Reliable WCF Service with MSMQ based on this architecture (http://www.devx.com/enterprise/Article/39015)
A message may be lost if queue is not available (even cluster doesn't provide zero downtime)
Take a look at the simple order processing workflow
A user enters credit card details and makes a payment
Application receives a success result from payment gateway
Application send a message as “fire and forget”/”one way” call to a backend service by WCF MSMQ binding
The user will be redirected on the “success” page
Message is stored in a REMOTE transactional queue (windows cluster)
The backend service dequeue and process the message, completes complex order processing workflow and, as a result, sends an as email confirmation to the user
Everything looks fine as excepted.
What I cannot understand how can we guarantee that all “one way” calls will be delivered in the queue?
Duplex communication is not a case due to the user should be redirected at the result web page ASAP.
Imagine the case when a user received “success” page with language “… Your payment was made, order has been starting to process, and you will email notifications later…” but the message itself is lost.
How durability can be implemented for step 3?
One of the possible solutions that I can see is
3a. Create a database record with a transaction details marked as uncompleted, just to have any record about the transaction. This record may be used as a start point to process the lost message in case of the message will not be saved in the queue.
I read this post
The main thing to understand about transactional MSMQ is that there
are three distinct transactions involved in a transactional send to a
remote queue.
The sender writes the message to a local queue.
The queue manager on the senders machine transmits the message across the wire to the queue manager on the recipient machine
The receiver service processes the queue message and then removes the message from the queue.
But it doesn’t solve described issue - as I know WCF netMsmqBinding doesn’t use local queue to send messages to remote one.
But it doesn’t solve described issue - as I know WCF netMsmqBinding
doesn’t use local queue to send messages to remote one.
Actually this is not correct. MSMQ always sends to a remote queue via local queue, regardless of whether you are using WCF or not.
If you send a message to a remote queue then look in Message Queuing in Server Management you will see in Outbound queues that a queue has been created with the address of the remote queue. This is a temporary queue which is automatically created for you. If the remote queue was for some reason unavailable, the message would sit in the local queue until it became available, and then it would be transmitted.
So durability is provided because of the three-phase commit:
transactionally write message locally
transactionally transmit message
transactionally receive and process message
There are instances where you may drop messages, for example, if your message processing happens outside the scope of the dequeue transaction, and also instances where it is not possible to know if the processing was successful (eg back-end web service call times out), and of course you could have a badly formed message which will never succeed processing, but in all cases it should be possible to design for these.
If you're using public queues on a clustered environment then I think there may be more scope for failure as clustering msmq introduces complexity (I have not really used so I don't know) so try to avoid if possible.