I am working on an orchestration system using Node-RED and MQTT.
I have decided to dissociate the event acquisition from the treatment. The main objective is to quickly push events on a queue and treat them soon as possible in real time.
The system operates like this :
I receive an event on an HTTP Rest API,
Push this event on an MQTT Topic,
On an another flow, listen and read events from the MQTT Topic,
Launch several actions/process from this event (up to 5/10 seconds).
But I am facing an issue: If I receive too quickly 2 related events, the second event could change the processing of the first event. To solve this, I would like to synchronize my event consumption/processing in order to keep them ordered.
MQTT QoS 2 messages will be delivered in order. How can I simply implement a synchronization paradigm in Node-RED ? Is it possible to avoid MQTT Client listening while processing an event ?
No, you can't turn the MQTT client off.
And no there is no concept of synchronisation, mainly because all NodeJS apps are purely single threaded so 2 things can't actually happen at once, tasks just yield normally when they get to something IO bound.
I'm not sure you actually gain anything receiving it via HTTP and then re-consuming it via MQTT.
If you want to queue the incoming events up you could use the delay node to rate limit the input to something you are sure the processing can manage. The rate limit option has 2 modes, one that drops messages and one the queues them,.
Place a simple message Q on output of MQTT client and get it to release next message by sending it trigger=true (or might be able to get it to release a message at a set rate). I am still looking at these.
https://flows.nodered.org/node/node-red-contrib-simple-message-queue
Here is my proposal to solve the problem: https://flows.nodered.org/flow/3003f194750b0dec19502e31ca234847.
Sync req/res REST API with async workers based on MQTT
Example implements one REST API endpoint, one inject node for doing requests to that endpoint and two workers doing their job async.
It's still in progress so expect changes. Feel free to contact me and chat about this or other solutions for orchestration along with Node-RED.
Related
I have an embedded device running BT5 with GATT server setup. On the server I have setup a service with various characteristics to allow a client (PC or Mobile Device) to adjust various parameters of the device by writing to the characteristics.
I would like, for the device to send a response back from the application level for each write. It's not clear to me what the recommended way would be.
I thought about having the client read or subscribe to a general status characteristic, but I want to make sure I am not missing an easier way to do this. I looked at the BT write with response command, but it seems the acknowledgement for that may happen lower than the application.
You should be able to use the Write Response as "application level response". I have not seen any Bluetooth stack where this response is sent at a lower level before the application has processed the request. The reason is probably because the application can even send an Application Error code instead of a Write Response, so it would be stupid to move the Write Response handling to a lower level. Even in Android (if you set up a GATT server) you send the Write Response from the application.
The situation is different with Indications, though, where the Bluetooth stack sometimes sends the Confirmation at a lower level than the application, before it even informs the application that an Indication has arrived, which I find a bit strange and makes Indications kind of pointless compared to Notifications.
I solved this using a Notification characteristic. The client first subscribes to notification events on that CCD, and then every command sent to the host/device is acknowledged by the host firing the notification. To better synchronize command-and-response, you could add an incremental command-id with every command, and have the command-id be part of the notification data that is sent back to the client.
However I implemented this because I needed a response after the device has processed the command, with the results sent back to the client. If all you want to know is whether or not the host has received the command, a Write-With-Response CCD is the way to go.
I looked at the BT write with response command, but it seems the acknowledgement for that may happen lower than the application.
Indeed, the Write-With-Response-Handler is almost always implemented on the BLE stack, not on application level. However I don't see why this would be a problem; you should get error reports by your BLE stack in some form when a Write-with-Response fails. If it's a blocking call it might even return a success-value.
We understand that there are port tear-down during transactions and different ports may be used when sending messages over to the counterparties. When a node goes down, the messages are still sent but they are being queued in the MQ, is there a recommended way how could we monitor these transactions/messages?
Unfortunately, you can't currently monitor these messages.
This is because Artemis does not store its queued messages in a human-readable/queryable format. Instead, the queued messages are stored in the form of a high-performance journal that contains a lot of information that is required in case the message queue's state needs to be restored from a hard bounce.
I approached this by finding the documents here: https://docs.corda.net/node-administration.html#monitoring-your-node
where it illustrates Corda flow metrics visualized using hawtio.
I just needed to download and startup hawt.io and connect it to any ( or the specified node PID ) net.corda.node.Corda and by going to the JMX tab we could see the messages in queue.
as we all know message bus like rabbitMQ is mainly meant for asynchronous messaging so standard approch is to fire and forget like publish something on bus and don't worry about who will process published message or when. But i'm thinking about latest talk in our development team about synchronous processing of message: case would be to publish message to service bus and as as publisher i want to wait for any subscriber to process message and return results to me - so it looks rather as request-response model. I'm thinking now of one con like degrading performance in this model. What are your thoughts? When to use async and when sync? What are the tradeoffs?
Synchronous messaging is possible but impacts scalability. If a publisher has to wait for its recipients to respond, then it will be limited in how much it can achieve at any given time.
However, you can achieve request-response using asynchronous messaging. In RabbitMQ, you do this by means of the Remote Procedure Call (RPC) pattern.
To put it simply, your publisher publishes a message, but doesn't wait for the response; it can continue doing other stuff in the meantime. The publisher does keep track of it though, by putting a CorrelationId on the message, and storing it locally. The message eventually reaches a consumer, who processes it and responds back to the publisher on a different queue. The reply has the same CorrelationId. When the publisher receives the reply, it can then mark that particular call (via the CorrelationId) as processed.
If you want, you can also do other things with the CorrelatonId, such as timeout those messages for which we haven't received a reply after e.g. 30 seconds.
I'm trying to get a deep understand how works the Push API communication between the client and the RabbitMQ server.
As I know - but correct me in case - the client open a TCP connenction to the broker (RabbitMQ) and keep this connenction alive until the client decision to close it. But during this connection the client can get messages immediately.
My question is, during this connection, do the client monitor the Broker to ask him for messages, or when the Broker forward a message to the Queue, where the client subscribed, just take that connencion and push the data to the client?
first case: client monitor the broker for messages
last case: client don't need to monitor the broker, broker just push the data
or other?
There are two options to receive messages
The client registers a consumer callback (basicConsume) on the channel; the broker then "pushes" messages to the consumer.
The client sends the broker a basicGet and receives one message (if present).
The first use case is the most common.
Since you tagged the question with spring-amqp I assume you are interested in Spring. For the first case, Spring AMQP has a listener container (and #RabbitListener annotation); for the second case, one of the RabbitTemplate receive operations can be used.
I suggest you look at the tutorials to get a basic understanding. They cover several languages including pure java and Spring AMQP.
You can also look at the Spring AMQP Reference Manual.
In order to overcome the (apparent) 4 minute idle connection timeout on the Azure load balancer, it seems necessary to send some data down the pipe to the client every now and again to keep the connection from being regarded as idle.
Our controller is set up as an AsyncController, and it fires several different asynchronous methods on other objects, all of which are set up to use IO Completion Ports. Thus, we return from our method immediately, and when the completion packet is processed, IIS hooks back up to the original request so that we can render our View.
Is there any way to periodically send a few bytes down the wire in this case? In a "classic" situation, we could have executed the method and then just spun while we waited, sending data every few seconds until the asynchronous method was complete. But, in this situation, the IIS thread is freed to go do other business, and we hook back up to it in our completion callback. What to do? Is this possible?
While your particular case concerns Windows Azure specific (the 4 minute timeout of LBs), the question is pure IIS / ASP.NET workwise. Anyway, I don't think it is possible to send "ping-backs" to the client while in AsyncController/AsyncPage. This is the whole idea of the AsyncPages/Controllers. The IIS leaves the socket aside having the thread serving other requests. And gets back only when you got the OutstandingOperations to zero with AsyncManager.OutstandingOperations.Decrement(); Only then the control is given back to send final response to the client. And once you are the point of sending response, there is no turning back.
I would rather argue for the architectural approach of why you thing someone would wait 4 minutes to get a response (even with a good animated "please wait")? A lot of things may happen during this time. From browser crash, through internet disruption to total power loss/disruption at client. If you are doing real Azure, why not just send tasks for a Worker Role via a Queue (Azure Storage Queues or Service Bus Queues). The other option that stays in front of you for so long running tasks is to use SingalR and fully AJAXed solution. Where you communicate via SignalR the status of the long running operation.
UPDATE 1 due to comments
In addition to the approach suggested by #knightpfhor this can be also achieved with a Queues. Requestor creates a task with some Unique ID and sends it to "Task submission queue". Then "listens" (or polls at regular/irregular intervals) a "Task completion" queue for a message with given Task ID.
In any way I don't see a reason for keeping client connected for the whole duration of the long running task. There are number of ways to decouple such communication.