I am building an UDP messaging system with Protocol Buffers. I have several messages (e.g. ChannelUpdate, MessageCreate, ACK). If I send a message to the remote, how can I also send what type of message I am sending?
Your best bet is to always send the same message type: a root message that has one of the others. For example:
message SomeRoot {
oneof content {
ChannelUpdate channelUpdate = 1;
MessageCreate msgCreate = 2;
// etc
}
}
Now you're using an inbuilt protocol feature to enforce the logic.
Related
we're trying to build an event-based integration between an Apache Camel based system with produces messages in an Azure Service Bus topic and an .NET based consumer of these messages.
The producer used the AMQP interface of the Service Bus, while the .NET based consumer uses the current API from Microsoft in namespace Azure.Messaging.ServiceBus.
When we try to access the body in a received message as follows:
private async Task ProcessMessagesAsync(ProcessMessageEventArgs args)
{
try {
message = Encoding.UTF8.GetString(args.Message.Body);
}
catch( Exception e)
{
_logger.LogError(e, "Body not decoded: Message: {#message}", e.Message);
}
_logger.LogInformation("Body Type: {#bodytype}, Content-Type: {#contenttype}, Message: {#message}, Properties: {#properties}", raw.Body.BodyType, args.Message.ContentType, message, args.Message.ApplicationProperties);
await args.CompleteMessageAsync(args.Message);
}
the following exception is raised:
Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
System.NotSupportedException: Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
at Azure.Messaging.ServiceBus.Amqp.AmqpMessageExtensions.GetBody(AmqpAnnotatedMessage message)
at Azure.Messaging.ServiceBus.ServiceBusReceivedMessage.get_Body()
When peeking the topic with service bus explorer the message looks strange:
#string3http://schemas.microsoft.com/2003/10/Serialization/�_{"metadata":{"version":"1.0.0","message_id":"AGHcNehoD-hK0pPJCSga9v9sXFwC","message_timestamp":"2022-01-10T13:34:32.778Z"},"data":{"source_timestamp":"2022-01-05T17:20:31.000","material":"101052"}}
When messages are sent to another topic with a .NET producer there's a plaintext JSON body in the topic, as expected.
Did anybody successfully build a solution with Azure Service Bus with components based on the two mentioned frameworks, and what did the trick so that interoperability did work? Who can a Camel AMQP producer create messages with a BodyType of Data so that the body can be decoded by the .NET Service Bus client libraries without need to use GetRawAmqpMessage?
I can't speak to what format Camel is using, but that error message indicates that the Body that you're trying to decode is not an AMQP data body, which is what the Service Bus client library uses and expects.
In order to read a body that is encoded as an AMQP value or sequence, you'll need to work with the data in AMQP format rather than by using the ServiceBusReceivedMessage convenience layer. To do so, you'll want to call GetRawAmqpMessage on the ServiceBusReceivedMessage, which will give you back an AmqpAnnotatedMessage.
The annotated message Body property will return an AmqpMessageBody instance which will allow you to query the BodyType and retrieve the data in its native format using one of the TryGetmethods on the AmqpMessageBody.
On our procuder side a SAP Cloud Integration is used, when the Message Type parapeter of the AMQP Adapter is set to Binary, according to:
https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/d5660c146a93483692335e9d79a8c58f.html.
This seem to correspond to Apache Camel jmsMessageType set to Bytes,
see https://camel.apache.org/components/3.14.x/amqp-component.html for details.
The decoding of the body in the ServiceBusReceivedMessage works as expected and the BodyType is set to Data. If using Text on the producer side, the BodyType will be set to Value as described which led to the problems with the decoding of the body.
I am implementing ApiGateway-MicroService communication protocol in my app with MassTransit and RabbitMQ. That protocol is meant to replace "traditional" REST API communication between ApiGateway and Microservices (I am talking about simple request-response here and not about any kind of events, sagas, etc). So on microservice(s) side I have consumers (which respond to requests) and on ApiGateway side I have request clients. Usually microservice has let's say ~10 consumers (for example OrderingMicroservice has consumers for following requests: CreateOrder, UpdateOrder, GetOrderById, ListUserOrders etc). I am trying to figure out best topology (Masstransit + RabbitMQ) for this scenario.
Here are my goals, at least I think it should work like this:
A. Request messages (that are routed to consumer queue) should be durable for short time only (for example 20s) and then removed from the consumer queue (and request client should receive timeout error) and not routed to any other queue. So when microservice is temporary down or it is temporary too busy to receive next request from queue then request messages should be kept in the queue for 20s and then disappear.
B. Since RequestClient should timeout after ~20s, Response messages (that are routed to client "response-queue") should also be durable for short amount of time (~20s). Then they can disappear. If ApiGW is offline / too busy to receive response then response(s) should be discarded.
So basically I want to use MassTransit/RabbitMQ as a short-lived buffer between ApiGW and microservice(s).
// ApiGw MassTransit configuration
services.AddMassTransit(x =>
{
x.SetKebabCaseEndpointNameFormatter();
x.UsingRabbitMq((context, cfg) =>
{
});
x.AddRequestClient<ICreateGroupPayload>();
});
// Service MassTransit configuration
services.AddMassTransit(x =>
{
x.SetKebabCaseEndpointNameFormatter();
var entryAssembly = Assembly.GetEntryAssembly();
x.AddConsumers(entryAssembly);
x.UsingRabbitMq((context, cfg) =>
{
cfg.ConfigureEndpoints(context);
});
});
// Single consumer definition in service
public class CreateGroupActionDefinition : ConsumerDefinition<CreateGroupAction>
{
public CreateGroupActionDefinition()
{
EndpointName = "group-service";
}
}
This setup creates following exchanges and queues:
exchange ICreateGroupPayload (fanout, durable) => bind exchange:group-service
exchange group-service (fanout, durable) => bind queue:group-service
exchange PublicGateway_bus_4wdoyyro5ycgmgbybdcx1gp3r3 (fanout, autoDelete) => bind queue:PublicGateway_bus_4wdoyyro5ycgmgbybdcx1gp3r3
queue group-service (durable)
queue PublicGateway_bus_4wdoyyro5ycgmgbybdcx1gp3r3 (x-expires: 60000)
When I terminate ApiGw following exchanges/queues are removed from RabbitMQ within ~1min:
exchange PublicGateway_bus_4wdoyyro5ycgmgbybdcx1gp3r3
queue PublicGateway_bus_4wdoyyro5ycgmgbybdcx1gp3r3
My questions are:
Should I use separate queues (endpoint names) for different consumers in a microservice? Or I can use same queue (group-service for example) for different consumers/message types?
How I can modify my configuration to set expiration time on my consumer queues? Right now it's durable but I want messages to be removed after ~20s. Also I think such queue should not be deleted after consumer is disconnected because it should be able to send requests even when consumer is offline (but only for 20s).
How I can modify my configuration to set expiration time on my request client response queue to be 20s (currently it seems it's 60s by default?).
Maybe someone have any other suggestions on how to adjust topology to best fit for this scenario? The aim is to have the setup as fast as possible just for simple request-response + short time buffering for edge cases.
All the work is done by MassTransit, as you can understand from the request documentation. You can change the default request timeout from 30 seconds to 20 seconds when adding the request client to the container. There is also an .AddGenericRequestClient() method to automatically add requests clients for whatever request type is needed.
You can also specify the request timeout for each request, and it will set the message TimeToLive to match that value. The responses should be sent with a TimeToLive as needed.
I'm new to gRPC programming. I need to write a gRPC client which receives file content sent from the gRPC server and send this content to web-page. As the file content is huge, I've set it as a stream.
Below is my .proto file
service LogService {
rpc fetchLogContent(LogRequest) returns (stream LogResponse);
}
message LogRequest {
string ip = 1;
string fileName = 2;
}
message LogResponse {
string ip = 1;
string logContent = 2;
}
Now, in the client, when I use the blockingStub to access fetchLogContent, it returns an Iterator . I understand from the examples provided grpc.io - java, if there are list of response objects, (list of Feature objects in the example provided in link), an Iterator will be valid. But in my case, I need a single LogResponse which was sent as a stream. Please provide any suggestions/alternatives for this. Thanks in advance. :)
In your Method definition:
rpc fetchLogContent(LogRequest) returns (stream LogResponse);
The stream keyword means that you will get 0 or more LogResponse messages. If the content is very large, what you can do is read chunks of the file (like 4kb) and send multiple LogResponse messages each with a part of the file. The client side can read the chunks repeatedly, and piece it back together.
Since the ip field will likely not be changing each time, you can make you server only set ip on the first message. On the client, just use the very first ip received.
I've wondered how to identify the current protocol if it's using websocket or polling.
-- in the client. (appended for certainty)
I've found a valid information from the debug console.
Meteor.connection._stream.socket.protocol
and it seems to have one value among...
['websocket',
'xdr-streaming',
'xhr-streaming',
'iframe-eventsource',
'iframe-htmlfile',
'xdr-polling',
'xhr-polling',
'iframe-xhr-polling',
'jsonp-polling'];
is there more grace way to identify the current protocol?
and when would it be the fastest timing to detect the protocol?
By the way, I need it to use a different DDP server when the sticky sessions needed since AWS ELB doesn't support sticky sessions and websocket at the same time.
Meteor uses DDP protocol. To connect to a different Meteor server and call its methods, use DDP.connect as follows.
import { DDP } from 'meteor/ddp-client'
DDP.connect(url)
Unfortunately, there is no graceful to get the protocol. onConnection returns an object which has some info.
Meteor.onConnection(obj =>
{ console.log(obj.httpHeaders.x-forwarded-proto); });
This returns 'ws' for websocket. This way of getting the protocol is not graceful!
Meteor.status() gives a reactive data source.
(https://docs.meteor.com/api/connections.html#DDP-connect)
if (Meteor.isClient) {
Tracker.autorun(() => {
const stat = Meteor.status();
if (stat.status === 'connected') {
console.log(Meteor.connection._stream.socket.protocol);
}
});
}
something like that will give the current protocol in the client side.
Is there a way to check if SmtpClient successfully delivered an email? SmtpClient.Send() does not appear to return anything other than an exception. So if the method completes is it safe to assume the email will be successfully sent to the appropriate email box? My code is below:
MailMessage emailMessage = new MailMessage();
emailMessage.Subject = SubjectText;
emailMessage.IsBodyHtml = IsBodyHtml;
emailMessage.Body = BodyText;
emailMessage.From = new MailAddress(Settings.Default.FromEmail,Settings.Default.FromDisplayName);
//add recipients
foreach (string recipientAddress in RecipientAddresses.Split(new char[{','},StringSplitOptions.RemoveEmptyEntries))
emailMessage.To.Add(recipientAddress);
using (SmtpClient smtpClient = new SmtpClient())
{
smtpClient.Send(emailMessage);
}
No, there is no reliable way to find out if a message was indeed delivered.
Doing so will require access to the end SMTP server for every person you are emailing.
If you do not get an exception, you can assume that the SMTP server did its best to deliver the email.
There's no way to be 100% sure that a mail message has been received when sent via SmtpClient due to the way email works. The fact that SmtpClient doesn't throw an exception essentially means that you've done everything right, but a failure can happen further down the line, for example:
The receiving mail server could reject the mail
An intermediate mail server could reject the mail
The server that SmtpClient is transmitting mail through could decide to refuse to transmit the mail
One solution you could use is to create an httphandler for your website images. If you send an HTML message which includes at least 1 image, then you could embed querystring data to the end of that image. This could even be something like a 1x1 transparent image. When the user reads the email, this sends the request to the server to fetch the image data, and in turn, you could capture that request and denote that the message was read.
This is not bulletproof however, because most email clients block images by default unless the user specifies they would like to view images in the email.
If the recipient e-mail address is valid you don't get an immediate return value about the successful delivery of the message; see the signature:
public void Send(MailMessage message)
The SMTP server will notify the sender (or whoever you specify for the notification) almost immediately with an 'Undeliverable' notification whenever the recipient e-mail address is invalid/fake.
SMTP servers are required to periodically retry delivery. When the recipient e-mail address is a valid address but for some reason the SMTP server could not deliver the message, the SMTP server will return a failure message to the sender if it cannot deliver the message after a certain period of time.
RFC 2821 contains more details.
From section 2.1 Basic Structure
In other words, message transfer can occur in a single connection
between the original SMTP-sender and the final SMTP-recipient, or can
occur in a series of hops through intermediary systems. In either
case, a formal handoff of responsibility for the message occurs: the
protocol requires that a server accept responsibility for either
delivering a message or properly reporting the failure to do so.
See sections 4.5.4 and 4.5.5
From section 6.1 Reliable Delivery and Replies by Email
If there is a delivery failure after acceptance of a message, the
receiver-SMTP MUST formulate and mail a notification message. This
notification MUST be sent using a null ("<>") reverse path in the
envelope. The recipient of this notification MUST be the address from
the envelope return path (or the Return-Path: line).