I'm using spring-cloud-stream-kafka and I don't understand how is possible to change key.serializer property when using transactions. This is my configuration:
spring:
cloud:
stream:
bindings:
accountSource:
contentType: application/*+avro
destination: account
kafka:
binder:
brokers: ${KAFKA_BOOTSTRAP_ADDRESSES}
transaction:
transaction-id-prefix: tx-
producer:
configuration:
retries: 1
acks: all
bindings:
accountSource:
producer:
configuration:
key.serializer: org.apache.kafka.common.serialization.StringSerializer # Ignored!
schema:
avro:
subjectNamingStrategy: com.example.CustomSubjectNamingStrategy
schemaRegistryClient:
endpoint: ${KAFKA_SCHEMA_REGISTRY_URL}
I read here that when setting transaction properties at binder level, all other specific binding producer properties. Does this mean all producers in the applications should use the same key.serializer? It seems very restrictive to me.
The problem is the transaction has to be started by the consumer binding (so it can send the offset to the transaction and commit or rollback after success/failure).
If there are multiple producer bindings, the consumer binding has no idea which one(s) the application will send data to, so we have to use one global producer.
One solution would be to write a custom delegating serializer and set a header in the output message to tell the serializer which delegate serializer to call.
Related
we're trying to build an event-based integration between an Apache Camel based system with produces messages in an Azure Service Bus topic and an .NET based consumer of these messages.
The producer used the AMQP interface of the Service Bus, while the .NET based consumer uses the current API from Microsoft in namespace Azure.Messaging.ServiceBus.
When we try to access the body in a received message as follows:
private async Task ProcessMessagesAsync(ProcessMessageEventArgs args)
{
try {
message = Encoding.UTF8.GetString(args.Message.Body);
}
catch( Exception e)
{
_logger.LogError(e, "Body not decoded: Message: {#message}", e.Message);
}
_logger.LogInformation("Body Type: {#bodytype}, Content-Type: {#contenttype}, Message: {#message}, Properties: {#properties}", raw.Body.BodyType, args.Message.ContentType, message, args.Message.ApplicationProperties);
await args.CompleteMessageAsync(args.Message);
}
the following exception is raised:
Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
System.NotSupportedException: Value cannot be retrieved using the Body property.Use GetRawAmqpMessage to access the underlying Amqp Message object.
at Azure.Messaging.ServiceBus.Amqp.AmqpMessageExtensions.GetBody(AmqpAnnotatedMessage message)
at Azure.Messaging.ServiceBus.ServiceBusReceivedMessage.get_Body()
When peeking the topic with service bus explorer the message looks strange:
#string3http://schemas.microsoft.com/2003/10/Serialization/�_{"metadata":{"version":"1.0.0","message_id":"AGHcNehoD-hK0pPJCSga9v9sXFwC","message_timestamp":"2022-01-10T13:34:32.778Z"},"data":{"source_timestamp":"2022-01-05T17:20:31.000","material":"101052"}}
When messages are sent to another topic with a .NET producer there's a plaintext JSON body in the topic, as expected.
Did anybody successfully build a solution with Azure Service Bus with components based on the two mentioned frameworks, and what did the trick so that interoperability did work? Who can a Camel AMQP producer create messages with a BodyType of Data so that the body can be decoded by the .NET Service Bus client libraries without need to use GetRawAmqpMessage?
I can't speak to what format Camel is using, but that error message indicates that the Body that you're trying to decode is not an AMQP data body, which is what the Service Bus client library uses and expects.
In order to read a body that is encoded as an AMQP value or sequence, you'll need to work with the data in AMQP format rather than by using the ServiceBusReceivedMessage convenience layer. To do so, you'll want to call GetRawAmqpMessage on the ServiceBusReceivedMessage, which will give you back an AmqpAnnotatedMessage.
The annotated message Body property will return an AmqpMessageBody instance which will allow you to query the BodyType and retrieve the data in its native format using one of the TryGetmethods on the AmqpMessageBody.
On our procuder side a SAP Cloud Integration is used, when the Message Type parapeter of the AMQP Adapter is set to Binary, according to:
https://help.sap.com/viewer/368c481cd6954bdfa5d0435479fd4eaf/Cloud/en-US/d5660c146a93483692335e9d79a8c58f.html.
This seem to correspond to Apache Camel jmsMessageType set to Bytes,
see https://camel.apache.org/components/3.14.x/amqp-component.html for details.
The decoding of the body in the ServiceBusReceivedMessage works as expected and the BodyType is set to Data. If using Text on the producer side, the BodyType will be set to Value as described which led to the problems with the decoding of the body.
On Symfony 5.1, I created two very basic message classes.
The first updates the database and then it calls the other to send a mail via SwiftMailer.
Both of them arrive to the consumer command as they are logged by the bin/console messenger:consume async -vv command, but while the class that updates the database is handled, the other remains suspended.
The fact is that if I stop and restart the consume command, this message becomes handled by the consumer and so the mail is sent.
I have tried with Redis and RabbitMQ transports, with the very same results.
This is the messenger configuration:
framework:
messenger:
transports:
async:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
routing:
'App\Message\FollowRequestMessage': async
'App\Message\EmailMessage': async
This is the MESSENGER_TRANSPORT_DSN in .env:
MESSENGER_TRANSPORT_DSN=amqp://guest:guest#127.0.0.1:5672/%2f/messages
UPDATE
I defined another transport to handle the FollowRequest and Email messages separately, by duplicating the "async" transport config in the yaml messenger configuration.
This has fixed the issue, but I don't know if it the right way to handle the case.
So, it is correct to create a different transport for each kind of message? Or is it a workaround because of an error in something else?
We have a project that partially uses DDD. When changing / creating an entity, we throw an event inside the entity. Next, the Doctrine Event Subscriber takes all the events and forwards them to the EventBus.
Let's say we have a MissionCreatedEvent event. I want to make routing key mission.event.created for it and tie it into 2 queues: new_mission_notification_to_part_time_employee and new_mission_area_notification_to_manager.
This is what the config looks like:
transports:
async_new_mission_notification_to_part_time_employee:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
exchange:
name: mission
type: topic
queues:
new_mission_notification_to_part_time_employee:
binding_keys: ['mission.event.created']
retry_strategy:
max_retries: 0
async_new_mission_area_notification_to_manager:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
exchange:
name: mission
type: topic
queues:
new_mission_area_notification_to_manager:
binding_keys: ['mission.event.created']
retry_strategy:
max_retries: 0
routing:
'App\Domain\Mission\Event\MissionCreatedEvent':
- async_new_mission_notification_to_part_time_employee
- async_new_mission_area_notification_to_manager
The problem is that Symfony Messenger first runs on transports and only then pushes the message into exchange. Since we have 2 transports, it turns out that the event with routing key mission.event.created is triggered twice. Because of this, my Message handlers run 2 times.
There is an option not to use routing key mission.event.created. But use similar to mission.cmd.notify_part_time_employee_about_new_mission
and mission.cmd.notify_manager_about_new_mission_in_area for each transport with direct exchange.
This will help solve duplication problems, but I can’t use all the features that RabbitMQ gives me :(
Trying to get spring cloud gateway to load balance across a couple of instances of our application, but just can't figure it out. We don't have a service registry at present (no Eureka etc).
I've been trying to use ribbon and have a configuration like so:
spring:
application:
name: gateway-service
cloud:
discovery:
locator:
enabled: true
gateway:
routes:
- id: my-service
uri: lb://my-load-balanced-service
predicates:
- Path=/
filters:
- TestFilter
ribbon:
eureka:
enabled: false
my-load-balanced-service:
ribbon:
listOfServers: localhost:8080, localhost:8081
However when I try a request to the gateway, I get a 200 response with content-length 0, and my stubs have not been hit.
I have a very basic setup, no beans defined.
How can I get ribbon to play nice / or an alternative?
You should check out whether spring-cloud-starter-netflix-ribbon dependency is on your project or not
I'm using the new Symfony Messenger Component 4.1 and RabbitMQ 3.6.10-1 to queue and asynchronously send email and SMS notifications from my Symfony 4.1 web application. My Messenger configuration (messenger.yaml) looks like this:
framework:
messenger:
transports:
amqp: '%env(MESSENGER_TRANSPORT_DSN_NOTIFICATIONS)%'
routing:
'App\NotificationBundle\Entity\NotificationQueueEntry': amqp
When a new notification is to be sent, I queue it like this:
use Symfony\Component\Messenger\MessageBusInterface;
// ...
$notificationQueueEntry = new NotificationQueueEntry();
// [Set notification details such as recipients, subject, and message]
$this->messageBus->dispatch($notificationQueueEntry);
Then I start the consumer like this on the command line:
$ bin/console messenger:consume-messages
I have implemented a SendNotificationHandler service where the actual delivery happens. The service configuration:
App\NotificationBundle\MessageHandler\SendNotificationHandler:
arguments:
- '#App\NotificationBundle\Service\NotificationQueueService'
tags: [ messenger.message_handler ]
And the class:
class SendNotificationHandler
{
public function __invoke(NotificationQueueEntry $entry): void
{
$this->notificationQueueService->sendNotification($entry);
}
}
Until this point, everything works smoothly and the notifications get delivered.
Now my question: It may happen that an email or SMS cannot be delivered due to a (temporary) network failure. In such a case, I would like my system to retry the delivery after a specified amount of time, up to a specified maximum number of retries. What is the way to go to achieve this?
I have read about Dead Letter Exchanges, however, I could not find any documentation or example on how to integrate this with the Symfony Messenger Component.
What you need to do is tell RabbitMQ, that the message is rejected instead of acknowledged. By default the messenger will take care of this inside the AmqpReceiver. As you can see there, if you throw an exception that implements the RejectMessageExceptionInterface inside your handler, the message will automatically be rejected for you.
You could also "simulate" this behaviour with custom middleware. I created something like it, in a small demo application. The mechanism consists of a middleware that wraps the (serialized) original message inside a new RetryMessage and sends it via a custom message bus to a different queue, used as a dead letter exchange. The handler for that message will then unpack the RetryMessage (getting the original message and deserializing it) and transmit it over the default bus:
See:
RetryMessage
RetryMiddleware
messenger.yaml
RetryMessageHandler
This is a basic setup which rejects the message and allows you to consume it again instantly(!). You probably want to add additional information such as headers for timestamps when delaying the consumption to improve on this. For this you should look at writing your own receiver, middleware and/or handler.