I am trying to use the Content-based routing in the latest version of the Spring Cloud stream. As per this document -> Content-based routing, it mentions how to use it in the legacy system/StreamListener.
This is my code with StreamListener
#StreamListener(target = EventChannels.FILE_REQUEST_IN
, condition = "headers['saga_request']=='FILE_SUBMIT'")
public void handleSubmitFile(#Payload FileSubmitRequest request) {
}
#StreamListener(target = EventChannels.FILE_REQUEST_IN
, condition = "headers['saga_request']=='FILE_CANCEL'")
public void handleCancelFile(#Payload FileCancelRequest request) {
}
By using the condition, it was possible to route the message to two different functions.
I am trying to consume the message with a Functional interface approach as below.
#Bean
public Consumer<String> consumeMessage(){
return event -> {
try {
LOGGER.info("Consumer is working: {}", event);
} catch (Exception ex) {
LOGGER.error("Exception while processing");
}
};
}
How can I achieve similar content-based routing in the functions? TIA.
Other details->
Spring boot version - 2.3.12.RELEASE
Spring cloud version - Hoxton.SR11
Have you seen this - https://docs.spring.io/spring-cloud-stream/docs/3.1.4/reference/html/spring-cloud-stream.html#_event_routing?
We provide two different routing models TO and FROM. The included link contains samples so please look through it and feel free to post any followups
Related
We are working on service which collect data from AWS SQS then send batch to client. We are using mediator to publish notifications. The diagram of program looks like:
The problem is in first NotificationHandler from Mediatr.
private readonly EventCollectorHostedService _collector;
public CollectIncomingEventNotificationHandler(EventCollectorHostedService collector)
{
_collector = collector;
}
Class EventCollectorHostedService is register after Mediator so is not visible during registering this NotificationHandler and additionally it use Mediator to publish notification that batch is ready to send.
The error is that cannot construct CollectIncomingEventNotificationHandler because -> Unable to resolve service for type 'Api.Services.HostedServices.EventCollectorHostedService'.
services.AddMediatR(typeof(Startup).GetTypeInfo().Assembly);
services.AddHostedService<EventCollectorHostedService>();
The ugly solution is to declare some functionality in EventCollectorHostedService as static or instead of injecting EventCollectorHostedService, inject IServiceProvider.
But these solution don't look clean for me so do you have any other better solution ?
Thanks in advance.
Maybe someone encountered with similar problem so finally i have a brilliant solution.
Background services have to be treat like separate microservices based on event driven architecture so we have to make internal message broker mechanism.
The very simple solution which cover my case is:
public class NotificationChannel : INotificationChannel
{
public event EventHandler<IncomingEventNotificataionEventArgs> IncomingEventReceived;
public void Publish<T>(T notification)
{
if(notification is IncomingEventNotification incomingEventNotification)
{
OnIncomingEventReceived(incomingEventNotification);
}
}
protected virtual void OnIncomingEventReceived(IncomingEventNotification notification)
{
if(IncomingEventReceived != null)
{
var args = new IncomingEventNotificataionEventArgs(notification);
IncomingEventReceived(this, args);
}
}
}
A few examples for implementing HealthIndicator needs a KafkaTemplate. I don't actually manually create a KafkaTemplate but the HealthIndicator requires one. Is there a way to automatically grab the created KafkaTemplate (that uses the application.yml configurations)? This is versus manually creating duplicated configurations that already exist in the application.yml in a newly created consumerFactory.
See this answer.
(S)he wanted to get access to the template's ProducerFactory but you can use the same technique to just get a reference to the template.
That said, the binder comes with its own health indicator.
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/blob/a7299df63f495af3a798637551c2179c947af9cf/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/KafkaBinderHealthIndicator.java#L52
We also had the same requirement of creating a custom health checker Spring Cloud stream. We leveraged the inbuild health checker(KafkaBinderHealthIndicator). But while injecting the KafkaBinderHealthIndicator bean facing lot of issue. So instead of that we inject the health checker holder HealthContributorRegistry, and got the KafkaBinderHealthIndicator bean from it.
Example:
#Component
#Slf4j
#RequiredArgsConstructor
public class KafkaHealthChecker implements ComponentHealthChecker {
private final HealthContributorRegistry registry;
public String checkHealth() {
String status;
try {
BindersHealthContributor bindersHealthContributor = (BindersHealthContributor)
registry.getContributor("binders");
KafkaBinderHealthIndicator kafkaBinderHealthIndicator = (KafkaBinderHealthIndicator)
bindersHealthContributor.getContributor("kafka");
Health health = kafkaBinderHealthIndicator.health();
status = UP.equals(health.getStatus()) ? "OK" : "FAIL";
} catch (Exception e) {
log.error("Error occurred while checking the kafka health ", e);
status = "DEGRADED";
}
return status;
}
}
Is there a simple way to say if a consumer (created with spring boot and #KafkaListener) is operating normally?
This includes - can access and poll a broker, has at least one partition assigned, etc.
I see there are ways to subscribe to different lifecycle events but this seems to be a very fragile solution.
Thanks in advance!
You can use the AdminClient to get the current group status...
#SpringBootApplication
public class So56134056Application {
public static void main(String[] args) {
SpringApplication.run(So56134056Application.class, args);
}
#Bean
public NewTopic topic() {
return new NewTopic("so56134056", 1, (short) 1);
}
#KafkaListener(id = "so56134056", topics = "so56134056")
public void listen(String in) {
System.out.println(in);
}
#Bean
public ApplicationRunner runner(KafkaAdmin admin) {
return args -> {
try (AdminClient client = AdminClient.create(admin.getConfig())) {
while (true) {
Map<String, ConsumerGroupDescription> map =
client.describeConsumerGroups(Collections.singletonList("so56134056")).all().get(10, TimeUnit.SECONDS);
System.out.println(map);
System.in.read();
}
}
};
}
}
{so56134056=(groupId=so56134056, isSimpleConsumerGroup=false, members=(memberId=consumer-2-32a80e0a-2b8d-4519-b71d-671117e7eaf8, clientId=consumer-2, host=/127.0.0.1, assignment=(topicPartitions=so56134056-0)), partitionAssignor=range, state=Stable, coordinator=localhost:9092 (id: 0 rack: null))}
We have been thinking about exposing getLastPollTime() to the listener container API.
getAssignedPartitions() has been available since 2.1.3.
I know that you haven't mentioned it in your post - but beware of adding items like this to a health check if you then deploy in AWS and use such a health check for your ELB scaling environment.
For example one scenario that can happen is that your app loses connectivity to Kafka - your health check turns RED - and then elastic beanstalks begins a process of killing and re-starting your instances (which will happen continually until your Kafka instances are available again). This could be costly!
There is also a more general philosophical question on whether health checks should 'cascade failures' or not e.g. kafka is down so app connected to kafka claims it is down, the next app in the chain also does the same, etc etc. This is often more normally implemented via circuit breakers which are designed to minimise slow calls destined for failure.
You could check using the AdminClient for the topic description.
final AdminClient client = AdminClient.create(kafkaConsumerFactory.getConfigurationProperties());
final String topic = "someTopicName";
final DescribeTopicsResult describeTopicsResult = client.describeTopics(Collections.singleton(topic));
final KafkaFuture<TopicDescription> future = describeTopicsResult.values().get(topic);
try {
// for healthcheck purposes we're fetching the topic description
future.get(10, TimeUnit.SECONDS);
} catch (final InterruptedException | ExecutionException | TimeoutException e) {
throw new RuntimeException("Failed to retrieve topic description for topic: " + topic, e);
}
I am building an ASP.NET 5 (vNext) site that will host dynamic pages, static content, and a REST Web API. I have found examples of how to create middleware using the new ASP.NET way of doing things but I hit a snag.
I am trying write my own authentication middleware. I would like to create a custom attribute to attach to the controller actions (or whole controllers) that specifies that it requires authentication. Then during a request, in my middleware, I would like to cross reference the list of actions that require authentication with the action that applies to this current request. It is my understanding that I configure my middleware before the MVC middleware so that it is called first in the pipeline. I need to do this so the authentication is done before the request is handled by the MVC controller so that I can't prevent the controller from ever being called if necessary. But doesn't this also mean that the MVC router hasn't determined my route yet? It appears to me the determination of the route and the execution of that routes action happen at one step in the pipeline right?
If I want to be able to determine if a request matches a controller's action in a middleware pipeline step that happens before the request is handled by the controller, am I going to have to write my own url parser to figure that out? Is there some way to get at the routing data for the request before it is actually handled by the controller?
Edit: I'm beginning to think that the RouterMiddleware might be the answer I'm looking for. I'm assuming I can figure out how to have my router pick up the same routes that the standard MVC router is using (I use attribute routing) and have my router (really authenticator) mark the request as not handled when it succeeds authentication so that the default mvc router does the actual request handling. I really don't want to fully implement all of what the MVC middleware is doing. Working on trying to figure it out. RouterMiddleware kind of shows me what I need to do I think.
Edit 2: Here is a template for the middleware in ASP.NET 5
public class TokenAuthentication
{
private readonly RequestDelegate _next;
public TokenAuthentication(RequestDelegate next)
{
_next = next;
}
public async Task Invoke(HttpContext context)
{
//do stuff here
//let next thing in the pipeline go
await _next(context);
//do exit code
}
}
I ended up looking through the ASP.NET source code (because it is open source now!) and found that I could copy the UseMvc extension method from this class and swap out the default handler for my own.
public static class TokenAuthenticationExtensions
{
public static IApplicationBuilder UseTokenAuthentication(this IApplicationBuilder app, Action<IRouteBuilder> configureRoutes)
{
var routes = new RouteBuilder
{
DefaultHandler = new TokenRouteHandler(),
ServiceProvider = app.ApplicationServices
};
configureRoutes(routes);
routes.Routes.Insert(0, AttributeRouting.CreateAttributeMegaRoute(
routes.DefaultHandler,
app.ApplicationServices));
return app.UseRouter(routes.Build());
}
}
Then you create your own version of this class. In my case I don't actually want to invoke the actions. I will let the typical Mvc middleware do that. Since that is the case I gut all the related code and kept just what I needed to get the route data which is in actionDescriptor variable. I probably can remove the code dealing with backing up the route data since I dont think what I will be doing will affect the data, but I have kept it in the example. This is the skeleton of what I will start with based on the mvc route handler.
public class TokenRouteHandler : IRouter
{
private IActionSelector _actionSelector;
public VirtualPathData GetVirtualPath(VirtualPathContext context)
{
EnsureServices(context.Context);
context.IsBound = _actionSelector.HasValidAction(context);
return null;
}
public async Task RouteAsync(RouteContext context)
{
var services = context.HttpContext.RequestServices;
EnsureServices(context.HttpContext);
var actionDescriptor = await _actionSelector.SelectAsync(context);
if (actionDescriptor == null)
{
return;
}
var oldRouteData = context.RouteData;
var newRouteData = new RouteData(oldRouteData);
if (actionDescriptor.RouteValueDefaults != null)
{
foreach (var kvp in actionDescriptor.RouteValueDefaults)
{
if (!newRouteData.Values.ContainsKey(kvp.Key))
{
newRouteData.Values.Add(kvp.Key, kvp.Value);
}
}
}
try
{
context.RouteData = newRouteData;
//Authentication code will go here <-----------
var authenticated = true;
if (!authenticated)
{
context.IsHandled = true;
}
}
finally
{
if (!context.IsHandled)
{
context.RouteData = oldRouteData;
}
}
}
private void EnsureServices(HttpContext context)
{
if (_actionSelector == null)
{
_actionSelector = context.RequestServices.GetRequiredService<IActionSelector>();
}
}
}
And finally, in the Startup.cs file's Configure method at the end of the pipeline I have it setup so that I use the same routing setup (I use attribute routing) for the both my token authentication and mvc router.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
//Other middleware delcartions here <----------------
Action<IRouteBuilder> routeBuilder = routes =>
{
routes.MapRoute(
name: "default",
template: "{controller=Home}/{action=Index}/{id?}");
};
app.UseTokenAuthentication(routeBuilder);
//Middleware after this point will be blocked if authentication fails by having the TokenRouteHandler setting context.IsHandled to true
app.UseMvc(routeBuilder);
}
Edit 1:
I should also note that at the moment I am not concerned about the extra time required to select the route twice which is what I think would happen here since both my middleware and the Mvc middleware will be doing that. If that becomes a performance problem then I will build the mvc and authentication in to one handler. That would be best idea performance-wise, but what I have shown here is the most modular approach I think.
Edit 2:
In the end to get the information I needed I had to cast the ActionDescriptor to a ControllerActionDescriptor. I am not sure what other types of actions you can have in ASP.NET but I am pretty sure all my action descriptors should be ControllerActionDescriptors. Maybe the old legacy Web Api stuff needs another type of ActionDescriptor.
if i receive a HttpServerRequest in a Handler, is it somehow possible to publish the request?
I want to implement a small demo website with an index.html and an unknown number of sub sites. At first there should be a main vert.x module, which starts the HttpServer. In this main module it should be possible to add other dependent modules. I will call them submodules now. I don't know how many submodules i will have later, but each submodule should contain the logic to handle the http response for a specific URL (the sub html files). I guess i have to do the same for the WebSocketHandler...
A small example of the code inside the start():
//My Main Module:
vertx.createHttpServer().requestHandler(new Handler<HttpServerRequest>() {
public void handle(HttpServerRequest req) {
vertx.eventBus().publish("HTTP_REQUEST_CONSTANT", req);
}
}).listen(8080);
// My submodule 1
vertx.eventBus().registerHandler("HTTP_REQUEST_CONSTANT", new Handler<HttpServerRequest>() {
#Override
public void handle(HttpServerRequest req) {
if (req.uri().equals("/")) {
req.response();
}
}
});
// Other submodules which handles other URLs
Or any other solutions? I just don't wanna have the logic for sub sites in the main module.
Edit: Or could i call the vertx.createHttpServer() method in each submodule?
I have a similar Vert.x based application and I ended up doing the following:
I have a HttpServerVerticle that is started from the MainVerticle. There I created an HttpServer with several matchers. Each matcher receives a request and forwards it to a dedicated verticle through theEventBus. Upon getting the response from a dedicated verticle it writes the answer to the response.
Here is a code snippet:
RouteMatcher restMatcher = new RouteMatcher();
EventBus eventBus = vertx.eventBus();
HttpServer httpServer = vertx.createHttpServer();
restMatcher.post("/your/url",
r -> {
r.bodyHandler(requestBody -> {
final int length = requestBody.length();
if(length == 0) {
//return bad request
r.response().setStatusCode(HttpResponseStatus.BAD_REQUEST.code());
r.response().end();
return;
}
eventBus.send("your.address.here",
requestBody.getString(0, length),
new Handler<Message<JsonObject>>(){
#Override
public void handle(Message<JsonObject> message) {
//return the response from the other verticle
r.response().setStatusCode(HttpResponseStatus.OK.code());
if (message.body() != null) {
r.response().end(message.body().toString());
}
else{
r.response().end();
}
}
});
});
});
httpServer.requestHandler(restMatcher);
httpServer.listen(yourPort,yourHost);
In the dedicated verticle you register a listener to the address:
vertx.eventBus().registerHandler("your.address.here", this::yourHandlerMethod);
The handler method would look something like the following:
protected void yourHandlerMethod(Message<String> message){
// do your magic, produce an answer
message.reply(answer);
}
This way you separate your logic from your HTTP mappings and can have different pieces of logic in separate verticles using multiple event bus addresses.
Hope this helps.