We're building a microservice app where clients can create projects. The following diagram shows the technical flow of this process:
My question: what HTTP response should the API gateway return to the client (step 1.)?
My initial idea was to give back a 202, but the problem there is that I don't know the Location yet (/projects/{id}), because the id of the project will be created at the Project Management Service.
Considering that the IDs of the newly created project entity is not known at the request time (i.e. it is generated after the insertion into the database) you indeed cannot generate the url to the project resource.
Instead, you could assign an ID (i.e. 1234-abcd-5678-efgh) to the command before sending to the bus and keep track of its execution status on the API gateway itself. Then you can respond to the client with an command execution status endpoint like /commands/1234-abcd-5678-efgh where it can query by polling.
The alternative would be to use another service that would reserve&deliver unique IDs but you must make a blocking call to it and this hurts scalability. Or you can host this service inside the API gateway itself (onto the same node) to minimize latency. Also, there is a risk of loosing some IDs in case of project creation failures but this can be compensated by releasing those IDs in those situations (thus increasing the architecture complexity).
A third solution could be the use of a project surogate ID, like a GUID, assigned as a property of the project, included in the command, having the purpose of an alternate identity that can be used only in the pre-creation phase of the process. Then, the response to the client could be like this: /projects/by-guid/1234-abcd-5678-efgh and after the project is created a GET to this url would permanently redirect to the final project url.
Related
I am stuck in a problem while using Kafka in a microservice architecture . I am not able to understand how a microservice handling HTTP requests will be able to send a response to the user. I want to take data from HTTP and then publish it to topic named A then another validator service will validate it and publish it on another topic named B. I want to send processed data to HTTP response from subscribed data from topic B.
In my experience, one option is to respond immediately with 201 accepted, or embed a blocking validator library into your API, and properly return a 400 Bad Request.
Then future GET calls are required to read eventually consistent data that might come back from any consumer. E.g. the consumer process is Kafka Connector that writes to some database, or is a Kafka Streams/KSQL table, future API queries will return data from that. Your original client may need to make periodic HTTP calls until that data is available.
I am using the Pact framework to test some APIs from a service. I have one API that initiates some backend execution. Let's call it request A and the response returns a unique execution ID. The second API (request B) send the execution ID returned from request A to pull the execution status. How do I set up the pact test in this case? The problem is the execution ID that is generated dynamically. I know a provider can inject some provider state to the consumer. So potentially, the execution ID could be injected. But I am not sure how to make the injection from the provider side. It requires access to the response from the response A and inject the execution ID (with the provider state callback, perhaps) for the second request.
You need to have a lot of control over what is happening in your provider for Pact to work for you.
Each interaction is verified individually (and in some frameworks, in a random order), and all state should be cleared in between interactions, so you need to use provider states to set up any data that would have been created by the initial request. In regards to something like the execution IDs, you could use a different implementation of the code that generates the IDs that you only use for Pact Tests.
There is a bank which creates a contract which is then accepted by the lender and the borrower. After signing the contract the lender provides fund to the borrower. The bank then creates an obligation state based on the data received by calling an external service automatically.
And Now
1) In API Layer, I am calling first flow which creates one state.
2) In API layer itself, On success of first flow , I am calling the http request to external service and get the data.
3) Now I pass the http response to the the second flow for creating the other state.
Can you please let me know if there is any issue with this approach.
Requirment is I want to trigger the first flow manually, but calling external service and initiating the second flow should happen automatically
I had referred the link given below.
Making asynchronous HTTP calls from flows
You'll make calls to an external service during the running of flows.
The best place to get started would be looking at the CorDapp samples here. In particular, take a look at the Accessing External Data section
I have following scenario:
User request for certain resource on server, This request is long running task and very like 2~3 seconds to 10 seconds. We issue a JobTicket to user, As our user want to wait.
On receiving request we store that request in persistence storage and issue a token to user as JobTicket (GUID).
User make connection with Hub to get information about that GUID.
In Background:
We have WAS Hosted as well as Windows Service to perform some operation on that request.
On complete, WAS Hosted/Windows Service call our Web Application that job has been completed.
From there based on job Ticket we identify which user and on its connection we let user know its job has been completed.
Now we have farm of servers, we are using Windows Server On Prem ServiceBus 1.1 which is working fine, But challenge we have is that we are not able to intercept ServiceBus based backplane message broadcast and message is going to all the client. As we have farm, user intermediately may have drop connection and connected to other server based on load balancer so we need to have scale out using Service Bus as its kind of seamless to integrate and we are also using for our internal purpose in our application so we don't want to user any other mix in complex solution.
I have tried using IHubPipelineModule but still Scale out message broadcast not passing thru that, I tried to hookup SignalR code directly and debug thru it but its taking long. I don't want to mess-up something arbitrary in actual code. As I can see that in OnReceive I can see message are coming, but not able to follow further. I just need small mechanism that I can intercept broadcast message and make sure that it goes to client it intended and not all the client by wasting resources, and security concern as well.
Please help me on this issue, it's kind of stuck from last 4 days and not able to come to any solution and same time I want to go with establish pattern and don't want to fork any special build for this kind of small issues which I am sure one of you expert knows how I can do that seamlessly.
Thanks,
Shrenik
After lots of struggling and not finding straight forward way, I have found the way as below for someone else in future it might help.
Scenario:
1. Web Farm: Host External User facing Web Pages
2. Backend Process: Which is mix of WebApi, SharePoint, Windows Service etc.
User from Web Page submit some request and get a unique id as return back. Internally on receiving request, we queue that request to Service Bus using TopicClient for processing.
There are pool of Windows Service watching on Message on Service Bus using SubscriptionClient and process that message. On completion of process which can run from 5 seconds to 30 seconds and some cases even more. We need to inform client that its job done if its waiting on web page or waiting for completion notification.
In this story, We are using SignalR to push job completion notification to client.
Now my earlier problem is How I let know from windows service to web application that job is done so send notification to client who submitted request.
One way is we hosted another hub internally in web application, Windows service act as client and call web application hosted hub, and in that hub method it will call external facing hub method to propagate message to specific client who submitted request, for which we are using Single user Group.
And as we have register service bus as backplane it will propagate to other servers and then appropriate client will get notification. So this is ideal solution and should work in most cases.
In above approach we have one limitation that, how Windows Service connect to Web Client, as we donot have windows auth, but we have openid based auth with ADFS. Now in such case Web Application required special code in which provide separate userid or password for windows service to communicate or have windows authentication also allowed for that hub for service account of windows service.
I was trying and trying how to remove all this hopes between interserver communication and again management of extra security.
So I did below with simplicity, though it tooks me whole night to find our internal of SignalR. But it works:
Approach is to send message directly to ServiceBus Backplane, and as all Web Server already hooked-up with ServiceBus backplane then they will get message.
Unfortunately SignalR doesn't provide such mechanism to send message directly to Backplane. I think its on pub/sub model so they don't want somebody to hack in their system :). or its violation of their pattern, but its make sense, in my case because of different roles and security, I have simplify code as below:
Create a ServiceBusMessageBus instance in my code, Same way as Below: Though I have created separate instance and store till lifetime of Windows Service, so I don't create instance every time:
ServiceBusMessageBus serviceBusBackplane = new ServiceBusMessageBus(new DefaultDependencyResolver(), new ServiceBusScaleoutConfiguration(connectionString, appName));
Create a ClientHubInvocation Object: This is a message which actually get created in SignalR infrastructure when Backplane based message broadcast:
ClientHubInvocation hubData = new ClientHubInvocation
{
Args = new object[] { msg },
Hub = "JobStatusHub",
Method = "onJobStatus",
State = null,
};
Create a Message object which accept by ServiceBusMessageBus.Publish, Yes, so this is a method which actually get called on base class ScaleoutMessageBus.Publish. This class is actually responsible for sending message to topic and other subscribers on other server nodes. Why not use that directly. Now to create Message Object, You need following code:
Message backplaneMessage = new Message(
sourceId,
"hg-JobStatusHub." + name,
new ArraySegment(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(hubData))));
In above second parameter is something interesting,
In case if you want to publish to all the client then syntax is "h-", in my case specific group user, so syntax is "hg-.. You can check the code here: https://github.com/SignalR/SignalR/blob/bc9412bcab0f5ef097c7dc919e3ea1b37fc8718c/src/Microsoft.AspNet.SignalR.Core/Infrastructure/PrefixHelper.cs
Publish your message to backplane directly as below:
await serviceBusBackplane.Publish(backplaneMessage);
I wish this PrefixHelper class have been public.
Remember: This is not recommended way and doent insulate from future upgrade for SignalR, as its internal they may change so any upgrade might come with small hazale to change this code. But in summary this works. Hope SignalR Team provide some mechanisam out of box to send message directly to backplane instead.
Thanks
I am designing a Biztalk solution which requires client applications to subscribe and receive only a certain subset of event messages depending on their user permissions. Subscription will be done through topic or content based routing. The client will subscribe once and receive many messages until they choose to unsubscribe.
Client applications will number in the 100s and subscribed topics could change on a regular basis, so defining an individual send port from Biztalk for each reciever isn't a viable solution.
I have thought I could build an additional message broker service which holds the individual client subscriptions and distributes messages sent from a biztalk port.
I have also seen that a recipient list pattern can be build using orchestrations. This appears to me to still follow a request-response pattern though and I am after 1 way subscribe message to many returned event messages.
My message broker solution seems to me to be doubling up on what Biztalk should be good at so I imagine I am missing some important functionality somewhere. Has anyone tried such an application before and can give some pointers? Should I be investingating the ESB toolkit as a solution? I have had a look on the net but nothing makes it very clear for this type of topic-subscription model.
Thanks,
Phil
Do take a look at the ESB Toolkit. You can use the itinerary functionality that it adds to BizTalk, either with one of the built-in resolvers (e.g., UDDI) or with your own custom resolver. This allows you to route messages based on configuration (stored in Business Rules or elsewhere).
You will find a developer-oriented overview video of the ESB Toolkit on MSDN, which is a decent introduction to the design process and tooling. There are several other helpful videos there as well.
Your specific scenario can accomplished with a single itinerary, as described here. Use a receive pipeline with the ESB Dispatch Disassembler component, configure multiple resolvers, and for each resolver a new message is produced.
There are also two samples to look at:
The Itinerary On-Ramp Sample - builds a set of SOAP headers that contain the itinerary that you create in the test client, loads the specific message file from disk, appends the itinerary headers to the message, and submits it to the ESB through an Itinerary on-ramp for processing.
The Scatter-Gather Sample - Also appends SOAP headers containing the itinerary to the message, which is submitted to the ESB through an on-ramp for processing. A Broker orchestration analyzes the settings for its itinerary step, retrieves a collection of resolvers associated with the itinerary step, and for each of those resolvers resolves the service endpoint. After that, the orchestration activates the proper ServiceDispatcher orchestration instances to dispatch the outbound request messages.
You should also look at "How to: Route a Single Message to Multiple Recipients Using an Itinerary Routing Slip" or perhaps look into creating a custom itinerary message service (documentation is here).