So I've this Microservice architecture where there is an ApiGateway, 2 microservices i.e., Configurations. API and API-1. The Configuration. API is mainly responsible to parse the JSON request and
access the DB and update Status tables, also to fetch required data, it even adds up more values to the JSON request and send it to the API-1. API-1 is responsible to just generate report based on the json passed.
Yes I can merge the configurations. API to the API-1 and make it a single service/container but the requirement is not to merge and create two different components i.e., 1 component purely based on
fetching the data, updating the status while the other just to generate the reports.
So here are some questions:
: Should I use gRPC for the configuration.API or is there a better way to achieve this.
Thank you.
RPC is a synchronous communication so you have to come up with strong reason to use it in service to service communication. it brings the fast and performant communication on the table but also coupling to the services. if you insist use rpc it is better to use MASSTRANSIT to implement the rpc in less coupled way. however in most cases the asynchronous event-base communication is recommended to avoid coupling (in that case look at CAP theory, SAGA, circuit breaker ).
since you said
but the requirement is not to merege and create two different
components
and that is your reason and also base on the fact
also to fetch requried data, it even adds up more values to the JSON
request and send it to the API-1
i think the second one makes scenes more. how ever i cant understand why you change the database position since you said the configuration service is responsible for that.
if your report service needs request huge data to generate report you have to think about the design. there is no more profile on you domain so there cannot be an absolute answer to this. but consider data reduce from insertion or request or some sort of pre-calculation if you could and also caching responses.
For the last week I've been researching a lot on the microservice architecture pattern and its requirements and constraints.
The majority of ressources suggest to use event buses/message brokers (asynchronous communication) to communicate between microservices rather than using REST API endpoints.
Synchronous calling would result in a higher response time and may cause cascading failure in case of a particular microservice failing in the chain.
Question:
Let's say the user requests a particular functionality or page on a website/mobile app which then needs to fetch data from multiple microservices and use theire respective functionalities to provide the desired outcome. But to achieve the desired outcome (response to client) ALL the services need to do their work before the backend sends the response back to the client (website/mobile app).
But if we use asynchronous service requests - which means the calling service doesnt wait for a response and would send its own response back to the client without getting the data from the asynchronously called service - the outcome might not be complete if an asychronously called service doesnt respond in time (service is unavailable or network issues). This would mean that the backend will send an incomplete response back to the client which is not acceptable.
How can I deal with this issue or did I get the concept wrong?
I'm thankful for every answer
If it's absolutely essential that a request gets a full response (i.e. that the request is synchronous), that's a strong argument in favor of the service stitching together synchronous requests and responses (and potentially needing to handle rollback in cases of partial success etc.).
Many requests don't fall into that pattern, though. For instance, a response might well be interpretable as "we've received your request and the operation will be performed. You can track the progress of your operation by using this request ID"; such an approach fits well with asynchronous messaging.
Here is my use case:
My Service is: ServiceA.
It depends on the following services: ServicesB and ServiceC.
ServiceA sends a POST request to ServiceB with some authentication details (username+password) and ServiceB replies back with a json document which has a sessionId.
Request:
POST /authenticate
{
"username": "_at_api",
"password": "xxx"
}
Response:
{
"sessionId": "axy235da7ad5a24abeb3e7fbb85d0ef45f"
}
The above sessionId is used for all api calls from ServiceA to ServiceC.
ServiceA requests to serviceC to start a job using a POST request and serviceC returns with a job id(alphanumeric).
Request:
POST /jobs/local/start
Header: Authentication: axy235da7ad5a24abeb3e7fbb85d0ef45f
{
...
}
Response:
{
"status": "RUNNING",
"jobId": "a209016e3fdf4425ea6e5846b8a46564abzt"
}
ServiceA keeps polling serviceC for the completion of the job using the jobId returned above:
Request:
GET /jobs/status/a209016e3fdf4425ea6e5846b8a46564abzt
Header: Authentication: axy235da7ad5a24abeb3e7fbb85d0ef45f
Response:
{
"status": "RUNNING"
}
The polling continues until the status is returned as COMPLETED or FAILED.
Response:
{
"status": "COMPLETED"
}
How can I use Pact to test serviceA?
My plan is to use only unit tests and contract tests to achieve code coverage of more than 90%. Is it a good idea, or do I need to have a separate tests using virtual servers? My understanding is that Pact is a superset of virtual server (example: mountebank) and everything which a virtual server can do, Pact can do. And so I do not need a separate Component testing. Also, it looks like Contract testing completely replaces end-to-end testing, so I do not need end-to-end testing as well. Is this right?
Also, it looks like Contract testing completely replaces end-to-end testing, so I do not need end-to-end testing as well. Is this right?
No. Contract testing is not functional testing (see this excellent article, with the same title)
What is contract testing?
Contract testing is about testing whether two components are able to communicate.
Consider a contract between a house and a postal worker: The postal worker needs to know that they can approach the house and deliver post (and, that sometimes they may be unable to do this - perhaps the mailbox is full).
From the postal worker's perspective, the contract looks like this:
Find postbox (with a success and fail case)
Deliver post to postbox (with a success and fail case)
Note that the postal worker doesn't know anything about the implementation of the postbox. Perhaps there are multiple reasons that delivering to the postbox might fail - maybe the door is jammed, maybe the box is full, maybe
the post is too big to fit in it.
In this hypothetical case, our postal worker doesn't do anything different in those cases - they just fail to deliver. So, from the perspective of the contract, the reason for the failure is irrelevant. The contract - that the worker can try to deliver post, and that they can be successful or unsuccessful - can be tested without enumerating all the possible reasons for failure.
See the article linked above for a more detailed example, but to quote from the end of it:
Contracts should be about catching:
bugs in the consumer
misunderstanding from the consumer about end-points or payload
breaking changes by the provider on end-points or payload
A really nice feature of Pact is that you can test multiple contracts against only the bits of communication that they rely on.
Note that the consumer contract tests only describe communication that the consumer needs to make or understand. A contract is not (necessarily) a full API description.
Ok, but why can't I use contract tests for end to end tests?
It's possible to use a tool like Pact to replace your end-to-end tests. However, although contract testing has a lot of similarities with the features you'd need for end-to-end testing, contract testing (and Pact in particular) isn't designed for end-to-end testing.
If you're doing end-to-end testing by extending an existing consumer's tests (say, adding all the possible reasons to failure to the postworker's tests), then it's no longer clear what the contract means. The contract now describes how the communication works along with the behaviour.
This will cause problems when you start adding more consumers (say, a parcel courier) - do you duplicate all of the failure cases in all the consumers, or do you just keep them in the original consumer tests? If you duplicate the tests, then you have a lot of things to change if you change the behaviour of the provider - and your tests will be brittle. If you don't duplicate the tests, then your end-to-end tests are stuck in one consumer - with all the problems of losing them if you decommission that consumer.
With pure contract tests, you (ideally) don't have to change anything if you're adding more possible reasons for failure that the consumers already understand.
There are many other reasons that you'll have headaches if you try this (your tests start relying heavily on exact data, and the meaning of failed verifications and can-i-deploy hooks would change if your tests are end-to-end tests), but the key takeaway is that Pact is not designed as a replacement for end-to-end testing. You can use it that way, but it's not advisable and is likely to lead to painful maintenance.
How can I use Pact to test serviceA?
You describe each request separately, using Pact provider state as the prerequisite for each request.
Additionally, you may find the question on PACT - Using provider state helpful.
Although this comes late, for everyone who stumbles across this:
The pact spec (v3.0) only supports a single request per interaction (but multiple interactions per consumer-provider-pair). On the provider side, every interaction results in an individual test. So while having a sequence or requests running on the provider side is not conceptually not a good idea, it also doesn't work technically.
On the consumer side though, you have an alternative - which I would encourage not to misuse for end-to-end tests. In my case, I had a class which implemented a template/strategy pattern which involved multiple requests and didn't allow injecting intermediate states on the consumer side. It required all requests to have valid responses.
In that case, pact-jvm-consumer-junit (=JUnit4) allows to specify multiple PactVerification annotations on a single test method (Kotlin in my case):
#Test
#PactVerifications(
value = [
PactVerification("Service B", fragment = "authenticateOk"),
PactVerification("Service C", fragment = "jobStartOk")
PactVerification("Service C", fragment = "jobStatusOk")
]
)
fun `test successful job execution`() {
// add your test here.
}
On the provider side, all fragments above are executed as individual tests, thus they need a proper state as specified in the answer above.
I got this approach running in JUnit4. Haven't found a quick way to mimic it JUnit5 though.
I have read that on Pact, the consumer is the one initiating the request. I have a service (let's call it A) that is used to draw pictures that will then be submitted (via POST) to a service (let's call it B) that will process those inputs and generate albums.
My question is: If the service B is the expert on knowing which kind of inputs and in which format should receive in order to create albums, how come is service A the consumer and therefore the one that will be writing the contract? Shouldn't it be the service B that specifies what kind of data should be receiving?
From [1]:
A component that initiates a HTTP request to another component (the service Provider). Note that this does not depend on the way the data flows - whether it is a GET or a PUT / POST / PATCH, the Consumer is the initiator of the HTTP request.
But I think your question is really asking about consumer driven contract testing [2]. You're conflating who is building the service with expertise / understanding of the requirements for its existence. The Consumer knows what it needs from the service, and the Provider knows how to implement it.
Consumer driven contracts reverses the typical API design process of creating an API first and then consumers coming along to use it. First, the consumer specifies what it requires, publishes the expectations as a contract and the Provider then implements it.
Pact generally assumes this mode, but it's not strictly necessary (i.e. the Provider API can exist in advance).
[1] https://docs.pact.io/documentation/how_does_pact_work.html
[2] https://martinfowler.com/articles/consumerDrivenContracts.html
After reading a lot about the differences between REST and SOAP, I got the impression that REST is just another word for HTTP. Can someone explain what functionality REST adds to HTTP?
Note: I'm not looking for a comparison of REST versus SOAP.
No, REST is the way HTTP should be used.
Today we only use a tiny bit of the HTTP protocol's methods – namely GET and POST. The REST way to do it is to use all of the protocol's methods.
For example, REST dictates the usage of DELETE to erase a document (be it a file, state, etc.) behind a URI, whereas, with HTTP, you would misuse a GET or POST query like ...product/?delete_id=22.
HTTP is a protocol used for communication, usually used to communicate with internet resources or any application with a web browser client.
REST means that the main concept you are using while designing the application is the Resource: for each action you want to perform you need to define a resource on which you often do only CRUD operation, which is a simple task. For that it's very convenient to use four verbs used in HTTP protocol against the four CRUD operations (GET for Read, POST is for CREATE, PUT is for UPDATE and DELETE is for DELETE).
That's unlike the older concept of RPC (Remote Procedure Call), in which you have a set of actions you want to perform as a result of the user's call. if you think for example on how to describe a Facebook like on a post, with RPC you might create services called AddLikeToPost and RemoveLikeFromPost, and manage it along with all your other services related to FB posts, thus you won't need to create special object for Like.
With REST you will have a Like object which will be managed separately with Delete and Create functions. It also means it will describe a separate entity in your DB. That might look like a small difference, but working like that would usually yield a much simpler code and a much simpler application. With that design, most of the app's logic is obvious from the object's structure (model), unlike RPC with which you would usually have to explicitly add a lot more logic.
Designing a RESTful application is often a lot harder because it requires you to describe complicated things in a simple manner. Describing all functionalities using only CRUD functions is tricky, but after doing that your life would be a lot simpler, and you will find that you write a lot shorter methods.
One more restraint REST architecture presents is not to use a session context when communicating with a client (stateless), meaning all the information needed to understand who is the client and what he wants is passed with the web message. Each call to a function is self-descriptive, there is no previous conversation with the client which can be referenced in the message. Therefore, a client could not tell you "give me the next page" since you don't have a session to store what is the previous page and what kind of page you want, the client would have to say "my name is Yuval, get me page 2 of a specific post in a specific forum". This means a bit more data would have to transfer in the communication, but think of the difference between finding a bug reported from the "get me the next page" function in oppose to "get me page 2 of question ID 2190836 in stack overflow".
Of course there is a lot more to it, but to my humble opinion these are the main concepts in a teaspoon.
REST enforces the use of the available HTTP commands as they were meant to be used.
For example, I could do:
GET
http://example.com?method=delete&item=xxx
But with rest I would use the "DELETE" request method, removing the need for the "method" query param
DELETE
http://example.com?item=xxx
HTTP is an application protocol. REST is a set of rules, that when followed, enable you to build a distributed application that has a specific set of desirable constraints.
If you are looking for the most significant constraints of REST that distinguish a RESTful application from just any HTTP application, I would say the "self-description" constraint and the hypermedia constraint (aka Hypermedia as the Engine of Application State (HATEOAS)) are the most important.
The self-description constraint requires a RESTful request to be completely self descriptive in the users intent. This allows intermediaries (proxies and caches) to act on the message safely.
The HATEOAS constraint is about turning your application into a web of links where the client's current state is based on its place in that web. It is a tricky concept and requires more time to explain than I have right now.
HTTP is a contract, a communication protocol and REST is a concept, an architectural style which may use HTTP, FTP or other communication protocols but is widely used with HTTP.
REST implies a series of constraints about how Server and Client should interact. HTTP is a communication protocol with a given mechanism for server-client data transfer, it's most commonly used in REST API just because REST was inspired by WWW (world wide web) which largely used HTTP before REST was defined, so it's easier to implement REST API style with HTTP.
There are three major constraints in REST (but there are more):
Interaction between server and client should be described via hypertext only.
Server and client should be loosely coupled and make no assumptions about each other. Client should only know resource entry point. Interaction data should be provided by the server in the response.
Server shouldn't store any information about request context. Requests must be independent and idempotent (means if same request is repeated infinitely, exactly same result is retrieved)
And HTTP is just a communication protocol (a tool) that can help to achieve this.
For more info check these links:
https://martinfowler.com/articles/richardsonMaturityModel.html
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
REST = Representational State Transfer
REST is a set of rules, that when followed, enable you to build a distributed application that has a specific set of desirable constraints.
REST is a protocol to exchange any(XML, JSON etc ) messages that can use HTTP to transport those messages.
Features:
It is stateless which means that ideally no connection should be maintained between the client and server.
It is the responsibility of the client to pass its context to the server and then the server can store this context to process the client's further request. For example, session maintained by server is identified by session identifier passed by the client.
Advantages of Statelessness:
Web Services can treat each method calls separately.
Web Services need not maintain the client's previous interaction.
This in turn simplifies application design.
HTTP is itself a stateless protocol unlike TCP and thus RESTful Web Services work seamlessly with the HTTP protocols.
Disadvantages of Statelessness:
One extra layer in the form of heading needs to be added to every request to preserve the client's state.
For security we need to add a header info to every request.
HTTP Methods supported by REST:
GET: /string/someotherstring
It is idempotent and should ideally return the same results every time a call is made
PUT:
Same like GET. Idempotent and is used to update resources.
POST: should contain a url and body
Used for creating resources. Multiple calls should ideally return different results and should create multiple products.
DELETE:
Used to delete resources on the server.
HEAD:
The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The meta information contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request.
OPTIONS:
This method allows the client to determine the options and/or requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval.
HTTP Responses
Go here for all the responses.
Here are a few important ones:
200 - OK
3XX - Additional information needed from the client and url redirection
400 - Bad request
401 - Unauthorized to access
403 - Forbidden
The request was valid, but the server is refusing action. The user might not have the necessary permissions for a resource, or may need an account of some sort.
404 - Not Found
The requested resource could not be found but may be available in the future. Subsequent requests by the client are permissible.
405 - Method Not Allowed
A request method is not supported for the requested resource; for example, a GET request on a form that requires data to be presented via POST, or a PUT request on a read-only resource.
404 - Request not found
500 - Internal Server Failure
502 - Bad Gateway Error
Not quite...
http://en.wikipedia.org/wiki/Representational_State_Transfer
REST was initially described in the
context of HTTP, but is not limited to
that protocol. RESTful architectures
can be based on other Application
Layer protocols if they already
provide a rich and uniform vocabulary
for applications based on the transfer
of meaningful representational state.
RESTful applications maximise the use
of the pre-existing, well-defined
interface and other built-in
capabilities provided by the chosen
network protocol, and minimise the
addition of new application-specific
features on top of it.
http://www.looselycoupled.com/glossary/SOAP
(Simple Object Access Protocol) The
standard for web services messages.
Based on XML, SOAP defines an envelope
format and various rules for
describing its contents. Seen (with
WSDL and UDDI) as one of the three
foundation standards of web services,
it is the preferred protocol for
exchanging web services, but by no
means the only one; proponents of REST
say that it adds unnecessary
complexity.
REST is a specific way of approaching the design of big systems (like the web).
It's a set of 'rules' (or 'constraints').
HTTP is a protocol that tries to obey those rules.
From You don't know the difference between HTTP and REST
So REST architecture and HTTP 1.1 protocol are independent from each
other, but the HTTP 1.1 protocol was built to be the ideal protocol to
follow the principles and constraints of REST. One way to look at the
relationship between HTTP and REST is, that REST is the design, and
HTTP 1.1 is an implementation of that design.
HTTP is a communications protocol that transports messages over a network.
SOAP is a protocol to exchange XML-based messages that can use HTTP to transport those messages.
Rest is a protocol to exchange any(XML or JSON) messages that can use HTTP to transport those messages.
REST is not necessarily tied to HTTP. RESTful web services are just web services that follow a RESTful architecture.
What is Rest -
1- Client-server
2- Stateless
3- Cacheable
4- Layered system
5- Code on demand
6- Uniform interface
REST APIs must be hypertext-driven
From Roy Fielding's blog here's a set of ways to check if you're building a HTTP API or a REST API:
API designers, please note the following rules before calling your creation a REST API:
A REST API should not be dependent on any single communication protocol, though its successful mapping to a given protocol may be dependent on the availability of metadata, choice of methods, etc. In general, any protocol element that uses a URI for identification must allow any URI scheme to be used for the sake of that identification. [Failure here implies that identification is not separated from interaction.]
A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of underspecified bits of standard protocols, such as HTTP’s PATCH method or Link header field. Workarounds for broken implementations (such as those browsers stupid enough to believe that HTML defines HTTP’s method set) should be defined separately, or at least in appendices, with an expectation that the workaround will eventually be obsolete. [Failure here implies that the resource interfaces are object-specific, not generic.]
A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC’s functional coupling].
A REST API should never have “typed” resources that are significant to the client. Specification authors may use resource types for describing server implementation behind the interface, but those types must be irrelevant and invisible to the client. The only types that are significant to a client are the current representation’s media type and standardized relation names. [ditto]
A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]
REST is a light version of SOAP, with no sugars or colors agents added.
The goal of both SOAP and REST is to allow the communication between Information Systems that may be written in different languages and use different communiation protocols.
While SOAP uses API contracts to expose it's services and define the way a client can call a service, what Parameters should be sent and what results are to be expected, REST on the other hand has uses no API contracts, for a client to know what services exist and how to call them it should look into the Rest API documentation (this can be defined in a yml file with a OpenAPI or Swagger).
Second SOAP is verbose, it reyes on XML to send a request and describe the services, parameteres and the results returned. On the other hand REST relyed on simple JSON Objects to send requests and receive results. JSON is simple to undersand, lightweight and does not use too much bandwith when sending requests or receiving results.