I want know about good practices with golang and gRPC and protobuf.
I am implementing the following gRPC service
service MyService {
rpc dosomethink(model.MyModel) returns (model.Model) {
option (google.api.http) = { post: "/my/path" body: "" };
}
}
I compiled the protobufs. In fact, the protobuf give us a httpproxy from http to grpc.
The code to implement this service:
import "google.golang.org/grpc/status"
func (Abcd) Dosomethink(c context.Context, sessionRequest *model.MyModel) (*model.Model, error) {
return nil, status.New(400,"Default error message for 400")
}
I want a 400 http error (in the http proxy) with the message "Default error message for 400", the message works, but the http error always is 500.
Do you know any post or doc about this?
You need to return empty model.Model object in order for protobufs to be able to properly serialise the message.
Try
import "google.golang.org/grpc/status"
func (Abcd) Dosomethink(c context.Context, sessionRequest *model.MyModel) (*model.Model, error) {
return &model.Model{}, status.Error(400,"Default error message for 400")
}
Error Handler:
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
return data, status.Errorf(
codes.InvalidArgument,
fmt.Sprintf("Your message", req.data),
)
For need more info about the error handling take a look below links.
https://grpc.io/docs/guides/error.html
http://avi.im/grpc-errors/
Related
I'm actually dealing with some issues.
In fact I'm coding an HTTP server with Deno and I have the same problem that I did with Node.js.
I use Postman to simulate POST request to my server.
When I do a Request with the URL of my localhost the server is working:
The deno server code :
const listener = Deno.listen({ port: 8000 });
console.log(`Server is running on http://localhost:8000`);
for await (const conn of listener) {
handleNewConnection(conn);
}
async function handleNewConnection(conn: Deno.Conn) {
for await (const req of Deno.serveHttp(conn)) {
await handleRequest(req.request, req.respondWith);
}
}
async function handleRequest(req: Request, resp: any) {
if (req.method === "POST") {
let body = await req.json();
console.log(body);
resp(new Response(null));
} else {
console.log(req.method)
resp(new Response(null));
}
}
The request that I'm sending in localhost:8000:
What I receive in console.log :
{
name: "Molecule Man",
age: 29,
secretIdentity: "Dan Jukes",
powers: [ "Radiation resistance", "Turning tiny", "Radiation blast" ]
}
Then I'm gonna show you when I use a Cloudflare tunnel.
This Cloudflare tunnel is directly sending the requests to my server on port 8000.
The POST request that I'm sending :
What I receive with the console.log() :
error: Uncaught (in promise) SyntaxError: Unexpected end of JSON input
let body = await req.json();
^
at parse (<anonymous>)
at packageData (deno:ext/fetch/22_body.js:328:16)
at Request.json (deno:ext/fetch/22_body.js:267:18)
at async handleRequest (file:///C:/Users/Utilisateur/Projets/poc-pdu/HTTP-serv-deno/http-serv-deno.ts:17:16)
at async handleNewConnection (file:///C:/Users/Utilisateur/Projets/poc-pdu/HTTP-serv-deno/http-serv-deno.ts:10:5)
That's the error, in my opinion, this is due to the time it takes for the request to arrive at it's destination.
I thank all the people who will take the time to respond to this post.
Bye!
I want to watch for every events in a Gitlab project and store them in an external service. For this, I use Gitlab Webhooks. I made a little local HTTP server in Go that listens for Gitlab's POSTs and forward them to an external service. Hooks contains every information I needed so it seems that this architecture is fine:
Gitlab > HTTPServer > External Service.
My problem is when the external service is down, I cannot manage to make Gitlab retry the failed requests. As the documentation says:
GitLab ignores the HTTP status code returned by your endpoint.
Your endpoint should ALWAYS return a valid HTTP response. If you do not do this then GitLab will think the hook failed and retry it.
It is very surprising that Gitlab does not have a proper way to ask for a webhook retry. I have to explicitly return an invalid http response. Moreover, I cannot find an API endpoint to list all failed webhooks and ask for resend.
Question: How to explicitly return an invalid HTTP response with the standard "net/http" library in order to force Gitlab to retry Webhooks?
As written in the comments, a webhook is a mere notification that an event occurred, and potentially some data is sent, typically as JSON data.
It is your responsibility to persist the event itself and the data you want/need to process that was sent with it. Below you will find a commented example. Note that this does not include incremental backoffs, but that should be easy to add:
package main
import (
"encoding/json"
"flag"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/joncrlsn/dque"
)
var (
bind string
queueDir string
segmentSize int
)
// You might want to add request headers and stuff
type webhookContent struct {
Foo string
Bar int
}
func init() {
flag.StringVar(&bind, "bind", ":8080", "The address to bind to")
flag.StringVar(&queueDir, "path", "./queue", "path to store the queue in")
flag.IntVar(&segmentSize, "size", 50, "number of entries for the queue")
}
// The "webserver" component
func runserver(q *dque.DQue) {
http.HandleFunc("/webhook", func(w http.ResponseWriter, r *http.Request) {
// A new decoder for each call, as we want to have a new LimitReader
// for each call. This is a simple, albeit a bit crude method to prevent
// accidental or malicious overload of your server.
dec := json.NewDecoder(io.LimitReader(r.Body, 4096))
defer r.Body.Close()
c := &webhookContent{}
if err := dec.Decode(c); err != nil {
log.Printf("reading body: %s", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// When the content is successfully decoded, we can persist it into
// our queue.
if err := q.Enqueue(c); err != nil {
log.Printf("enqueueing webhook data: %s", err)
// PROPER ERROR HANDLING IS MISSING HERE
}
})
http.ListenAndServe(bind, nil)
}
func main() {
flag.Parse()
var (
q *dque.DQue
err error
)
if !dirExists(queueDir) {
if err = os.MkdirAll(queueDir, 0750); err != nil {
log.Fatalf("creating queue dir: %s", err)
}
}
if !dirExists(filepath.Join(queueDir, "webhooks")) {
q, err = dque.New("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
} else {
q, err = dque.Open("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
}
if err != nil {
log.Fatalf("setting up queue: %s", err)
}
defer q.Close()
go runserver(q)
var (
// Placeholder during event loop
i interface{}
// Payload
w *webhookContent
// Did the type assertion succeed
ok bool
)
for {
// We peek only. The semantic of this is that
// you can already access the next item in the queue
// without removing it from the queue and "mark" it as read.
// We use PeekBlock since we want to wait for an item in the
// queue to be available.
if i, err = q.PeekBlock(); err != nil {
// If we can not peek, something is SERIOUSLY wrong.
log.Fatalf("reading from queue: %s", err)
}
if w, ok = i.(*webhookContent); !ok {
// If the type assertion fails, something is seriously wrong, too.
log.Fatalf("reading from queue: %s", err)
}
if err = doSomethingUseful(w); err != nil {
log.Printf("Something went wrong: %s", err)
log.Println("I strongly suggest entering an incremental backoff!")
continue
}
// We did something useful, so we can dequeue the item we just processed from the queue.
q.Dequeue()
}
}
func doSomethingUseful(w *webhookContent) error {
log.Printf("Instead of this log message, you can do something useful with: %#v", w)
return nil
}
func dirExists(path string) bool {
fileInfo, err := os.Stat(path)
if err == nil {
return fileInfo.IsDir()
}
return false
}
Now when you do something like:
$ curl -X POST --data '{"Foo":"Baz","Bar":42}' http://localhost:8080/webhook
you should get a log entry like
2020/04/18 11:34:23 Instead of this log message, you can do something useful with: &main.webhookContent{Foo:"Baz", Bar:42}
Note that See GitLab 15.7 (December 2022) implements an opposite approach:
Automatic disabling of failing webhooks
To protect GitLab and users across the system from any potential abuse or misuse, we’ve implemented a feature to disable webhooks that fail consistently.
Webhooks that return response codes in the 5xx range are understood to be failing intermittently and are temporarily disabled. These webhooks are initially disabled for 1 minute, which is extended on each retry up to a maximum of 24 hours.
Webhooks that fail with 4xx errors are permanently disabled.
All project owners and maintainers are alerted in the app to investigate and re-enable any failed webhooks.
This feature is now available on GitLab.com and self-managed instances along with feature enhancements including handling cold starts.
See Epic and Documentation.
So not only sending back "an invalid HTTP response" would not work, it would result in a disabled webhook, starting with GitLab 15.7+.
I am sending an HTTP PUT request to my Elasticsearch server using Fuel library on Kotlin. However, I cannot see an error body if the server returns 404 or 400. I am expecting to get an error message similar to the following:
{
"error": {
"root_cause": [
{
"type": "invalid_snapshot_name_exception",
"reason": "[snap1:kopya3]Invalid snapshot name [kopya3], snapshot with the same name already exists"
}
],
"type": "invalid_snapshot_name_exception",
"reason": "[snap1:kopya3]Invalid snapshot name [kopya3], snapshot with the same name already exists"
},
"status": 400
}
Here's my code:
val (request, response, result) = fullUrl
.httpPut()
.body(payload)
.responseString()
val (bytes, error) = result
print(error)
Instead what I see is:
HTTP Exception 400 Bad Request
com.github.kittinunf.fuel.core.FuelError$Companion.wrap(FuelError.kt:84)
com.github.kittinunf.fuel.core.DeserializableKt.response(Deserializable.kt:168)
com.github.kittinunf.fuel.core.requests.DefaultRequest.responseString(DefaultRequest.kt:475)
com.a.b.c.d.model.Cluster.createSnapshot(Cluster.kt:67)
com.a.b.c.d.model.Cluster.createSnapshot$default(Cluster.kt:57)
com.a.b.c.d.model.ClusterKt.main(Cluster.kt:85)
Caused by: HTTP Exception 400 Bad Request
com.github.kittinunf.fuel.core.FuelError$Companion.wrap(FuelError.kt:86)
Caused by: com.github.kittinunf.fuel.core.HttpException: HTTP Exception 400 Bad Request
com.github.kittinunf.fuel.core.requests.RequestTask.prepareResponse(RequestTask.kt:35)
com.github.kittinunf.fuel.core.requests.RequestTask.call(RequestTask.kt:47)
com.github.kittinunf.fuel.core.requests.RequestTask.call(RequestTask.kt:14)
com.github.kittinunf.fuel.core.DeserializableKt.response(Deserializable.kt:166)
com.github.kittinunf.fuel.core.requests.DefaultRequest.responseString(DefaultRequest.kt:475)
com.a.b.c.d.model.Cluster.createSnapshot(Cluster.kt:67)
com.a.b.c.d.model.Cluster.createSnapshot$default(Cluster.kt:57)
com.a.b.c.d.model.ClusterKt.main(Cluster.kt:85)
How can I see the actual error message? Thanks in advance.
I solved my problem by looking at response.data which is a bytes array. Converting it to string by String(response.data) I could see the error message.
It's really quite annoying that the default exception doesn't just print this.
I struggled with the generics on this one to pull the handler into its own function, so I thought I share my complete code here.
fun <T> genericErrorHandler(response: Response, result: Result.Failure<Exception>): T {
println("Request to url ${response.url} failed, server returned:")
println(String(response.data))
throw result.getException()
}
private fun someRequest(url: String, jsonBody: String): SomeDataFormatYouExpectOnSuccess {
val (_, response, result) = url.httpPost().header("Content-Type", "application/json; utf-8")
.header("Authorization", "") // adding some headers
.jsonBody(jsonBody)
.responseString()
return when (result) {
is Result.Failure -> genericErrorHandler(response, result)
is Result.Success -> {
val data = result.get()
gson.fromJson(data, SomeDataFormatYouExpectOnSuccess::class.java)
}
}
}
I am using Jon Calhoun's Go MVC framework from github.
The framework uses julienschmidt/httprouter as its only dependency.
I have a similar main method as found in the example:
func main() {
//register routes
router := httprouter.New()
//default
router.GET("/", controllers.Login.Perform(controllers.Login.Index))
//login
router.GET("/login", controllers.Login.Perform(controllers.Login.Login))
router.POST("/login", controllers.Login.Perform(controllers.Login.PostLogin))
//dashboard
router.GET("/dashboard", controllers.Dashboard.Perform(controllers.Dashboard.Index))
//listen and handle requests
log.Fatal(http.ListenAndServe(":"+helpers.ReadConfig("port_http"), router))
}
I make a post to the login url, and it calls the following method:
func (self LoginController) PostLogin(w http.ResponseWriter, r *http.Request, ps httprouter.Params) error {
//create our api url
var url = helpers.ReadConfig("api") + "login"
//fill model to post
login := models.LoginModel{
Password: r.FormValue("password"),
Email: r.FormValue("username"),
}
//render json from model
bytes, err := json.Marshal(login)
if err != nil {
panic(err)
}
//post to the API helpers
var resp = helpers.ApiPost(url, r, string(bytes))
//check response if successful
if resp.Code != constants.ApiResp_Success {
//TODO: Handle API Errors
login.Password = ""
errors := make(map[int]string)
errors[1] = "Please provide valid credntials."
login.Common = models.CommonModel{
ErrorList: errors,
}
return views.Login.Index.Render(w, login, helpers.AcceptsGzip(r))
}
log.Println("---Redirect--")
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
}
Basically, if the login was not correct I return the same view. If the login is correct I want to redirect to another method in a different controller.
However when I call http.Redirect(w, r, "/dashboard", 307), it returns the following error:
http: multiple response.WriteHeader calls
I'm not sure exactly why this is happening, but I suspect that it has something to do with my listener calling the Perform function, which creates a http.handler, as shown below.
func (c *Controller) Perform(a Action) httprouter.Handle {
return httprouter.Handle(
func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
//set response headers
//TODO: set appropriate responce headers
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Cache-Control", "public, max-age=0")
w.Header().Set("Token", "NOT-A-VALID-TOKEN")
w.WriteHeader(200)
if err := a(w, r, ps); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
}
Does anyone have any idea how to redirect using this MVC framework? Or have a one off solution?
http.ResponseWriter's WriteHeader method can only be called once per HTTP response, for obvious reasons: You can only have a single response code, and you can only send the headers once.
The error you see means that it is called a second time on the same response.
Your middleware calls:
w.WriteHeader(200)
Then your handler also calls:
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
Your middleware should never call WriteHeader, until after the fate of the response is known.
Further, without knowing about your particular MVC framework, it seems possible that after you send the 307 status, then you also tell the MVC framework to render a response, which may also call WriteHeader again.
This question already has answers here:
Angular2 handling http response
(3 answers)
Closed 6 years ago.
I've been using the http error handling strategy used in the angular.io documentation:
getHeroes () {
return this.http.get(this._heroesUrl)
.map(res => <Hero[]> res.json().data)
.catch(this.handleError);
}
private handleError (error: Response) {
console.error(error);
return Observable.throw(error.json().error || 'Server error');
}
}
In some scenarios, instead of a JSON response I will receive a 204 status code (No Data). In this case, the error handler doesn't get invoked until failing to parse the result via res.json(), so the error passed to handleError is "error.json is not a function".
How can I interrogate the response stream to check for a 200 (OK) status code or a response header content-type of "application/json", and signal an error handler with a more relevant error message?
http .get('Some Url') .map(res => {
// If request fails, throw an Error that will be caught
if(res.statu != 200) {
throw new Error('This request has failed ' + res.status); } // If everything went fine, return the response
else {return res.json();
} })
This may help you.
This is just for understanding how to use the status from the response, you can modify according to your requirement.
Error is thrown only for non 200 code check this commit