Force Gitlab to retry webhooks on failure with Go - http

Question

I want to watch for every events in a Gitlab project and store them in an external service. For this, I use Gitlab Webhooks. I made a little local HTTP server in Go that listens for Gitlab's POSTs and forward them to an external service. Hooks contains every information I needed so it seems that this architecture is fine:
Gitlab > HTTPServer > External Service.
My problem is when the external service is down, I cannot manage to make Gitlab retry the failed requests. As the documentation says:
GitLab ignores the HTTP status code returned by your endpoint.
Your endpoint should ALWAYS return a valid HTTP response. If you do not do this then GitLab will think the hook failed and retry it.
It is very surprising that Gitlab does not have a proper way to ask for a webhook retry. I have to explicitly return an invalid http response. Moreover, I cannot find an API endpoint to list all failed webhooks and ask for resend.
Question: How to explicitly return an invalid HTTP response with the standard "net/http" library in order to force Gitlab to retry Webhooks?

Answer 1

As written in the comments, a webhook is a mere notification that an event occurred, and potentially some data is sent, typically as JSON data.
It is your responsibility to persist the event itself and the data you want/need to process that was sent with it. Below you will find a commented example. Note that this does not include incremental backoffs, but that should be easy to add:
package main
import (
"encoding/json"
"flag"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/joncrlsn/dque"
)
var (
bind string
queueDir string
segmentSize int
)
// You might want to add request headers and stuff
type webhookContent struct {
Foo string
Bar int
}
func init() {
flag.StringVar(&bind, "bind", ":8080", "The address to bind to")
flag.StringVar(&queueDir, "path", "./queue", "path to store the queue in")
flag.IntVar(&segmentSize, "size", 50, "number of entries for the queue")
}
// The "webserver" component
func runserver(q *dque.DQue) {
http.HandleFunc("/webhook", func(w http.ResponseWriter, r *http.Request) {
// A new decoder for each call, as we want to have a new LimitReader
// for each call. This is a simple, albeit a bit crude method to prevent
// accidental or malicious overload of your server.
dec := json.NewDecoder(io.LimitReader(r.Body, 4096))
defer r.Body.Close()
c := &webhookContent{}
if err := dec.Decode(c); err != nil {
log.Printf("reading body: %s", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// When the content is successfully decoded, we can persist it into
// our queue.
if err := q.Enqueue(c); err != nil {
log.Printf("enqueueing webhook data: %s", err)
// PROPER ERROR HANDLING IS MISSING HERE
}
})
http.ListenAndServe(bind, nil)
}
func main() {
flag.Parse()
var (
q *dque.DQue
err error
)
if !dirExists(queueDir) {
if err = os.MkdirAll(queueDir, 0750); err != nil {
log.Fatalf("creating queue dir: %s", err)
}
}
if !dirExists(filepath.Join(queueDir, "webhooks")) {
q, err = dque.New("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
} else {
q, err = dque.Open("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
}
if err != nil {
log.Fatalf("setting up queue: %s", err)
}
defer q.Close()
go runserver(q)
var (
// Placeholder during event loop
i interface{}
// Payload
w *webhookContent
// Did the type assertion succeed
ok bool
)
for {
// We peek only. The semantic of this is that
// you can already access the next item in the queue
// without removing it from the queue and "mark" it as read.
// We use PeekBlock since we want to wait for an item in the
// queue to be available.
if i, err = q.PeekBlock(); err != nil {
// If we can not peek, something is SERIOUSLY wrong.
log.Fatalf("reading from queue: %s", err)
}
if w, ok = i.(*webhookContent); !ok {
// If the type assertion fails, something is seriously wrong, too.
log.Fatalf("reading from queue: %s", err)
}
if err = doSomethingUseful(w); err != nil {
log.Printf("Something went wrong: %s", err)
log.Println("I strongly suggest entering an incremental backoff!")
continue
}
// We did something useful, so we can dequeue the item we just processed from the queue.
q.Dequeue()
}
}
func doSomethingUseful(w *webhookContent) error {
log.Printf("Instead of this log message, you can do something useful with: %#v", w)
return nil
}
func dirExists(path string) bool {
fileInfo, err := os.Stat(path)
if err == nil {
return fileInfo.IsDir()
}
return false
}
Now when you do something like:
$ curl -X POST --data '{"Foo":"Baz","Bar":42}' http://localhost:8080/webhook
you should get a log entry like
2020/04/18 11:34:23 Instead of this log message, you can do something useful with: &main.webhookContent{Foo:"Baz", Bar:42}

Answer 2

Note that See GitLab 15.7 (December 2022) implements an opposite approach:
Automatic disabling of failing webhooks
To protect GitLab and users across the system from any potential abuse or misuse, we’ve implemented a feature to disable webhooks that fail consistently.
Webhooks that return response codes in the 5xx range are understood to be failing intermittently and are temporarily disabled. These webhooks are initially disabled for 1 minute, which is extended on each retry up to a maximum of 24 hours.
Webhooks that fail with 4xx errors are permanently disabled.
All project owners and maintainers are alerted in the app to investigate and re-enable any failed webhooks.
This feature is now available on GitLab.com and self-managed instances along with feature enhancements including handling cold starts.
See Epic and Documentation.
So not only sending back "an invalid HTTP response" would not work, it would result in a disabled webhook, starting with GitLab 15.7+.