Redirects return http: multiple response.WriteHeader calls - http

I am using Jon Calhoun's Go MVC framework from github.
The framework uses julienschmidt/httprouter as its only dependency.
I have a similar main method as found in the example:
func main() {
//register routes
router := httprouter.New()
//default
router.GET("/", controllers.Login.Perform(controllers.Login.Index))
//login
router.GET("/login", controllers.Login.Perform(controllers.Login.Login))
router.POST("/login", controllers.Login.Perform(controllers.Login.PostLogin))
//dashboard
router.GET("/dashboard", controllers.Dashboard.Perform(controllers.Dashboard.Index))
//listen and handle requests
log.Fatal(http.ListenAndServe(":"+helpers.ReadConfig("port_http"), router))
}
I make a post to the login url, and it calls the following method:
func (self LoginController) PostLogin(w http.ResponseWriter, r *http.Request, ps httprouter.Params) error {
//create our api url
var url = helpers.ReadConfig("api") + "login"
//fill model to post
login := models.LoginModel{
Password: r.FormValue("password"),
Email: r.FormValue("username"),
}
//render json from model
bytes, err := json.Marshal(login)
if err != nil {
panic(err)
}
//post to the API helpers
var resp = helpers.ApiPost(url, r, string(bytes))
//check response if successful
if resp.Code != constants.ApiResp_Success {
//TODO: Handle API Errors
login.Password = ""
errors := make(map[int]string)
errors[1] = "Please provide valid credntials."
login.Common = models.CommonModel{
ErrorList: errors,
}
return views.Login.Index.Render(w, login, helpers.AcceptsGzip(r))
}
log.Println("---Redirect--")
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
}
Basically, if the login was not correct I return the same view. If the login is correct I want to redirect to another method in a different controller.
However when I call http.Redirect(w, r, "/dashboard", 307), it returns the following error:
http: multiple response.WriteHeader calls
I'm not sure exactly why this is happening, but I suspect that it has something to do with my listener calling the Perform function, which creates a http.handler, as shown below.
func (c *Controller) Perform(a Action) httprouter.Handle {
return httprouter.Handle(
func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
//set response headers
//TODO: set appropriate responce headers
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Cache-Control", "public, max-age=0")
w.Header().Set("Token", "NOT-A-VALID-TOKEN")
w.WriteHeader(200)
if err := a(w, r, ps); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
}
Does anyone have any idea how to redirect using this MVC framework? Or have a one off solution?

http.ResponseWriter's WriteHeader method can only be called once per HTTP response, for obvious reasons: You can only have a single response code, and you can only send the headers once.
The error you see means that it is called a second time on the same response.
Your middleware calls:
w.WriteHeader(200)
Then your handler also calls:
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
Your middleware should never call WriteHeader, until after the fate of the response is known.
Further, without knowing about your particular MVC framework, it seems possible that after you send the 307 status, then you also tell the MVC framework to render a response, which may also call WriteHeader again.

Related

Force Gitlab to retry webhooks on failure with Go

I want to watch for every events in a Gitlab project and store them in an external service. For this, I use Gitlab Webhooks. I made a little local HTTP server in Go that listens for Gitlab's POSTs and forward them to an external service. Hooks contains every information I needed so it seems that this architecture is fine:
Gitlab > HTTPServer > External Service.
My problem is when the external service is down, I cannot manage to make Gitlab retry the failed requests. As the documentation says:
GitLab ignores the HTTP status code returned by your endpoint.
Your endpoint should ALWAYS return a valid HTTP response. If you do not do this then GitLab will think the hook failed and retry it.
It is very surprising that Gitlab does not have a proper way to ask for a webhook retry. I have to explicitly return an invalid http response. Moreover, I cannot find an API endpoint to list all failed webhooks and ask for resend.
Question: How to explicitly return an invalid HTTP response with the standard "net/http" library in order to force Gitlab to retry Webhooks?
As written in the comments, a webhook is a mere notification that an event occurred, and potentially some data is sent, typically as JSON data.
It is your responsibility to persist the event itself and the data you want/need to process that was sent with it. Below you will find a commented example. Note that this does not include incremental backoffs, but that should be easy to add:
package main
import (
"encoding/json"
"flag"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/joncrlsn/dque"
)
var (
bind string
queueDir string
segmentSize int
)
// You might want to add request headers and stuff
type webhookContent struct {
Foo string
Bar int
}
func init() {
flag.StringVar(&bind, "bind", ":8080", "The address to bind to")
flag.StringVar(&queueDir, "path", "./queue", "path to store the queue in")
flag.IntVar(&segmentSize, "size", 50, "number of entries for the queue")
}
// The "webserver" component
func runserver(q *dque.DQue) {
http.HandleFunc("/webhook", func(w http.ResponseWriter, r *http.Request) {
// A new decoder for each call, as we want to have a new LimitReader
// for each call. This is a simple, albeit a bit crude method to prevent
// accidental or malicious overload of your server.
dec := json.NewDecoder(io.LimitReader(r.Body, 4096))
defer r.Body.Close()
c := &webhookContent{}
if err := dec.Decode(c); err != nil {
log.Printf("reading body: %s", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// When the content is successfully decoded, we can persist it into
// our queue.
if err := q.Enqueue(c); err != nil {
log.Printf("enqueueing webhook data: %s", err)
// PROPER ERROR HANDLING IS MISSING HERE
}
})
http.ListenAndServe(bind, nil)
}
func main() {
flag.Parse()
var (
q *dque.DQue
err error
)
if !dirExists(queueDir) {
if err = os.MkdirAll(queueDir, 0750); err != nil {
log.Fatalf("creating queue dir: %s", err)
}
}
if !dirExists(filepath.Join(queueDir, "webhooks")) {
q, err = dque.New("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
} else {
q, err = dque.Open("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
}
if err != nil {
log.Fatalf("setting up queue: %s", err)
}
defer q.Close()
go runserver(q)
var (
// Placeholder during event loop
i interface{}
// Payload
w *webhookContent
// Did the type assertion succeed
ok bool
)
for {
// We peek only. The semantic of this is that
// you can already access the next item in the queue
// without removing it from the queue and "mark" it as read.
// We use PeekBlock since we want to wait for an item in the
// queue to be available.
if i, err = q.PeekBlock(); err != nil {
// If we can not peek, something is SERIOUSLY wrong.
log.Fatalf("reading from queue: %s", err)
}
if w, ok = i.(*webhookContent); !ok {
// If the type assertion fails, something is seriously wrong, too.
log.Fatalf("reading from queue: %s", err)
}
if err = doSomethingUseful(w); err != nil {
log.Printf("Something went wrong: %s", err)
log.Println("I strongly suggest entering an incremental backoff!")
continue
}
// We did something useful, so we can dequeue the item we just processed from the queue.
q.Dequeue()
}
}
func doSomethingUseful(w *webhookContent) error {
log.Printf("Instead of this log message, you can do something useful with: %#v", w)
return nil
}
func dirExists(path string) bool {
fileInfo, err := os.Stat(path)
if err == nil {
return fileInfo.IsDir()
}
return false
}
Now when you do something like:
$ curl -X POST --data '{"Foo":"Baz","Bar":42}' http://localhost:8080/webhook
you should get a log entry like
2020/04/18 11:34:23 Instead of this log message, you can do something useful with: &main.webhookContent{Foo:"Baz", Bar:42}
Note that See GitLab 15.7 (December 2022) implements an opposite approach:
Automatic disabling of failing webhooks
To protect GitLab and users across the system from any potential abuse or misuse, we’ve implemented a feature to disable webhooks that fail consistently.
Webhooks that return response codes in the 5xx range are understood to be failing intermittently and are temporarily disabled. These webhooks are initially disabled for 1 minute, which is extended on each retry up to a maximum of 24 hours.
Webhooks that fail with 4xx errors are permanently disabled.
All project owners and maintainers are alerted in the app to investigate and re-enable any failed webhooks.
This feature is now available on GitLab.com and self-managed instances along with feature enhancements including handling cold starts.
See Epic and Documentation.
So not only sending back "an invalid HTTP response" would not work, it would result in a disabled webhook, starting with GitLab 15.7+.

Not handling GET in net/http golang

I am trying to turn off handling GET requests in golang.
I just want to handle POST.
Is it possible to do?
Reason for doing so is that i can see more and more memory being allocated by golang whenever i go to localhost:8080 and refresh page multiple times.
Here is my test code:
package main
import (
"fmt"
"net/http"
"encoding/json"
)
type test_struct struct {
Test string
}
var t test_struct
func handlePOST(rw http.ResponseWriter, req *http.Request) {
switch req.Method {
case "POST":
decoder := json.NewDecoder(req.Body)
decoder.Decode(&t)
defer req.Body.Close()
fmt.Println(t.Test)
}
}
func main() {
http.HandleFunc("/", handlePOST)
http.ListenAndServe(":8080", nil)
}
You cannot not handle GET requests, Go's HTTP server (or rather its http.ServeMux) only allows you to specify a path pattern before dispatching the request to your handler. HTTP method related routing can only happen at the handler level.
Note that some external mux libraries allow you to register handlers to specific HTTP methods only, but the decision and routing based on that also happens in "hidden" handlers of those libraries.
What you're doing is the best: simply do nothing in the handler if the HTTP method is not the one you intend to handle, or even better: send back a http.StatusMethodNotAllowed error response:
func myHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Only POST is allowed", http.StatusMethodNotAllowed)
return
}
var t test_struct // Use local var not global, else it's a data race
decoder := json.NewDecoder(r.Body)
if err := decoder.Decode(&t); err != nil {
fmt.Println("Error decoding:", err)
}
fmt.Println(t.Test)
}

Terminating or aborting an HTTP request

What's the way to abort my API serving with some error message?
Link to call my service:
http://creative.test.spoti.io/api/getVastPlayer?add=
{"Json":Json}&host=api0.spoti.io&domain=domain&userAgent=userAgent&mobile=true
To call my service the client need to send a Json and some params.
I want to test if the params that I get are correct, if not I want send a error message.
The response should be a Json Code {"Result":"Result","Error":"error message"}
I tried log.fatal and os.Exit(1) they stop the service, not just the call request. panic aborts the call but it prevents me to send a http.ResponseWriter which is my error message.
I read something about panic, defer, recover but I don't really know how can I use them to solve this problem.
return works:
mobile :=query.Get("mobile")
if mobile=="mobile" {
str:=`{"Resultt":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
fmt.Println("No successfull Operation!!")
return}
But I can use it just in the main function, because in the other functions it exits just the func not the caller function (request).
Terminating the serving of an HTTP request is nothing more than to return from the ServeHTTP() method, e.g.:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
str := `{"Result":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
return
}
// Do normal API serving
})
panic(http.ListenAndServe(":8080", nil))
Notes:
If the input params of your API service are invalid, you should consider returning an HTTP error code instead of the implied default 200 OK. For this you can use the http.Error() function, for example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
http.Error(w, `Invalid input params!`, http.StatusBadRequest)
return
}
// Do normal API serving
})
For a more sophisticated example where you send back JSON data along with the error code:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// examine incoming params
if !ok {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusBadRequest)
str := `{"Result":"","Error":"No valide Var"}`
fmt.Fprint(w, str)
return
}
// Do normal API serving
})
Example showing how to propagate "returning"
If the error is detected outside of ServeHTTP(), e.g. in a function that is called from ServeHTTP(), you have to return this error state so that ServeHTTP() can return.
Let's assume you have the following custom type for your required parameters and a function which is responsible to decode them from a request:
type params struct {
// fields for your params
}
func decodeParams(r *http.Request) (*params, error) {
p := new(params)
// decode params, if they are invalid, return an error:
if !ok {
return nil, errors.New("Invalid params")
}
// If everything goes well:
return p, nil
}
Using these:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
p, err := decodeParams(r)
if err != nil {
http.Error(w, `Invalid input params!`, http.StatusBadRequest)
return
}
// Do normal API serving
})
Also see this related question: Golang, how to return in func FROM another func?

Can I check response when the request is redirected?

We can register CheckRedirect to check the next request when the request is redirected. Is there a way that I can get the response for the first request when it's redirected?
The way it is currently implemented, it doesn't seem possible to have a look at the response by default (unless you implement yourself what Do() does).
See src/net/http/client.go#L384-L399:
if shouldRedirect(resp.StatusCode) {
// Read the body if small so underlying TCP connection will be re-used.
// No need to check for errors: if it fails, Transport won't reuse it anyway.
const maxBodySlurpSize = 2 << 10
if resp.ContentLength == -1 || resp.ContentLength <= maxBodySlurpSize {
io.CopyN(ioutil.Discard, resp.Body, maxBodySlurpSize)
}
resp.Body.Close()
if urlStr = resp.Header.Get("Location"); urlStr == "" {
err = fmt.Errorf("%d response missing Location header", resp.StatusCode)
break
}
base = req.URL
via = append(via, req)
continue
}
What do you want to do with the first response? It will be pretty boring.
I think the most sensible thing would be to disable automatically following redirects (always return a non-nil error from CheckRedirect) and handle the redirection yourself in which case you have full access to all requests/responses.
My workaround for this:
Any http request / response could be patched within a custom RoundTripper, including redirects:
type RedirectChecker struct{}
func (RedirectChecker) RoundTrip(req *http.Request) (*http.Response, error) {
// e.g. patch the request before send it
req.Header.Set("user-agent", "curl/7.64.1")
resp, err := http.DefaultTransport.RoundTrip(req)
if err == nil && resp != nil {
switch resp.StatusCode {
case
http.StatusMovedPermanently,
http.StatusFound,
http.StatusTemporaryRedirect,
http.StatusPermanentRedirect:
// e.g. stop further redirections
// roughly equivalent to http.ErrUseLastResponse
resp.StatusCode = http.StatusOK
// e.g. read the Set-Cookie headers
// unfortunately cookie jars do not handle redirects in a proper manner
// and that's why I came to this question...
fmt.Printf("%+v", resp.Cookies())
}
}
}
httpClient := &http.Client{Transport: RedirectChecker{}}
httpClient.Do(...)

Go http cannot handle HTTP requests with no PATH

I am writing a small HTTP server that receives HTTP POSTs from some embedded devices. Unfortunately these devices send malformed POST request that contain no PATH component:
POST HTTP/1.1
Host: 192.168.13.130:8080
Content-Length: 572
Connection: Keep-Alive
<?xml version="1.0"?>
....REST OF XML BODY
Due to this the Go http never passes the request to any of my handlers and always responds with 400 Bad Request.
Since these are embedded devices and changing the way they send the request is not an option I though maybe I could intercept the HTTP requests and if no PATH is present add one (e.g. /) to it before it passes to the SeverMux.
I tried this by creating my own CameraMux but Go always responds with 400 Bad Request even before calling the ServeHTTP() method from my custom ServeMux (see code below).
Is there a way to modify the Request object at some point before Go http responds Bad Request or there is a way to make Go accept the request even if it has no PATH?
package main
import (
"net/http"
"log"
"os"
)
type CameraMux struct {
mux *http.ServeMux
}
func (handler *CameraMux) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Try to fix URL.Path here but the server never reaches this method.
log.Printf("URL %v\n", r.URL.Path)
handler.mux.ServeHTTP(w, r)
}
func process(path string) error {
log.Printf("Processing %v\n", path)
// Do processing based on path and body
return nil
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path[1:]
log.Printf("Processing path %v\n", path)
err := process(path)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
} else {
w.WriteHeader(http.StatusOK)
}
})
err := http.ListenAndServe(":8080", &CameraMux{http.DefaultServeMux})
if err != nil {
log.Println(err)
os.Exit(1)
}
os.Exit(0)
}
The error you are seeing occurs within the request parsing logic, which happens before ServeHTTP is called.
The HTTP request is read from the socket by the ReadRequest function from the net/http package. It will tokenize the first line of the request with an empty URL portion, but then goes on to parse the URL:
if req.URL, err = url.ParseRequestURI(rawurl); err != nil {
return nil, err
}
Unfortunately this function will return an error for an empty URL string, which will in turn aborts the request reading process.
So it doesn't look like there is an easy way to achieve what you're after without modifying the standard library code.
I'm unsure if Go's HTTP parser will allow requests with no URI path element. If it doesn't then you're out of luck. If it does however; you could overwrite the request's path like this:
type FixPath struct {}
func (f *FixPath) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.RequestURI = "/dummy/path" // fix URI path
http.DefaultServeMux.ServeHTTP(w, r) // forward the fixed request to http.DefaultServeMux
}
func main() {
// register handlers with http.DefaultServeMux through http.Handle or http.HandleFunc, and then...
http.ListenAndServe(":8080", &FixPath{})
}

Resources