We can register CheckRedirect to check the next request when the request is redirected. Is there a way that I can get the response for the first request when it's redirected?
The way it is currently implemented, it doesn't seem possible to have a look at the response by default (unless you implement yourself what Do() does).
See src/net/http/client.go#L384-L399:
if shouldRedirect(resp.StatusCode) {
// Read the body if small so underlying TCP connection will be re-used.
// No need to check for errors: if it fails, Transport won't reuse it anyway.
const maxBodySlurpSize = 2 << 10
if resp.ContentLength == -1 || resp.ContentLength <= maxBodySlurpSize {
io.CopyN(ioutil.Discard, resp.Body, maxBodySlurpSize)
}
resp.Body.Close()
if urlStr = resp.Header.Get("Location"); urlStr == "" {
err = fmt.Errorf("%d response missing Location header", resp.StatusCode)
break
}
base = req.URL
via = append(via, req)
continue
}
What do you want to do with the first response? It will be pretty boring.
I think the most sensible thing would be to disable automatically following redirects (always return a non-nil error from CheckRedirect) and handle the redirection yourself in which case you have full access to all requests/responses.
My workaround for this:
Any http request / response could be patched within a custom RoundTripper, including redirects:
type RedirectChecker struct{}
func (RedirectChecker) RoundTrip(req *http.Request) (*http.Response, error) {
// e.g. patch the request before send it
req.Header.Set("user-agent", "curl/7.64.1")
resp, err := http.DefaultTransport.RoundTrip(req)
if err == nil && resp != nil {
switch resp.StatusCode {
case
http.StatusMovedPermanently,
http.StatusFound,
http.StatusTemporaryRedirect,
http.StatusPermanentRedirect:
// e.g. stop further redirections
// roughly equivalent to http.ErrUseLastResponse
resp.StatusCode = http.StatusOK
// e.g. read the Set-Cookie headers
// unfortunately cookie jars do not handle redirects in a proper manner
// and that's why I came to this question...
fmt.Printf("%+v", resp.Cookies())
}
}
}
httpClient := &http.Client{Transport: RedirectChecker{}}
httpClient.Do(...)
Related
I want to watch for every events in a Gitlab project and store them in an external service. For this, I use Gitlab Webhooks. I made a little local HTTP server in Go that listens for Gitlab's POSTs and forward them to an external service. Hooks contains every information I needed so it seems that this architecture is fine:
Gitlab > HTTPServer > External Service.
My problem is when the external service is down, I cannot manage to make Gitlab retry the failed requests. As the documentation says:
GitLab ignores the HTTP status code returned by your endpoint.
Your endpoint should ALWAYS return a valid HTTP response. If you do not do this then GitLab will think the hook failed and retry it.
It is very surprising that Gitlab does not have a proper way to ask for a webhook retry. I have to explicitly return an invalid http response. Moreover, I cannot find an API endpoint to list all failed webhooks and ask for resend.
Question: How to explicitly return an invalid HTTP response with the standard "net/http" library in order to force Gitlab to retry Webhooks?
As written in the comments, a webhook is a mere notification that an event occurred, and potentially some data is sent, typically as JSON data.
It is your responsibility to persist the event itself and the data you want/need to process that was sent with it. Below you will find a commented example. Note that this does not include incremental backoffs, but that should be easy to add:
package main
import (
"encoding/json"
"flag"
"io"
"log"
"net/http"
"os"
"path/filepath"
"github.com/joncrlsn/dque"
)
var (
bind string
queueDir string
segmentSize int
)
// You might want to add request headers and stuff
type webhookContent struct {
Foo string
Bar int
}
func init() {
flag.StringVar(&bind, "bind", ":8080", "The address to bind to")
flag.StringVar(&queueDir, "path", "./queue", "path to store the queue in")
flag.IntVar(&segmentSize, "size", 50, "number of entries for the queue")
}
// The "webserver" component
func runserver(q *dque.DQue) {
http.HandleFunc("/webhook", func(w http.ResponseWriter, r *http.Request) {
// A new decoder for each call, as we want to have a new LimitReader
// for each call. This is a simple, albeit a bit crude method to prevent
// accidental or malicious overload of your server.
dec := json.NewDecoder(io.LimitReader(r.Body, 4096))
defer r.Body.Close()
c := &webhookContent{}
if err := dec.Decode(c); err != nil {
log.Printf("reading body: %s", err)
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// When the content is successfully decoded, we can persist it into
// our queue.
if err := q.Enqueue(c); err != nil {
log.Printf("enqueueing webhook data: %s", err)
// PROPER ERROR HANDLING IS MISSING HERE
}
})
http.ListenAndServe(bind, nil)
}
func main() {
flag.Parse()
var (
q *dque.DQue
err error
)
if !dirExists(queueDir) {
if err = os.MkdirAll(queueDir, 0750); err != nil {
log.Fatalf("creating queue dir: %s", err)
}
}
if !dirExists(filepath.Join(queueDir, "webhooks")) {
q, err = dque.New("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
} else {
q, err = dque.Open("webhooks", queueDir, segmentSize, func() interface{} { return &webhookContent{} })
}
if err != nil {
log.Fatalf("setting up queue: %s", err)
}
defer q.Close()
go runserver(q)
var (
// Placeholder during event loop
i interface{}
// Payload
w *webhookContent
// Did the type assertion succeed
ok bool
)
for {
// We peek only. The semantic of this is that
// you can already access the next item in the queue
// without removing it from the queue and "mark" it as read.
// We use PeekBlock since we want to wait for an item in the
// queue to be available.
if i, err = q.PeekBlock(); err != nil {
// If we can not peek, something is SERIOUSLY wrong.
log.Fatalf("reading from queue: %s", err)
}
if w, ok = i.(*webhookContent); !ok {
// If the type assertion fails, something is seriously wrong, too.
log.Fatalf("reading from queue: %s", err)
}
if err = doSomethingUseful(w); err != nil {
log.Printf("Something went wrong: %s", err)
log.Println("I strongly suggest entering an incremental backoff!")
continue
}
// We did something useful, so we can dequeue the item we just processed from the queue.
q.Dequeue()
}
}
func doSomethingUseful(w *webhookContent) error {
log.Printf("Instead of this log message, you can do something useful with: %#v", w)
return nil
}
func dirExists(path string) bool {
fileInfo, err := os.Stat(path)
if err == nil {
return fileInfo.IsDir()
}
return false
}
Now when you do something like:
$ curl -X POST --data '{"Foo":"Baz","Bar":42}' http://localhost:8080/webhook
you should get a log entry like
2020/04/18 11:34:23 Instead of this log message, you can do something useful with: &main.webhookContent{Foo:"Baz", Bar:42}
Note that See GitLab 15.7 (December 2022) implements an opposite approach:
Automatic disabling of failing webhooks
To protect GitLab and users across the system from any potential abuse or misuse, we’ve implemented a feature to disable webhooks that fail consistently.
Webhooks that return response codes in the 5xx range are understood to be failing intermittently and are temporarily disabled. These webhooks are initially disabled for 1 minute, which is extended on each retry up to a maximum of 24 hours.
Webhooks that fail with 4xx errors are permanently disabled.
All project owners and maintainers are alerted in the app to investigate and re-enable any failed webhooks.
This feature is now available on GitLab.com and self-managed instances along with feature enhancements including handling cold starts.
See Epic and Documentation.
So not only sending back "an invalid HTTP response" would not work, it would result in a disabled webhook, starting with GitLab 15.7+.
I was using http.NewRequest to make a GET Request.
I deliberately tried to tamper the API url just to check if my err handling works.
But its not working as expected . In err value is returned and I am unable to compare it.
jsonData := map[string]string{"firstname": "Nic", "lastname": "Raboy"}
jsonValue, _ := json.Marshal(jsonData)
request, err := http.NewRequest("POST", "http://httpbin.org/postsdf", bytes.NewBuffer(jsonValue))
request.Header.Set("Content-Type", "application/json")
client := &http.Client{}
response, err := client.Do(request)
if err != nil {
fmt.Println("wrong")
} else {
data, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(data))
}
The output is as below:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
But I am expecting to print "Wrong".
The HTTP call succeeds (the call went through to the server, and response came back), that's why err is nil. It's just that the HTTP status code is not http.StatusOK (but by judging the response document, it's http.StatusNotFound).
You should check the HTTP status code like this:
response, err := client.Do(request)
if err != nil {
fmt.Println("HTTP call failed:", err)
return
}
// Don't forget, you're expected to close response body even if you don't want to read it.
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
fmt.Println("Non-OK HTTP status:", response.StatusCode)
// You may read / inspect response body
return
}
// All is OK, server reported success.
Also note that certain API endpoints might return other than http.StatusOK for success, such as HTTP 201 - Created, or HTTP 202 - Accepted etc. If you want to check all success status codes, you can do it like this:
// Success is indicated with 2xx status codes:
statusOK := response.StatusCode >= 200 && response.StatusCode < 300
if !statusOK {
fmt.Println("Non-OK HTTP status:", response.StatusCode)
// You may read / inspect response body
return
}
you should user the status code, you can also write a small handler for every http request that you create.
response, err := client.Do(request)
switch response.StatusCode {
case 200:
fmt.Println("work!")
break
case 404:
fmt.Println("not found!")
break
default:
fmt.Println("http", response.Status)
}
I am using Jon Calhoun's Go MVC framework from github.
The framework uses julienschmidt/httprouter as its only dependency.
I have a similar main method as found in the example:
func main() {
//register routes
router := httprouter.New()
//default
router.GET("/", controllers.Login.Perform(controllers.Login.Index))
//login
router.GET("/login", controllers.Login.Perform(controllers.Login.Login))
router.POST("/login", controllers.Login.Perform(controllers.Login.PostLogin))
//dashboard
router.GET("/dashboard", controllers.Dashboard.Perform(controllers.Dashboard.Index))
//listen and handle requests
log.Fatal(http.ListenAndServe(":"+helpers.ReadConfig("port_http"), router))
}
I make a post to the login url, and it calls the following method:
func (self LoginController) PostLogin(w http.ResponseWriter, r *http.Request, ps httprouter.Params) error {
//create our api url
var url = helpers.ReadConfig("api") + "login"
//fill model to post
login := models.LoginModel{
Password: r.FormValue("password"),
Email: r.FormValue("username"),
}
//render json from model
bytes, err := json.Marshal(login)
if err != nil {
panic(err)
}
//post to the API helpers
var resp = helpers.ApiPost(url, r, string(bytes))
//check response if successful
if resp.Code != constants.ApiResp_Success {
//TODO: Handle API Errors
login.Password = ""
errors := make(map[int]string)
errors[1] = "Please provide valid credntials."
login.Common = models.CommonModel{
ErrorList: errors,
}
return views.Login.Index.Render(w, login, helpers.AcceptsGzip(r))
}
log.Println("---Redirect--")
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
}
Basically, if the login was not correct I return the same view. If the login is correct I want to redirect to another method in a different controller.
However when I call http.Redirect(w, r, "/dashboard", 307), it returns the following error:
http: multiple response.WriteHeader calls
I'm not sure exactly why this is happening, but I suspect that it has something to do with my listener calling the Perform function, which creates a http.handler, as shown below.
func (c *Controller) Perform(a Action) httprouter.Handle {
return httprouter.Handle(
func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
//set response headers
//TODO: set appropriate responce headers
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Cache-Control", "public, max-age=0")
w.Header().Set("Token", "NOT-A-VALID-TOKEN")
w.WriteHeader(200)
if err := a(w, r, ps); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
}
Does anyone have any idea how to redirect using this MVC framework? Or have a one off solution?
http.ResponseWriter's WriteHeader method can only be called once per HTTP response, for obvious reasons: You can only have a single response code, and you can only send the headers once.
The error you see means that it is called a second time on the same response.
Your middleware calls:
w.WriteHeader(200)
Then your handler also calls:
http.Redirect(w, r, "/dashboard", 307)
log.Println("-----")
return views.Dashboard.Index.Render(w, login, helpers.AcceptsGzip(r))
Your middleware should never call WriteHeader, until after the fate of the response is known.
Further, without knowing about your particular MVC framework, it seems possible that after you send the 307 status, then you also tell the MVC framework to render a response, which may also call WriteHeader again.
I'd like to log 301s vs 302s but can't see a way to read the response status code in Client.Do, Get, doFollowingRedirects, CheckRedirect. Will I have to implement redirection myself to achieve this?
The http.Client type allows you to specify a custom transport, which should allow you to do what you're after. Something like the following should do:
type LogRedirects struct {
Transport http.RoundTripper
}
func (l LogRedirects) RoundTrip(req *http.Request) (resp *http.Response, err error) {
t := l.Transport
if t == nil {
t = http.DefaultTransport
}
resp, err = t.RoundTrip(req)
if err != nil {
return
}
switch resp.StatusCode {
case http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther, http.StatusTemporaryRedirect:
log.Println("Request for", req.URL, "redirected with status", resp.StatusCode)
}
return
}
(you could simplify this a little if you only support chaining to the default transport).
You can then create a client using this transport, and any redirects should be logged:
client := &http.Client{Transport: LogRedirects{}}
Here is a full example you can experiment with: http://play.golang.org/p/8uf8Cn31HC
I am writing a small HTTP server that receives HTTP POSTs from some embedded devices. Unfortunately these devices send malformed POST request that contain no PATH component:
POST HTTP/1.1
Host: 192.168.13.130:8080
Content-Length: 572
Connection: Keep-Alive
<?xml version="1.0"?>
....REST OF XML BODY
Due to this the Go http never passes the request to any of my handlers and always responds with 400 Bad Request.
Since these are embedded devices and changing the way they send the request is not an option I though maybe I could intercept the HTTP requests and if no PATH is present add one (e.g. /) to it before it passes to the SeverMux.
I tried this by creating my own CameraMux but Go always responds with 400 Bad Request even before calling the ServeHTTP() method from my custom ServeMux (see code below).
Is there a way to modify the Request object at some point before Go http responds Bad Request or there is a way to make Go accept the request even if it has no PATH?
package main
import (
"net/http"
"log"
"os"
)
type CameraMux struct {
mux *http.ServeMux
}
func (handler *CameraMux) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Try to fix URL.Path here but the server never reaches this method.
log.Printf("URL %v\n", r.URL.Path)
handler.mux.ServeHTTP(w, r)
}
func process(path string) error {
log.Printf("Processing %v\n", path)
// Do processing based on path and body
return nil
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path[1:]
log.Printf("Processing path %v\n", path)
err := process(path)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
} else {
w.WriteHeader(http.StatusOK)
}
})
err := http.ListenAndServe(":8080", &CameraMux{http.DefaultServeMux})
if err != nil {
log.Println(err)
os.Exit(1)
}
os.Exit(0)
}
The error you are seeing occurs within the request parsing logic, which happens before ServeHTTP is called.
The HTTP request is read from the socket by the ReadRequest function from the net/http package. It will tokenize the first line of the request with an empty URL portion, but then goes on to parse the URL:
if req.URL, err = url.ParseRequestURI(rawurl); err != nil {
return nil, err
}
Unfortunately this function will return an error for an empty URL string, which will in turn aborts the request reading process.
So it doesn't look like there is an easy way to achieve what you're after without modifying the standard library code.
I'm unsure if Go's HTTP parser will allow requests with no URI path element. If it doesn't then you're out of luck. If it does however; you could overwrite the request's path like this:
type FixPath struct {}
func (f *FixPath) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.RequestURI = "/dummy/path" // fix URI path
http.DefaultServeMux.ServeHTTP(w, r) // forward the fixed request to http.DefaultServeMux
}
func main() {
// register handlers with http.DefaultServeMux through http.Handle or http.HandleFunc, and then...
http.ListenAndServe(":8080", &FixPath{})
}