Golang "301 Moved Permanently" if request path contains additional slash - http

I have been using golang's default http.ServeMux for http route handling.
wrap := func(h func(t *MyStruct, w http.ResponseWriter, r *http.Request)) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
h(t, w, r)
}
}
// Register handlers with default mux
httpMux := http.NewServeMux()
httpMux.HandleFunc("/", wrap(payloadHandler))
Assume this server is accessible via http://example.com/
Very few of my client's requests were of path http://example.com/api//module (note the extra slash) which is redirected as 301 Moved Permanently. Exploring inside golang's http ServeMux.Handler(r *Request) function, seems it's intended.
path := cleanPath(r.URL.Path)
// if there is any change between actual and cleaned path, the request is 301 ed
if path != r.URL.Path {
_, pattern = mux.handler(host, path)
url := *r.URL
url.Path = path
return RedirectHandler(url.String(), StatusMovedPermanently), pattern
}
I've looked into other similar issue.
go-web-server-is-automatically-redirecting-post-requests
Above qn has problem with redundant / in register pattern itself, but my use case is not with register pattern (in some nested path which is irrelevent to register pattern)
Problem is, since my client's requests are POST, browsers handle 301 with new GET request with exact query params and POST body. But change in the HTTP method causes the request to fail.
I have already instructed client to fix the redundant / in url, but the fix might take few (?) weeks to be deployed in all client locations.
Also these redundant / are handled fine in Apache Tomcat, but fails only in golang server. So is this the intended behaviour in my use case (redundant / in nested path) with golang or possible bug?
I am thinking of way to override the Handler func of ServeMux, but it won't be useful since Handler calls are made internally. Looking to disable this 301 behaviour, help would be appreciated.
Relevant links
http-post-method-is-actally-sending-a-get

The clean and redirect is intended behavior.
Wrap the mux with a handler that removes the double slashes:
type slashFix struct {
mux http.Handler
}
func (h *slashFix) ServeHTTP(w http.ResponseWriter, r *http.Request) {
r.URL.Path = strings.Replace(r.URL.Path, "//", "/", -1)
h.mux.ServeHTTP(w, r)
}
Use it like this:
httpMux := http.NewServeMux()
httpMux.HandleFunc("/", wrap(payloadHandler))
http.ListenAndServe(addr, &slashFix{httpMux})

Accepeted answer solved the problem
One more way is to use Gorilla mux and setting SkipClean(true). But be sure to know about the side effects in its doc
SkipClean defines the path cleaning behaviour for new routes. The initial value is false. Users should be careful about which routes are not cleaned. When true, if the route path is "/path//to", it will remain with the double slash. This is helpful if you have a route like: /fetch/http://xkcd.com/534/
When false, the path will be cleaned, so /fetch/http://xkcd.com/534/ will become /fetch/http/xkcd.com/534
func (r *Router) SkipClean(value bool) *Router {
r.skipClean = value
return r
}

Related

How do I get Go's net/http package to stop removing double slashes?

Consider the following very basic "net/http"-program:
package main
import (
"net/http"
"log"
)
func entry(w http.ResponseWriter, req *http.Request) {
log.Println(req.URL.Path)
path := []byte(req.URL.Path)
w.Write(path)
}
func main() {
http.HandleFunc("/", entry)
err := http.ListenAndServe("localhost:10000", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
When accessing the URL, http://localhost:10000/a//b, the DefaultServerMux, which is what the default HandleFunc() and ListenAndServe() uses, redirects it to /a/b, effectively removing the double slash.
The documentation for ServerMux does specify that it 'sanitises' URL requests:
ServeMux also takes care of sanitizing the URL request path, redirecting any request containing . or .. elements or repeated slashes to an equivalent, cleaner URL.
But what if I don't want it to? I have a scenario where // != / in my URLs. I could find another solution. But is there a way to still use Go's "net/http" package, while it not sanitising my URLs like this? Preferably with as little re-writing as possible.
(I will probably find a different solution than having // and / being distinct, since I am probably happy with other features that ServerMux provides (in case a solution requires me to use another multiplexer), but now I am curious whether there is a solution with Go's standard "net/http" package.)
Go for gorilla mux to route the server using the url. By default the DefaultSeveMux uses Clean which modify the url against various unwanted characters. That's the reason the url changed whn you are using // double slash.
func Clean(path string) string
While if you do not want to sanitize the url. Gorilla mux provide a method SkipClean() which when set to true. It will not sanitize the url.
func (r *Router) SkipClean(value bool) *Router {
r.skipClean = value
return r
}
It is mentioned in the document of gorilla mux for SkipClean() as:
SkipClean defines the path cleaning behaviour for new routes. The
initial value is false. Users should be careful about which routes are
not cleaned When true, if the route path is "/path//to", it will
remain with the double slash. This is helpful if you have a route
like: /fetch/http://xkcd.com/534/ When false, the path will be
cleaned, so /fetch/http://xkcd.com/534/ will become
/fetch/http/xkcd.com/534

How do I pass a URL as a parameter in a Golang route?

I'm trying to pass a URL as a parameter in Golang, and I haven't been able to find a solution in all of the tutorials I've looked at. The problem is that I can only get the url to return minus a crucial forward slash.
My handler looks like this:
router.HandleFunc("/new/{url}", createURL)
So the request would look like:
www.myapp.heroku.com/new/https://www.google.com
However, the url that I results is missing a slash:
http:/www.google.com
I sure it's probably got something to do with RFC3986, but is there a way to pass in the url as it is?
After reading the other question, I understand what do you mean. Implement a kind of URL re-writer before URL goes to gorilla/mux. The function will look like:
func Rewriter(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
//Simple URL rewriter. Rewrite if it's started with API path
pathReq := r.RequestURI
if strings.HasPrefix(pathReq, "/new/") {
//Use url.QueryEscape for pre go1.8
pe := url.PathEscape(strings.TrimLeft(pathReq, "/new/"))
r.URL.Path = "/new/" + pe
r.URL.RawQuery = ""
}
h.ServeHTTP(w, r)
})
}
Wrap gorilla router when starting the http server:
r := mux.NewRouter()
// ... other handler
r.HandleFunc("/new/{original-url}", NewHandler)
//Wrap mux.Router using Rewriter
log.Fatal(http.ListenAndServe(":8080", Rewriter(r)))
Then in your URL shortener handler, the original URL can be extracted using the following code:
func NewHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
ou := vars["original-url"]
//Use url.QueryUnascape for pre go1.8
originalURL, err := url.PathUnescape(ou)
//... other processing
}
IMHO, implementing URL shortener service like this is not recommended, mainly due to incorrect use of HTTP verbs. Any GET request should not leave side effect in server e.g. no record creation in database, etc.
This particular behavior in Gorilla Mux can be changed by setting SkipClean to true.
router := mux.NewRouter()
router.SkipClean(true)
router.HandleFunc("/new/", index)
router.HandleFunc("/", index)
http.ListenAndServe(":"+port, router)
The relevant documentation can be found here.

Golang Modify HTTP Request Parameters Such As URL Path Before Routing

It's common practice in some cases to pass plain URIs as suffix of the path instead of a query parameter. Here is an example from Internet Archive's Wayback Machine.
https://web.archive.org/web/20150825082012/http://example.com/
In this example, user is requesting a copy of http://example.com/ as captured at 2015-08-25 08:20:12. If we were to implement similar service in Go, we probably would have a router as follows:
http.HandleFunc("/web/", returnArchivedCopy)
Then in the returnArchivedCopy handler function, we will split r.URL.Path (where r is the Request object) to extract the date-time and the target URL. However there is a problem in this style of URL scheme; Go's net/http package calls cleanPath function on the path portion to sanitize it. This sanitization process does various cleanup tasks such as eeliminating . and .. from the path and replace multiple slashes with a single one. This later operation makes sense when because in Unix systems // in the file path are same as /. However this causes an issue in the above described use case as http://example becomes http:/example and the server internally returns a redirect response to the client with the sanitized path.
I am wondering, what are my options in this case? Is there a way to ask HTTP not to sanitize the request path while still utilizing all the default behavior that is shipped with the default (or slightly modified) server, multiplexer, and handler? Or is there a way to modify the request parameters (path in this case) before it hits the multiplexer's routing patterns. If the later is possible, we might try to perform something like URL encoding to avoid the redirect and later decode the URL back in the handler function before extracting desired bits.
I have experimented with some custom handlers and multiplexers, but I am new to Go, hence I was not quite sure how to delegate the routing back to the default handlers after making changes in the request.
You can implement a wrapper mux, that falls back to the default one, here's a very simple example:
func main() {
http.HandleFunc("/blah", func(w http.ResponseWriter, req *http.Request) {
w.Write([]byte("w00t"))
})
http.ListenAndServe(":9090", http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
p := strings.SplitN(req.URL.RequestURI()[1:] /*trim the first slash*/, "/", 3)
if len(p) != 3 || p[0] != "web" {
http.DefaultServeMux.ServeHTTP(w, req)
return
}
t, err := time.Parse("20060102150405", p[1])
if err != nil {
http.Error(w, "invalid time", 400)
return
}
url := p[2]
fmt.Fprintf(w, "requested url %v # %v", url, t)
}))
}

Overriding Go's default HTTP Sever redirect behaviour

Go's default HTTP server implementation merges slashes in HTTP requests, returning an HTTP redirect response to the "cleaned" path:
https://code.google.com/p/go/source/browse/src/pkg/net/http/server.go#1420
So if you make a HTTP request GET /http://foo.com/, the server responds with 301 Moved Permanently ... Location: /http:/foo.com/.
I'd like to disable this behaviour and handle all paths myself.
I'm a Go newbie, and it seems as if I could create my own Server instance and override the Handler attribute, but I'm not sure how to?
I'd like to disable this behaviour and handle all paths myself.
I'm a Go newbie, and it seems as if I could create my own Server instance and override the Handler attribute, but I'm not sure how to?
Instead of registering handlers with the http.DefaultServeMux through the http.Handle or http.HandleFunc methods just call:
http.ListenAndServe(":8080", MyHandler)
where MyHandler is an instance of a type that implements the http.Handler interface.
http.ListenAndServe in turn is just a short-hand method that does the following:
func ListenAndServe(addr string, handler http.Handler) error {
server := &http.Server{Addr: addr, Handler: handler}
return server.ListenAndServe()
}
so you could do that directly instead as well.
Inside your handler you can then parse/route the URI however you wish like this:
func (h *MyHandlerType) ServeHTTP(w http.ResponseWriter, r *http.Request) {
uri := r.URL.Path
// ...use uri...
}

Setting HTTP headers

I'm trying to set a header in my Go web server. I'm using gorilla/mux and net/http packages.
I'd like to set Access-Control-Allow-Origin: * to allow cross domain AJAX.
Here's my Go code:
func saveHandler(w http.ResponseWriter, r *http.Request) {
// do some stuff with the request data
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/save", saveHandler)
http.Handle("/", r)
http.ListenAndServe(":"+port, nil)
}
The net/http package has documentation describing sending http request headers as if it were a client - I'm not exactly sure how to set response headers?
Never mind, I figured it out - I used the Set() method on Header() (doh!)
My handler looks like this now:
func saveHandler(w http.ResponseWriter, r *http.Request) {
// allow cross domain AJAX requests
w.Header().Set("Access-Control-Allow-Origin", "*")
}
Maybe this will help someone as caffeine deprived as myself sometime :)
All of the above answers are wrong because they fail to handle the OPTIONS preflight request, the solution is to override the mux router's interface. See AngularJS $http get request failed with custom header (alllowed in CORS)
func main() {
r := mux.NewRouter()
r.HandleFunc("/save", saveHandler)
http.Handle("/", &MyServer{r})
http.ListenAndServe(":8080", nil);
}
type MyServer struct {
r *mux.Router
}
func (s *MyServer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
if origin := req.Header.Get("Origin"); origin != "" {
rw.Header().Set("Access-Control-Allow-Origin", origin)
rw.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
rw.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
// Stop here if its Preflighted OPTIONS request
if req.Method == "OPTIONS" {
return
}
// Lets Gorilla work
s.r.ServeHTTP(rw, req)
}
Do not use '*' for Origin, until You really need a completely public behavior.
As Wikipedia says:
"The value of "*" is special in that it does not allow requests to supply credentials,
meaning HTTP authentication, client-side SSL certificates, nor does it allow cookies
to be sent."
That means, you'll get a lot of errors, especially in Chrome when you'll try to implement for example a simple authentication.
Here is a corrected wrapper:
// Code has not been tested.
func addDefaultHeaders(fn http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token")
w.Header().Set("Access-Control-Allow-Credentials", "true")
fn(w, r)
}
}
And don't forget to reply all these headers to the preflight OPTIONS request.
If you don't want to override your router (if you don't have your app configured in a way that supports this, or want to configure CORS on a route by route basis), add an OPTIONS handler to handle the pre flight request.
Ie, with Gorilla Mux your routes would look like:
accounts := router.Path("/accounts").Subrouter()
accounts.Methods("POST").Handler(AccountsCreate)
accounts.Methods("OPTIONS").Handler(AccountsCreatePreFlight)
Note above that in addition to our POST handler, we're defining a specific OPTIONS method handler.
And then to actual handle the OPTIONS preflight method, you could define AccountsCreatePreFlight like so:
// Check the origin is valid.
origin := r.Header.Get("Origin")
validOrigin, err := validateOrigin(origin)
if err != nil {
return err
}
// If it is, allow CORS.
if validOrigin {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "POST")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
What really made this all click for me (in addition to actually understanding how CORS works) is that the HTTP Method of a preflight request is different from the HTTP Method of the actual request. To initiate CORS, the browser sends a preflight request with HTTP Method OPTIONS, which you have to handle explicitly in your router, and then, if it receives the appropriate response "Access-Control-Allow-Origin": origin (or "*" for all) from your application, it initiates the actual request.
I also believe that you can only do "*" for standard types of requests (ie: GET), but for others you'll have to explicitly set the origin like I do above.
Set a proper golang middleware, so you can reuse on any endpoint.
Helper Type and Function
type Adapter func(http.Handler) http.Handler
// Adapt h with all specified adapters.
func Adapt(h http.Handler, adapters ...Adapter) http.Handler {
for _, adapter := range adapters {
h = adapter(h)
}
return h
}
Actual middleware
func EnableCORS() Adapter {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
// Stop here if its Preflighted OPTIONS request
if r.Method == "OPTIONS" {
return
}
h.ServeHTTP(w, r)
})
}
}
Endpoint
REMEBER! Middlewares get applyed on reverse order( ExpectGET() gets fires first)
mux.Handle("/watcher/{action}/{device}",Adapt(api.SerialHandler(mux),
api.EnableCORS(),
api.ExpectGET(),
))
I create wrapper for this case:
func addDefaultHeaders(fn http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
fn(w, r)
}
}
I had the same issue as described above the solutions given above are correct, the set up I have is as follows
1) Angularjs for the Client
2) Beego framework for GO server
Please following these points
1) CORS settings must be enabled only on GO server
2) Do NOT add any type of headers in angularJS except for this
.config(['$httpProvider', function($httpProvider) {
$httpProvider.defaults.useXDomain = true;
delete $httpProvider.defaults.headers.common['X-Requested-With'];
}])
In you GO server add the CORS settings before the request starts to get processed so that the preflight request receives a 200 OK after which the the OPTIONS method will get converted to GET,POST,PUT or what ever is your request type.
I know this is a different twist on the answer, but isn't this more of a concern for a web server? For example, nginx, could help.
The ngx_http_headers_module module allows adding the “Expires” and “Cache-Control” header fields, and arbitrary fields, to a response header
...
location ~ ^<REGXP MATCHING CORS ROUTES> {
add_header Access-Control-Allow-Methods POST
...
}
...
Adding nginx in front of your go service in production seems wise. It provides a lot more feature for authorizing, logging,and modifying requests. Also, it gives the ability to control who has access to your service and not only that but one can specify different behavior for specific locations in your app, as demonstrated above.
I could go on about why to use a web server with your go api, but I think that's a topic for another discussion.

Resources