I've implemented the HTTP Caching guidelines described by Google in a URL Shortening API that I've developed.
Here's how I'm sending the response:
const urlResponseCacheControlMaxAge = 172800 // 2 days
type urlResponse struct {
LongURL string `json:"longUrl"`
ShortURL string `json:"shortUrl"`
}
func (u urlResponse) Hash() string {
parts := strings.Split(u.ShortURL, "/")
return parts[len(parts)-1]
}
func sendURLResponse(w http.ResponseWriter, req *http.Request, urlResponse *urlResponse) {
if eTag, ok := req.Header["ETag"]; ok && urlResponse.Hash() == eTag[0] {
w.WriteHeader(http.StatusNotModified)
io.WriteString(w, "")
return
}
cacheControl := fmt.Sprintf(
"max-age:%d, public",
urlResponseCacheControlMaxAge,
)
w.Header().Set("Cache-Control", cacheControl)
w.Header().Set("Content-Type", "application/json;charset=utf-8")
w.Header().Set("ETag", urlResponse.Hash())
w.WriteHeader(http.StatusOK)
encoder := json.NewEncoder(w)
err := encoder.Encode(urlResponse)
if err != nil {
SendError(w, NewError(
URLResponseEncoding,
"Error encoding response",
map[string]string{"error": err.Error()},
))
return
}
}
Basically, when the browser sends a request to the API (using GET), I return an ETag and Cache-Control header in the response; the Cache Control header sets a max age of two days.
What I expect to happen is that in subsequent requests, the browser uses the cached response. After 2 days have elapsed, the browser should send the ETag in the request header to check if the response has changed.
However, what I'm observing is that each time I click on the submit button, the browser resends the request. On Google Chrome Developer Console, I've unchecked 'Disable Caching' and yet it still sends requests each time.
Whatsmore is that the browser is not sending the ETag back with the request headers.
Is there something that I'm missing that's causing the cache to not work as expected?
cacheControl := fmt.Sprintf(
"max-age:%d, public",
The Cache-Control header must contain the caching time with max-age=... not max-age:... as you use (= vs :). The value you've tried to set in the wrong way will be simply ignored.
Whatsmore is that the browser is not sending the ETag back with the request headers.
if eTag, ok := req.Header["ETag"]; ok && urlResponse.Hash() == eTag[0] {
w.WriteHeader(http.StatusNotModified)
The browser will not send etag back within an ETag header. This header is only used to set the etag. The browser will instead ask the server to provide the resource if it was modified compared and will do this by putting the received etag into a If-None-Match: ... header - meaning: please return the resource if it does not match the given etag.
For ETags to be used and the browser to send the If-Modified-Since and If-None-Match headers, the cache control must be set to no-cache.
Either by the server using the Cache-Control: no-cache header or by the browser through the Request.cache option.
Related
I've got a POST request from my frontend app to my backend app written in Go using Gin. I was getting an error saying:
"No 'Access-Control-Allow-Origin' header is present on the requested resource"
so that pointed me to implement CORS in my backend app. So I did by using "github.com/gin-contrib/cors":
web.go:
func NewApp() *App {
db := SetUpDB()
router := gin.Default()
router.Use(cors.New(cors.Config{
//AllowOrigins: []string{"http://localhost:3000", "http://127.0.0.1:3000"},
AllowMethods: []string{"PUT", "POST", "GET", "OPTIONS","DELETE"},
AllowHeaders: []string{"Origin"},
AllowAllOrigins: true,
//ExposeHeaders: []string{"Content-Length"},
AllowCredentials: true,
MaxAge: 12 * time.Hour,
}))
return &App{
Db: db,
Router: router,
}
}
and in main.go I've got:
app := core.NewApp()
//app.Router.Use(CORS())
defer func() {
app.Db.Close()
log.Printf("core: database stopping")
}()
app.Router.Use(func(c *gin.Context) {
c.Set("db", app.Db)
})
app.Router.GET("/", func(ctx *gin.Context) {
ctx.JSON(http.StatusOK, gin.H{"data": "welcome TEST"})
})
// Initialize all api routes
routes.InitializeRoutes(app.Router.Group("/api/v1"))
as you can see I only set PUT in AllowMethods with the intention of testing CORS was actually working. By allowing only PUT I was expecting no methods other than PUT were allowed but I was wrong. I've performed a GET request from my frontend app and it goes through (it returns data), this leads me to think than the CORS implementation is not being picked up.
While browsing around, I've found people not using the package "github.com/gin-contrib/cors" but creating their own middleware:
func CORS() gin.HandlerFunc {
return func(c *gin.Context) {
fmt.Println(c.Request.Header)
c.Writer.Header().Set("Access-Control-Allow-Origin", "*")
c.Writer.Header().Set("Access-Control-Allow-Credentials", "true")
c.Writer.Header().Set("Access-Control-Allow-Headers", "Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization, accept, Origin, Cache-Control, X-Requested-With")
//c.Writer.Header().Set("Access-Control-Allow-Methods", "POST, OPTIONS, GET, PUT, DELETE")
c.Writer.Header().Set("Access-Control-Allow-Methods", "PUT, DELETE")
if c.Request.Method == "OPTIONS" {
c.AbortWithStatus(204)
return
}
c.Next()
}
}
and then:
func NewApp() *App {
db := SetUpDB()
router := gin.Default()
router.Use(CORS())
return &App{
Db: db,
Router: router,
}
}
I tried this as well with no luck. Same results are coming back.
Furthermore, when I perform the GET and print the method in my backend (c.Request.Method) the result is GET. But when I perform a POST and print the method I'm getting OPTIONS
What am I missing? Why router is not using the provided middleware?
There are two pieces to this question:
The first one was indicated above by Heiko: Get is a simple request so the result is always gonna be returned for these kind of requests.
Now, after testing back my POST, I was still getting errors. I had checked over and over the CORS config, changing things here and there just to find out that the routes for Category were define such as:
categoryRouter.POST("/", controllers.CreateNewCategory)
categoryRouter.GET("/", controllers.ListAllCategories)
as you can see there is a trailing / which was causing my request to be redirected and an error to be returned since the url used for the request was http://127.0.0.1:8080/api/v1/categories. I updated the routes to be:
categoryRouter.POST("", controllers.CreateNewCategory)
categoryRouter.GET("", controllers.ListAllCategories)
and now it is working as expected.
The Access-Control-Allow-Methods header is only checked for CORS requests that cannot result from a Javascript-less HTML page (so-called non-simple requests). For simple requests such as GET with only standard headers, only the Access-Control-Allow-Origin header is checked and the Access-Control-Allow-Methods header plays no role.
I am having trouble getting access to a custom respoone header being provided by our backend server. Server is written in Go. I am attempting to access it using axios as my HTTP client. I can see the header in the chrome dev-tools console, but I cannot get access to the header through axios.
chrome dev tools response headers. I am looking to get the x-***-token at the bottom
headers I have access to through axios (cors enabled)
I can, however, access the header through axios when I run chrome with cors disabled:
open -n -a /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --user-data-dir="/tmp/chrome_dev_test" --disable-web-security
headers I have access to through axios (cors disabled)
This tells me that its not an issue with axios, but with the header configuration I have.
Below is how we have configured our headers, I am trying to get access the x-custom-token header:
func allowCORS(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" && contains(cors, origin) {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
if contains(cors, "*") {
w.Header().Set("Access-Control-Allow-Origin", "*")
}
if r.Method == "OPTIONS" || r.Method == "GET" && r.Header.Get("Access-Control-Request-Method") != "" &&
w.Header().Get("Access-Control-Allow-Origin") != "" {
headers := []string{"Content-Type", "Accept-Encoding", "X-CSRF-Token", "Authorization", "accept", "origin", "Cache-Control", "X-Requested-With", "x-custom-token"}
w.Header().Set("Access-Control-Allow-Headers", strings.Join(headers, ","))
methods := []string{"GET", "HEAD", "POST", "PUT", "DELETE"}
w.Header().Set("Access-Control-Allow-Methods", strings.Join(methods, ","))
expose := []string{"x-custom-token"}
w.Header().Set("Access-Control-Expose-Headers", strings.Join(expose, ","))
return
}
h.ServeHTTP(w, r)
})}
Can anyone tell me what header I have set incorrectly?
Your issue is that you are only exposing for OPTIONS requests.
In your if statement, one of your criteria are r.Header.Get("Access-Control-Request-Method") != "".
This header is typically only ever used in pre-flight requests and so your expose headers are not getting written on GET or other methods.
In addition, your statement returns after writing the headers. This is correct for pre-flight but would cause no content to be sent for other methods.
Solution - adjust your logic to handle exposing headers separately from the pre-flight headers.
I created a simple http2 server,
If I send a request to it with curl, it responds with some headers, although I did not set them explicity. How can I acces them inside the requesthandling function ( sayhello )? My code ( I've never used golang before)
server.go
package main
import (
"net/http"
"strings"
"fmt"
"github.com/gorilla/mux"
"golang.org/x/net/http2"
)
func sayHello(w http.ResponseWriter, r *http.Request) {
message := r.URL.Path
message = strings.TrimPrefix(message, "/")
message = "Hello " + message
w.Header().Set("myFirst", "golangQuestion")
w.Write([]byte(message))
for k, v := range w.Header() {
fmt.Println("[RESPONSE][Header]", k,":", v)
}
}
func main() {
router := mux.NewRouter()
router.PathPrefix("/").HandlerFunc(sayHello) // catch everything else rule
var srv = &http.Server{
Addr: "127.0.0.1:8081",
}
http2.ConfigureServer(srv, nil)
srv.Handler = router
sslCert := "./ssl.cert"
sslKey := "./ssl.key"
if err := srv.ListenAndServeTLS(sslCert, sslKey); err != nil {
panic(err)
}
}
Sending request:
curl --head --insecure https://127.0.0.1:8081
HTTP/1.1 200 OK
Myfirst: golangQuestion
Date: Tue, 18 Jun 2019 09:18:29 GMT
Content-Length: 6
Content-Type: text/plain; charset=utf-8
I can see that some headers are sent back, the one which I set explicitly is also recieved, but the output of
go run server.go
[RESPONSE][Header] Myfirst : [golangQuestion]
How can I acces the other headers, which were not explicitly set, but recieved by curl as well? I loopd through w.Headers, but it did not contain the implicitly set headers
for k, v := range w.Header() {
fmt.Println("[RESPONSE][Header]", k,":", v)
}
My expectation that the output of go run server.go shall be something like this:
[RESPONSE][Header] Myfirst : [golangQuestion]
[RESPONSE][Header] Date: [2019.02.12 ]
[RESPONSE][Header] Content-Length: [6]
Those headers are sent automatically when you call ResponseWriter.Write(). Quoting from its doc:
// Write writes the data to the connection as part of an HTTP reply.
//
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
//
// Depending on the HTTP protocol version and the client, calling
// Write or WriteHeader may prevent future reads on the
// Request.Body. For HTTP/1.x requests, handlers should read any
// needed request body data before writing the response. Once the
// headers have been flushed (due to either an explicit Flusher.Flush
// call or writing enough data to trigger a flush), the request body
// may be unavailable. For HTTP/2 requests, the Go HTTP server permits
// handlers to continue to read the request body while concurrently
// writing the response. However, such behavior may not be supported
// by all HTTP/2 clients. Handlers should read before writing if
// possible to maximize compatibility.
Write([]byte) (int, error)
ResponseWriter.Header() contains only the headers set explicitly. The Content-Type and Content-Length were sent by w.Write().
Note: if you want to suppress such automatic headers, you have to set their values to nil, e.g.:
w.Header()["Date"] = nil
Also note that if you set the values of such headers manually, those values will be sent without being changed.
I'm trying to set a header in my Go web server. I'm using gorilla/mux and net/http packages.
I'd like to set Access-Control-Allow-Origin: * to allow cross domain AJAX.
Here's my Go code:
func saveHandler(w http.ResponseWriter, r *http.Request) {
// do some stuff with the request data
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/save", saveHandler)
http.Handle("/", r)
http.ListenAndServe(":"+port, nil)
}
The net/http package has documentation describing sending http request headers as if it were a client - I'm not exactly sure how to set response headers?
Never mind, I figured it out - I used the Set() method on Header() (doh!)
My handler looks like this now:
func saveHandler(w http.ResponseWriter, r *http.Request) {
// allow cross domain AJAX requests
w.Header().Set("Access-Control-Allow-Origin", "*")
}
Maybe this will help someone as caffeine deprived as myself sometime :)
All of the above answers are wrong because they fail to handle the OPTIONS preflight request, the solution is to override the mux router's interface. See AngularJS $http get request failed with custom header (alllowed in CORS)
func main() {
r := mux.NewRouter()
r.HandleFunc("/save", saveHandler)
http.Handle("/", &MyServer{r})
http.ListenAndServe(":8080", nil);
}
type MyServer struct {
r *mux.Router
}
func (s *MyServer) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
if origin := req.Header.Get("Origin"); origin != "" {
rw.Header().Set("Access-Control-Allow-Origin", origin)
rw.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
rw.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
// Stop here if its Preflighted OPTIONS request
if req.Method == "OPTIONS" {
return
}
// Lets Gorilla work
s.r.ServeHTTP(rw, req)
}
Do not use '*' for Origin, until You really need a completely public behavior.
As Wikipedia says:
"The value of "*" is special in that it does not allow requests to supply credentials,
meaning HTTP authentication, client-side SSL certificates, nor does it allow cookies
to be sent."
That means, you'll get a lot of errors, especially in Chrome when you'll try to implement for example a simple authentication.
Here is a corrected wrapper:
// Code has not been tested.
func addDefaultHeaders(fn http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
}
w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token")
w.Header().Set("Access-Control-Allow-Credentials", "true")
fn(w, r)
}
}
And don't forget to reply all these headers to the preflight OPTIONS request.
If you don't want to override your router (if you don't have your app configured in a way that supports this, or want to configure CORS on a route by route basis), add an OPTIONS handler to handle the pre flight request.
Ie, with Gorilla Mux your routes would look like:
accounts := router.Path("/accounts").Subrouter()
accounts.Methods("POST").Handler(AccountsCreate)
accounts.Methods("OPTIONS").Handler(AccountsCreatePreFlight)
Note above that in addition to our POST handler, we're defining a specific OPTIONS method handler.
And then to actual handle the OPTIONS preflight method, you could define AccountsCreatePreFlight like so:
// Check the origin is valid.
origin := r.Header.Get("Origin")
validOrigin, err := validateOrigin(origin)
if err != nil {
return err
}
// If it is, allow CORS.
if validOrigin {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "POST")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
What really made this all click for me (in addition to actually understanding how CORS works) is that the HTTP Method of a preflight request is different from the HTTP Method of the actual request. To initiate CORS, the browser sends a preflight request with HTTP Method OPTIONS, which you have to handle explicitly in your router, and then, if it receives the appropriate response "Access-Control-Allow-Origin": origin (or "*" for all) from your application, it initiates the actual request.
I also believe that you can only do "*" for standard types of requests (ie: GET), but for others you'll have to explicitly set the origin like I do above.
Set a proper golang middleware, so you can reuse on any endpoint.
Helper Type and Function
type Adapter func(http.Handler) http.Handler
// Adapt h with all specified adapters.
func Adapt(h http.Handler, adapters ...Adapter) http.Handler {
for _, adapter := range adapters {
h = adapter(h)
}
return h
}
Actual middleware
func EnableCORS() Adapter {
return func(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if origin := r.Header.Get("Origin"); origin != "" {
w.Header().Set("Access-Control-Allow-Origin", origin)
w.Header().Set("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
w.Header().Set("Access-Control-Allow-Headers",
"Accept, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
}
// Stop here if its Preflighted OPTIONS request
if r.Method == "OPTIONS" {
return
}
h.ServeHTTP(w, r)
})
}
}
Endpoint
REMEBER! Middlewares get applyed on reverse order( ExpectGET() gets fires first)
mux.Handle("/watcher/{action}/{device}",Adapt(api.SerialHandler(mux),
api.EnableCORS(),
api.ExpectGET(),
))
I create wrapper for this case:
func addDefaultHeaders(fn http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
fn(w, r)
}
}
I had the same issue as described above the solutions given above are correct, the set up I have is as follows
1) Angularjs for the Client
2) Beego framework for GO server
Please following these points
1) CORS settings must be enabled only on GO server
2) Do NOT add any type of headers in angularJS except for this
.config(['$httpProvider', function($httpProvider) {
$httpProvider.defaults.useXDomain = true;
delete $httpProvider.defaults.headers.common['X-Requested-With'];
}])
In you GO server add the CORS settings before the request starts to get processed so that the preflight request receives a 200 OK after which the the OPTIONS method will get converted to GET,POST,PUT or what ever is your request type.
I know this is a different twist on the answer, but isn't this more of a concern for a web server? For example, nginx, could help.
The ngx_http_headers_module module allows adding the “Expires” and “Cache-Control” header fields, and arbitrary fields, to a response header
...
location ~ ^<REGXP MATCHING CORS ROUTES> {
add_header Access-Control-Allow-Methods POST
...
}
...
Adding nginx in front of your go service in production seems wise. It provides a lot more feature for authorizing, logging,and modifying requests. Also, it gives the ability to control who has access to your service and not only that but one can specify different behavior for specific locations in your app, as demonstrated above.
I could go on about why to use a web server with your go api, but I think that's a topic for another discussion.
I'm trying to implement the client-side caching of web service calls, and based on information from the web, I was able to do it according to the SetCachingPolicy() function as shown in code 1 below.
I was able to successfully get it working with a web method, RetrieveX, but not with method RetrieveY. I noticed that RetrieveX has no parameters and RetrieveY has one string parameter, and on inspection under Fiddler, the difference seems to be that the HTTP GET request of the web service call from RetrieveY has a query string for the parameter.
All HTTP GET web service calls so far without a query string is doing the caching properly, but not this call that has a query string in it.
Examination under Fiddler indicates that RetrieveX has the following caching information in output 1, and RetrieveY has the information in output 2.
Is this a limitation of this caching method or can I do something to get the client side caching of RetrieveY working?
Code 1: SetCachingPolicy
private void SetCachingPolicy()
{
HttpCachePolicy cache = HttpContext.Current.Response.Cache;
cache.SetCacheability(HttpCacheability.Private);
cache.SetExpires(DateTime.Now.AddSeconds((double)30));
FieldInfo maxAgeField = cache.GetType().GetField(
"_maxAge", BindingFlags.Instance | BindingFlags.NonPublic);
maxAgeField.SetValue(cache, new TimeSpan(0, 0, 30));
}
Code 2: RetrieveX
[System.Web.Services.WebMethod]
[System.Web.Script.Services.ScriptMethod(UseHttpGet = true)]
public string[] RetrieveX()
{
SetCachingPolicy();
// Implementation details here.
return array;
}
Code 3: RetrieveY
[System.Web.Services.WebMethod]
[System.Web.Script.Services.ScriptMethod(UseHttpGet = true)]
public string[] RetrieveY(string arg1)
{
SetCachingPolicy();
// Implementation details here.
return array;
}
Output 1: RetrieveX caching info
HTTP/200 responses are cacheable by default, unless Expires, Pragma, or Cache-Control headers are present and forbid caching.
HTTP/1.0 Expires Header is present: Wed, 12 Sep 2012 03:16:50 GMT
HTTP/1.1 Cache-Control Header is present: private, max-age=30
private: This response MUST NOT be cached by a shared cache.
max-age: This resource will expire in .5 minutes. [30 sec]
Output 2: RetrieveY caching info
HTTP/200 responses are cacheable by default, unless Expires, Pragma, or Cache-Control headers are present and forbid caching.
HTTP/1.0 Expires Header is present: -1
Legacy Pragma Header is present: no-cache
HTTP/1.1 Cache-Control Header is present: no-cache
no-cache: This response MUST NOT be reused without successful revalidation with the origin server.
I ran into this issue as well, I thought I'd share what worked for me. The underlying issue is that VaryByParams is not being set on the response. If you add this to your SetCachingPolicy() method RetrieveY should begin working as expected:
cache.VaryByParams["*"] = true