I am trying to send request from Go/wasm with Go's net/http package (I am not sure if I should use javascript's fetch function from wasm) . I cannot reach all of the response Headers and Cookies from Go/WASM even though I can see properly all of the headers and cookies on the browser (on network tab of browser and also i can see all of the headers with curl request) . When I try to print all of the headers i can only see 2 headers on the console.Those are "Content-Length" and "Content-Type" . Does anybody knows what is the reason of this ?
Here is example code of server side:
import "github.com/gorilla/sessions"
var store = sessions.NewCookieStore([]byte("super-secret-key-4"))
func (a *App) TestHandler(w http.ResponseWriter, r *http.Request) {
cookieSession, _ := store.Get(r, "session")
cookieSession.Values["test"] = "test"
cookieSession.Save(r, w)
w.Header().Set("Test", "test")
io.WriteString(w, `{"test":"test"}`)
return
}
Client Side:
func TestRequest(userName string) {
type Payload struct {
Name string `json:"name"`
}
payload := Payload{
Name: userName,
}
payloadBytes, _ := json.Marshal(payload)
body := bytes.NewReader(payloadBytes)
req, _:= http.NewRequest("POST","localhost:8080/Test", body)
req.Header.Set("Content-Type", "application/json")
resp, _:= http.DefaultClient.Do(req)
//a, _ := ioutil.ReadAll(resp.Body)
//bodyString := string(a)
for name, values := range resp.Header {
for _, value := range values {
log.Println(name, value)
}
}
for _, cookie := range resp.Cookies() {
log.Println(cookie.Name)
}
defer resp.Body.Close()
}
This is what i get on the browser console:
wasm_exec.js:51 2021/08/04 21:08:48 Content-Length 274
wasm_exec.js:51 2021/08/04 21:08:48 Content-Type text/plain; charset=utf-8
I finally see that i need to add a special header to response for use my custom headers on client side.If i add this line to my middleware i can see my custom headers on client side.
w.Header().Set("Access-Control-Expose-Headers",
"MyHeader,Myheader2")
Related
I'm starting to work with fasthttp in Golang and I can't figure out how to send key:value format. In default net/http I did it via url.values. I would appreciate it if you could help me with some sample code!
Image from Burp Suite (How it must look like)
var client *fasthttp.Client
var headerContentTypeJson = []byte("application/json")
type loginData struct {
login string
pass string
}
func main() {
readTimeout, _ := time.ParseDuration("500ms")
writeTimeout, _ := time.ParseDuration("500ms")
maxIdleConnDuration, _ := time.ParseDuration("1h")
client = &fasthttp.Client{
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
MaxIdleConnDuration: maxIdleConnDuration,
NoDefaultUserAgentHeader: true, //
DisableHeaderNamesNormalizing: true, //
DisablePathNormalizing: true,
Dial: (&fasthttp.TCPDialer{
Concurrency: 4096,
DNSCacheDuration: time.Hour,
}).Dial,
}
reqTimeout := time.Duration(100) * time.Millisecond
reqData := &loginData{
login: "login",
pass: "password",
}
reqDataByte, _ := json.Marshal(reqData)
req := fasthttp.AcquireRequest()
req.SetRequestURI("https://oskelly.ru/api/v2/account/rawauth")
req.Header.SetMethod(fasthttp.MethodPost)
req.Header.SetContentTypeBytes(headerContentTypeJson)
req.SetBodyRaw(reqDataByte)
resp := fasthttp.AcquireResponse()
err := client.DoTimeout(req, resp, reqTimeout)
fasthttp.ReleaseRequest(req)
if err == nil {
statusCode := resp.StatusCode()
respBody := resp.Body()
fmt.Printf("DEEBUG Response: %s\n", respBody)
if statusCode == http.StatusOK {
respData := &loginData{}
err := json.Unmarshal(respBody, respData)
if err == io.EOF || err == nil {
fmt.Printf("DEBUG Parsed data Response %v\n")
} else {
fmt.Printf("ERR invalid HTTP response code: %d\n", statusCode)
}
}
fasthttp.ReleaseResponse(resp)
}}
enter image description here
Tried to figure out how to integrate url.values into fasthttp
For request Body parameters:
req.Header.SetContentType("application/x-www-form-urlencoded; charset=UTF-8")
Form the request body in key=val string and not json. Can do a simple Stringer impl or req.PostArgs().Add(key, val)
req.PostArgs().Add("login", reqData.Login)
req.PostArgs().Add("pass", reqData.Pass)
Output request
POST / HTTP/1.1
Host: ....
Content-Length: 25
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
login=login&pass=password
I observed that Content-Length header is not getting set for PATCH requests with empty/nil payload. Even if we manually set it by req.Header.Set("content-length", "0") it is not actually getting set in the out going request.
This strange behaviour (Go bug?) happens only for PATCH requests and only when the payload is empty or nil (or set to http.NoBody)
package main
import (
"fmt"
"io/ioutil"
"net/http"
"strings"
)
func main() {
url := "http://localhost:9999"
method := "PATCH"
payload := strings.NewReader("")
client := &http.Client {
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Authorization", "Bearer my-token")
req.Header.Set("Content-Length", "0") //this is not honoured
res, err := client.Do(req)
defer res.Body.Close()
body, err := ioutil.ReadAll(res.Body)
fmt.Println(string(body))
}
This is reproducible even in the latest go version 1.15.
Just run the above code against a simple http server and see for yourself.
Is there any solution/workaround to send a PATCH request with Content-Length set to 0 ?
You can tell the HTTP client to include a Content-Length header with value 0 by setting TransferEncoding to identity as follows:
url := "http://localhost:9999"
method := "PATCH"
client := &http.Client{}
req, err := http.NewRequest(method, url, http.NoBody)
if err != nil {
panic(err)
}
req.TransferEncoding = []string{"identity"}
req.Header.Set("Authorization", "Bearer my-token")
// req.Header.Set("Content-Length", "0")
Note the following changes to your original code:
the important one: req.TransferEncoding = []string{"identity"}
the idiomatic way of specifying an empty body: http.NoBody (no impact on sending the length)
commented out req.Header.Set("Content-Length", "0"), the client fills it in by itself
also changed to panic on an error, you probably don't want to continue
The transfer encoding of identity is not written to the request, so except for the header Content-Length = 0, the request looks the same as before.
This is unfortunately not documented (feel free to file an issue with the Go team), but can be seen in the following code:
The tedious details:
transferWriter.writeHeader checks the following to write the Content-Length header:
// Write Content-Length and/or Transfer-Encoding whose values are a
// function of the sanitized field triple (Body, ContentLength,
// TransferEncoding)
if t.shouldSendContentLength() {
if _, err := io.WriteString(w, "Content-Length: "); err != nil {
return err
}
if _, err := io.WriteString(w, strconv.FormatInt(t.ContentLength, 10)+"\r\n"); err != nil {
return err
}
In turn, shouldCheckContentLength looks at the transfer encoding in case of zero length:
if t.ContentLength == 0 && isIdentity(t.TransferEncoding) {
if t.Method == "GET" || t.Method == "HEAD" {
return false
}
return true
}
The isIdentity verifies that TransferEncoding is exactly []string{"identity"}:
func isIdentity(te []string) bool { return len(te) == 1 && te[0] == "identity" })
Go version: go1.8.1 windows/amd64
Sample code for HTTP request is:
func (c *Client) RoundTripSoap12(action string, in, out Message) error {
fmt.Println("****************************************************************")
headerFunc := func(r *http.Request) {
r.Header.Add("Content-Type", fmt.Sprintf("text/xml; charset=utf-8"))
r.Header.Add("SOAPAction", fmt.Sprintf(action))
r.Cookies()
}
return doRoundTrip(c, headerFunc, in, out)
}
func doRoundTrip(c *Client, setHeaders func(*http.Request), in, out Message) error {
req := &Envelope{
EnvelopeAttr: c.Envelope,
NSAttr: c.Namespace,
Header: c.Header,
Body: Body{Message: in},
}
if req.EnvelopeAttr == "" {
req.EnvelopeAttr = "http://schemas.xmlsoap.org/soap/envelope/"
}
if req.NSAttr == "" {
req.NSAttr = c.URL
}
var b bytes.Buffer
err := xml.NewEncoder(&b).Encode(req)
if err != nil {
return err
}
cli := c.Config
if cli == nil {
cli = http.DefaultClient
}
r, err := http.NewRequest("POST", c.URL, &b)
if err != nil {
return err
}
setHeaders(r)
if c.Pre != nil {
c.Pre(r)
}
fmt.Println("*************", r)
resp, err := cli.Do(r)
if err != nil {
fmt.Println("error occured is as follows ", err)
return err
}
fmt.Println("response headers are: ", resp.Header.Get("sprequestguid"))
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
// read only the first Mb of the body in error case
limReader := io.LimitReader(resp.Body, 1024*1024)
body, _ := ioutil.ReadAll(limReader)
return fmt.Errorf("%q: %q", resp.Status, body)
}
return xml.NewDecoder(resp.Body).Decode(out)
I will call the RoundTripSoap12 function on the corresponding HTTP client.
When I send a request for the first time I will be getting some headers in the HTTP response, so these HTTP response headers should be sent as-is in my next HTTP request.
You may be interested in the httputil package and the reverse proxy example provided if you wish to proxy requests transparently:
https://golang.org/src/net/http/httputil/reverseproxy.go
You can copy the headers from one request to another one fairly easily - the Header is a separate object, if r and rc are http.Requests and you don't mind them sharing a header (you may need to clone instead if you want independent requests):
rc.Header = r.Header // note shallow copy
fmt.Println("Headers", r.Header, rc.Header)
https://play.golang.org/p/q2KUHa_qiP
Or you can look through keys and values and only copy certain headers, and/or do a clone instead to ensure you share no memory. See the http util package here for examples of this - see the functions cloneHeader and copyHeader inside reverseproxy.go linked above.
We can register CheckRedirect to check the next request when the request is redirected. Is there a way that I can get the response for the first request when it's redirected?
The way it is currently implemented, it doesn't seem possible to have a look at the response by default (unless you implement yourself what Do() does).
See src/net/http/client.go#L384-L399:
if shouldRedirect(resp.StatusCode) {
// Read the body if small so underlying TCP connection will be re-used.
// No need to check for errors: if it fails, Transport won't reuse it anyway.
const maxBodySlurpSize = 2 << 10
if resp.ContentLength == -1 || resp.ContentLength <= maxBodySlurpSize {
io.CopyN(ioutil.Discard, resp.Body, maxBodySlurpSize)
}
resp.Body.Close()
if urlStr = resp.Header.Get("Location"); urlStr == "" {
err = fmt.Errorf("%d response missing Location header", resp.StatusCode)
break
}
base = req.URL
via = append(via, req)
continue
}
What do you want to do with the first response? It will be pretty boring.
I think the most sensible thing would be to disable automatically following redirects (always return a non-nil error from CheckRedirect) and handle the redirection yourself in which case you have full access to all requests/responses.
My workaround for this:
Any http request / response could be patched within a custom RoundTripper, including redirects:
type RedirectChecker struct{}
func (RedirectChecker) RoundTrip(req *http.Request) (*http.Response, error) {
// e.g. patch the request before send it
req.Header.Set("user-agent", "curl/7.64.1")
resp, err := http.DefaultTransport.RoundTrip(req)
if err == nil && resp != nil {
switch resp.StatusCode {
case
http.StatusMovedPermanently,
http.StatusFound,
http.StatusTemporaryRedirect,
http.StatusPermanentRedirect:
// e.g. stop further redirections
// roughly equivalent to http.ErrUseLastResponse
resp.StatusCode = http.StatusOK
// e.g. read the Set-Cookie headers
// unfortunately cookie jars do not handle redirects in a proper manner
// and that's why I came to this question...
fmt.Printf("%+v", resp.Cookies())
}
}
}
httpClient := &http.Client{Transport: RedirectChecker{}}
httpClient.Do(...)
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/