I need to send an unencoded GET request to an API which is not conforming to the specs. The destination API doesn't accept encoded URLs, I can only send requests as it is without any query encoding. By default net/http encodes requests. I tried this:
client := &http.Client{}
req, _ := http.NewRequest("GET", `https://api.website.com/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`, nil)
req.Header.Set("Authorization", "Bearer " + viper.GetString("ACCESS_TOKEN"))
response, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(response.Body)
fmt.Printf("%s\n", data)
I tried to construct my own Request and manipulating RawQuery, without success. I keep getting a bad request, while if I send the same request through Postman, everything's fine.
EDIT: That's what I tried to do to set the URL myself, and if I print it I can see it's not encoded. The error I get is: panic: runtime error: invalid memory address or nil pointer dereference.
client := &http.Client{}
req, _ := http.NewRequest("GET", "", nil)
req.URL = &url.URL{
Host: "api.website.com",
Scheme: "https",
RawQuery: `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`,
}
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
fmt.Printf("%s\n", data)
Specify the protocol and host when creating the request. Set URL.Opaque to the desired request URI:
req, _ := http.NewRequest("GET", "https://api.website.com/", nil)
req.URL.Opaque = `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
Run it on the playground.
Related
I'm having a real head scratcher of a problem. I'm syncing data for clients from an external api. Currently it's structured like this
func SyncInitHandler(w http.ResponseWriter, r *http.Request) {
accounts := data.GetAccounts()
done := make(chan error)
var numRequests int
for _, account := range accounts {
numRequests++
go func(a models.Account) {
subTaskURL := fmt.Sprintf("http://localhost:8080/sync/account/%v", a.Id)
_, err := http.Get(subTaskURL)
done <- err
}(account)
}
for i := 0; i < numRequests; i++ {
<-done
}
}
func SyncAccountHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
accountId, _ := strconv.ParseInt(vars["account-id"], 10, 64)
account, _ := data.GetAccount(accountId)
externalAPIURL := fmt.Sprintf("https://www.example.com/account/%v", account.ApiToken)
req, _ := http.NewRequest("POST", apiUrl, nil)
client := http.Client{} // urlfetch.Client(c)
resp, _ := client.Do(req)
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
apiResp := models.ApiRespModel{}
err = json.Unmarshal(body, &apiResp)
if err != nil {
log.Printf("An Error Occurred Parsing JSON: %v", err)
return
}
log.Printf("Account API Token: %v, Response API Token: %v", account.ApiToken, apiResp.Token)
}
What should happen is that the Account API Token printed and the response Api Token should be the same (i.e. the returned data is for the same account that requested it). What is happening is that the tokens don't match, and the wrong data is being returned to the wrong account.
How is this even possible? As you can see, the external API requests are being spawned in separate requests on the local server. Go http requests are supposed to be thread safe aren't they?
I need to use a proxy with auth using PostForm method.
If I use something like (simplified):
request, err := http.NewRequest("GET", url.String(), nil)
response, err := client.Do(request)
I can with ease do request.Header.Add("Proxy-Authorization", basicAuth) and it works fine.
But now, I am editing third-party package, and I try to add proxy to the existing code:
proxyStr := "http://proxy.com:8080"
proxyURL, _ := url.Parse(proxyStr)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
bot.Client = &http.Client{
Transport: transport,
}
resp, err := bot.Client.PostForm(method, params)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
resp.Header.Add("Proxy-Authorization", basicAuth)
It does not work, and it fails, to my mind, at string resp.Header.Add("Proxy-Authorization", basicAuth).
Proxy without auth works fine, in this example.
Does anybody know, can I use proxy with auth in this case?
You can create the client once by using the following code. Then substitute your HTTP client in the third-party package.
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(&url.URL{
Scheme: "http",
User: url.UserPassword("username", "password"),
Host: "146.137.9.45:65233",
}),
},
}
or you can parse the URL as well
url, _ := url.Parse("http://username:password#146.137.9.45:65233")
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(url),
}
}
You are trying to add a header to a response, which isn't what you send to the server but what you receive. You have to add headers and data to the request, which you have to assemble first and then execute it like this:
data := url.Values{} // the form data
data.Add("foo-key", "some data")
req, err := http.NewRequest("POST","https://yoururl", strings.NewReader(data.Encode()))
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
req.Header.Add("Proxy-Authorization", basicAuth)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := bot.Client.Do(req)
Then you just use the response (resp)
Thanks to all!
I found such a solution (may be it would be useful to someone):
// Uncomment to use proxy with auth
/*
proxyStr := "http://proxy.com:3128"
proxyURL, _ := url.Parse(proxyStr)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
hdr := http.Header{}
hdr.Add("Proxy-Authorization", basicAuth)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
ProxyConnectHeader: hdr,
}
bot.Client = &http.Client{
Transport: transport,
}
*/
resp, err := bot.Client.PostForm(method, params)
I have a standard piece of go http request that I am trying to get right in the code below.
func IssueSearchQuery(conf Config) (string, error) {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
var client = &http.Client{
Timeout: time.Second * 60,
Transport: tr,
}
URL := conf.URL + "/api/search"
v := url.Values{}
v.Set("query_expression", "select * from test")
req, err := http.NewRequest("POST", URL,
bytes.NewBufferString(v.Encode()))
req.SetBasicAuth(conf.User, conf.Password)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s, err
}
In the above code, I am connecting to the server, basic authentication works, but the server responds complaining saying the required query_expression query parameter is missing. If I do curl -u user:pass -d "query_expression=select * from test" --insecure -XPOST 'https://ip:port/api/search' the server responds with the required result.
The code also works if I don't use url.Values{} and instead manually encode my query into the url like URL := conf.URL + "/api/ariel/searches?query_expression=select%20*20from%20test"
I am not able to figure out what I am doing wrong with my query parameters. Can someone help? Thanks!
You appear to be trying to set the POST body to the url-encoded values. Either set your Content-Type header to application/x-www-form-urlencoded, and put them in the body as you're doing (thanks #hobbs), or use your url-encoded values in the actual url rather than the body:
u, err := url.Parse(conf.URL + "/api/search")
v := url.Values{}
v.Set("query_expression", "select * from test")
u.RawQuery = v.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
Typically people expect POST data to be in the body, so just adding the req.Header.Set("Content-Type", "application/x-www-form-urlencoded") is probably the best.
I am writing a wrapper for an API in go. The api uses basic auth and then POST request requires PostForm value. I'm doing something like this:
func NewFoo(name string) string {
client := &http.Client{}
URL := HOST + "foo/"
req, err := http.NewRequest("POST", URL, nil)
v := url.Values{}
v.Set("name", name)
req.Form = v
req.SetBasicAuth(EMAIL, PASSWORD)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s
}
I had a similar, GET request without the form value and it works. When I run it, it tells me that the "name" value is required. (so it's not getting it)
Is there any reason this does not work?
From http://golang.org/pkg/net/http/#Request
// Form contains the parsed form data, including both the URL
// field's query parameters and the POST or PUT form data.
// This field is only available after ParseForm is called.
// The HTTP client ignores Form and uses Body instead.
Form url.Values
You have to pass your url.Values to the request's body instead.
func NewFoo(name string) string {
client := &http.Client{}
URL := HOST + "foo/"
v := url.Values{}
v.Set("name", name)
//pass the values to the request's body
req, err := http.NewRequest("POST", URL, strings.NewReader(v.Encode()))
req.SetBasicAuth(EMAIL, PASSWORD)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s
}
I'm doing a simple http GET in Go:
client := &http.Client{}
req, _ := http.NewRequest("GET", url, nil)
res, _ := client.Do(req)
But I can't found a way to customize the request header in the doc, thanks
The Header field of the Request is public. You may do this :
req.Header.Set("name", "value")
Pay attention that in http.Request header "Host" can not be set via Set method
req.Header.Set("Host", "domain.tld")
but can be set directly:
req.Host = "domain.tld":
req, err := http.NewRequest("GET", "http://10.0.0.1/", nil)
if err != nil {
...
}
req.Host = "domain.tld"
client := &http.Client{}
resp, err := client.Do(req)
If you want to set more than one header, this can be handy rather than writing set statements.
client := http.Client{}
req , err := http.NewRequest("GET", url, nil)
if err != nil {
//Handle Error
}
req.Header = http.Header{
"Host": {"www.host.com"},
"Content-Type": {"application/json"},
"Authorization": {"Bearer Token"},
}
res , err := client.Do(req)
if err != nil {
//Handle Error
}
Go's net/http package has many functions that deal with headers. Among them are Add, Del, Get and Set methods. The way to use Set is:
func yourHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("header_name", "header_value")
}