Getting http post with basic authentication right in GO - http

I have a standard piece of go http request that I am trying to get right in the code below.
func IssueSearchQuery(conf Config) (string, error) {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
var client = &http.Client{
Timeout: time.Second * 60,
Transport: tr,
}
URL := conf.URL + "/api/search"
v := url.Values{}
v.Set("query_expression", "select * from test")
req, err := http.NewRequest("POST", URL,
bytes.NewBufferString(v.Encode()))
req.SetBasicAuth(conf.User, conf.Password)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s, err
}
In the above code, I am connecting to the server, basic authentication works, but the server responds complaining saying the required query_expression query parameter is missing. If I do curl -u user:pass -d "query_expression=select * from test" --insecure -XPOST 'https://ip:port/api/search' the server responds with the required result.
The code also works if I don't use url.Values{} and instead manually encode my query into the url like URL := conf.URL + "/api/ariel/searches?query_expression=select%20*20from%20test"
I am not able to figure out what I am doing wrong with my query parameters. Can someone help? Thanks!

You appear to be trying to set the POST body to the url-encoded values. Either set your Content-Type header to application/x-www-form-urlencoded, and put them in the body as you're doing (thanks #hobbs), or use your url-encoded values in the actual url rather than the body:
u, err := url.Parse(conf.URL + "/api/search")
v := url.Values{}
v.Set("query_expression", "select * from test")
u.RawQuery = v.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
Typically people expect POST data to be in the body, so just adding the req.Header.Set("Content-Type", "application/x-www-form-urlencoded") is probably the best.

Related

Send unencoded GET requests in Go

I need to send an unencoded GET request to an API which is not conforming to the specs. The destination API doesn't accept encoded URLs, I can only send requests as it is without any query encoding. By default net/http encodes requests. I tried this:
client := &http.Client{}
req, _ := http.NewRequest("GET", `https://api.website.com/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`, nil)
req.Header.Set("Authorization", "Bearer " + viper.GetString("ACCESS_TOKEN"))
response, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(response.Body)
fmt.Printf("%s\n", data)
I tried to construct my own Request and manipulating RawQuery, without success. I keep getting a bad request, while if I send the same request through Postman, everything's fine.
EDIT: That's what I tried to do to set the URL myself, and if I print it I can see it's not encoded. The error I get is: panic: runtime error: invalid memory address or nil pointer dereference.
client := &http.Client{}
req, _ := http.NewRequest("GET", "", nil)
req.URL = &url.URL{
Host: "api.website.com",
Scheme: "https",
RawQuery: `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`,
}
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
fmt.Printf("%s\n", data)
Specify the protocol and host when creating the request. Set URL.Opaque to the desired request URI:
req, _ := http.NewRequest("GET", "https://api.website.com/", nil)
req.URL.Opaque = `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
Run it on the playground.

Go http proxy with auth

I need to use a proxy with auth using PostForm method.
If I use something like (simplified):
request, err := http.NewRequest("GET", url.String(), nil)
response, err := client.Do(request)
I can with ease do request.Header.Add("Proxy-Authorization", basicAuth) and it works fine.
But now, I am editing third-party package, and I try to add proxy to the existing code:
proxyStr := "http://proxy.com:8080"
proxyURL, _ := url.Parse(proxyStr)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
bot.Client = &http.Client{
Transport: transport,
}
resp, err := bot.Client.PostForm(method, params)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
resp.Header.Add("Proxy-Authorization", basicAuth)
It does not work, and it fails, to my mind, at string resp.Header.Add("Proxy-Authorization", basicAuth).
Proxy without auth works fine, in this example.
Does anybody know, can I use proxy with auth in this case?
You can create the client once by using the following code. Then substitute your HTTP client in the third-party package.
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(&url.URL{
Scheme: "http",
User: url.UserPassword("username", "password"),
Host: "146.137.9.45:65233",
}),
},
}
or you can parse the URL as well
url, _ := url.Parse("http://username:password#146.137.9.45:65233")
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(url),
}
}
You are trying to add a header to a response, which isn't what you send to the server but what you receive. You have to add headers and data to the request, which you have to assemble first and then execute it like this:
data := url.Values{} // the form data
data.Add("foo-key", "some data")
req, err := http.NewRequest("POST","https://yoururl", strings.NewReader(data.Encode()))
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
req.Header.Add("Proxy-Authorization", basicAuth)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := bot.Client.Do(req)
Then you just use the response (resp)
Thanks to all!
I found such a solution (may be it would be useful to someone):
// Uncomment to use proxy with auth
/*
proxyStr := "http://proxy.com:3128"
proxyURL, _ := url.Parse(proxyStr)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
hdr := http.Header{}
hdr.Add("Proxy-Authorization", basicAuth)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
ProxyConnectHeader: hdr,
}
bot.Client = &http.Client{
Transport: transport,
}
*/
resp, err := bot.Client.PostForm(method, params)

Go to test website status (ping)

Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.

Go Http Post request with basic auth and formvalue

I am writing a wrapper for an API in go. The api uses basic auth and then POST request requires PostForm value. I'm doing something like this:
func NewFoo(name string) string {
client := &http.Client{}
URL := HOST + "foo/"
req, err := http.NewRequest("POST", URL, nil)
v := url.Values{}
v.Set("name", name)
req.Form = v
req.SetBasicAuth(EMAIL, PASSWORD)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s
}
I had a similar, GET request without the form value and it works. When I run it, it tells me that the "name" value is required. (so it's not getting it)
Is there any reason this does not work?
From http://golang.org/pkg/net/http/#Request
// Form contains the parsed form data, including both the URL
// field's query parameters and the POST or PUT form data.
// This field is only available after ParseForm is called.
// The HTTP client ignores Form and uses Body instead.
Form url.Values
You have to pass your url.Values to the request's body instead.
func NewFoo(name string) string {
client := &http.Client{}
URL := HOST + "foo/"
v := url.Values{}
v.Set("name", name)
//pass the values to the request's body
req, err := http.NewRequest("POST", URL, strings.NewReader(v.Encode()))
req.SetBasicAuth(EMAIL, PASSWORD)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s
}

How to set headers in http get request?

I'm doing a simple http GET in Go:
client := &http.Client{}
req, _ := http.NewRequest("GET", url, nil)
res, _ := client.Do(req)
But I can't found a way to customize the request header in the doc, thanks
The Header field of the Request is public. You may do this :
req.Header.Set("name", "value")
Pay attention that in http.Request header "Host" can not be set via Set method
req.Header.Set("Host", "domain.tld")
but can be set directly:
req.Host = "domain.tld":
req, err := http.NewRequest("GET", "http://10.0.0.1/", nil)
if err != nil {
...
}
req.Host = "domain.tld"
client := &http.Client{}
resp, err := client.Do(req)
If you want to set more than one header, this can be handy rather than writing set statements.
client := http.Client{}
req , err := http.NewRequest("GET", url, nil)
if err != nil {
//Handle Error
}
req.Header = http.Header{
"Host": {"www.host.com"},
"Content-Type": {"application/json"},
"Authorization": {"Bearer Token"},
}
res , err := client.Do(req)
if err != nil {
//Handle Error
}
Go's net/http package has many functions that deal with headers. Among them are Add, Del, Get and Set methods. The way to use Set is:
func yourHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("header_name", "header_value")
}

Resources