I need to use a proxy with auth using PostForm method.
If I use something like (simplified):
request, err := http.NewRequest("GET", url.String(), nil)
response, err := client.Do(request)
I can with ease do request.Header.Add("Proxy-Authorization", basicAuth) and it works fine.
But now, I am editing third-party package, and I try to add proxy to the existing code:
proxyStr := "http://proxy.com:8080"
proxyURL, _ := url.Parse(proxyStr)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
bot.Client = &http.Client{
Transport: transport,
}
resp, err := bot.Client.PostForm(method, params)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
resp.Header.Add("Proxy-Authorization", basicAuth)
It does not work, and it fails, to my mind, at string resp.Header.Add("Proxy-Authorization", basicAuth).
Proxy without auth works fine, in this example.
Does anybody know, can I use proxy with auth in this case?
You can create the client once by using the following code. Then substitute your HTTP client in the third-party package.
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(&url.URL{
Scheme: "http",
User: url.UserPassword("username", "password"),
Host: "146.137.9.45:65233",
}),
},
}
or you can parse the URL as well
url, _ := url.Parse("http://username:password#146.137.9.45:65233")
&http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(url),
}
}
You are trying to add a header to a response, which isn't what you send to the server but what you receive. You have to add headers and data to the request, which you have to assemble first and then execute it like this:
data := url.Values{} // the form data
data.Add("foo-key", "some data")
req, err := http.NewRequest("POST","https://yoururl", strings.NewReader(data.Encode()))
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
req.Header.Add("Proxy-Authorization", basicAuth)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded")
resp, err := bot.Client.Do(req)
Then you just use the response (resp)
Thanks to all!
I found such a solution (may be it would be useful to someone):
// Uncomment to use proxy with auth
/*
proxyStr := "http://proxy.com:3128"
proxyURL, _ := url.Parse(proxyStr)
auth := "username:password"
basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))
hdr := http.Header{}
hdr.Add("Proxy-Authorization", basicAuth)
transport := &http.Transport{
Proxy: http.ProxyURL(proxyURL),
ProxyConnectHeader: hdr,
}
bot.Client = &http.Client{
Transport: transport,
}
*/
resp, err := bot.Client.PostForm(method, params)
Related
I need to send an unencoded GET request to an API which is not conforming to the specs. The destination API doesn't accept encoded URLs, I can only send requests as it is without any query encoding. By default net/http encodes requests. I tried this:
client := &http.Client{}
req, _ := http.NewRequest("GET", `https://api.website.com/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`, nil)
req.Header.Set("Authorization", "Bearer " + viper.GetString("ACCESS_TOKEN"))
response, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(response.Body)
fmt.Printf("%s\n", data)
I tried to construct my own Request and manipulating RawQuery, without success. I keep getting a bad request, while if I send the same request through Postman, everything's fine.
EDIT: That's what I tried to do to set the URL myself, and if I print it I can see it's not encoded. The error I get is: panic: runtime error: invalid memory address or nil pointer dereference.
client := &http.Client{}
req, _ := http.NewRequest("GET", "", nil)
req.URL = &url.URL{
Host: "api.website.com",
Scheme: "https",
RawQuery: `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`,
}
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
fmt.Printf("%s\n", data)
Specify the protocol and host when creating the request. Set URL.Opaque to the desired request URI:
req, _ := http.NewRequest("GET", "https://api.website.com/", nil)
req.URL.Opaque = `/rest/v1/item/0?search=[{"key":"tag","value":"myvalue"}]`
req.Header.Set("Authorization", "Bearer " + viper.GetString("RAINDROP_TOKEN"))
res, _ := client.Do(req)
defer req.Body.Close()
data, _ := ioutil.ReadAll(res.Body)
Run it on the playground.
I need to make a GET request to an API with a bearer token in the authorization request. How can I do this in Go? I have the following code, but I haven't had success.
package main
import (
"io/ioutil"
"log"
"net/http"
)
func main() {
url := "https://api.globalcode.com.br/v1/publico/eventos"
resp, err := http.Get(url)
resp.Header.Add("Bearer", "token")
if err != nil {
log.Println("Erro ao realizar request.\n[ERRO] -", err)
}
body, _ := ioutil.ReadAll(resp.Body)
log.Println(string([]byte(body)))
}
For control over HTTP client headers, redirect policy, and other settings, create a Client:
package main
import (
"io/ioutil"
"log"
"net/http"
)
func main() {
url := "https://api.globalcode.com.br/v1/publico/eventos"
// Create a Bearer string by appending string access token
var bearer = "Bearer " + <ACCESS TOKEN HERE>
// Create a new request using http
req, err := http.NewRequest("GET", url, nil)
// add authorization header to the req
req.Header.Add("Authorization", bearer)
// Send req using http Client
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Println("Error on response.\n[ERROR] -", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Println("Error while reading the response bytes:", err)
}
log.Println(string([]byte(body)))
}
The Client's Transport typically has internal state (cached TCP
connections), so Clients should be reused instead of created as
needed. Clients are safe for concurrent use by multiple goroutines.
A Client is higher-level than a RoundTripper (such as Transport) and
additionally handles HTTP details such as cookies and redirects.
For more information on Client and Transport check golang spec for net/http package
I had to add a client.CheckRedirect Function(seen below) in order to pass the Bearer token to the API.
bearer := "Bearer " + token
req, err := http.NewRequest("GET", url, bytes.NewBuffer(nil))
req.Header.Set("Authorization", bearer)
req.Header.Add("Accept", "application/json")
client := &http.Client{}
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
for key, val := range via[0].Header {
req.Header[key] = val
}
return err
}
resp, err := client.Do(req)
if err != nil {
log.Println("Error on response.\n[ERRO] -", err)
} else {
defer resp.Body.Close()
data, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(data))
}
}
I made a super-little-basic library for execute basic request like:
package main
import (
request "github.com/alessiosavi/Requests"
)
func main(){
// Create a key-value list of headers
headers := request.CreateHeaderList(`Accept`, `application/json`, "Authorization", "Bearer "+IAMToken)
resp :=request.SendRequest(`http://your_site.com`, `GET`, headers, nil))
}
Here you can find the request implementation:
https://github.com/alessiosavi/Requests/blob/e7ca66bde738b6224fba2b6f146a8dbee67d3323/Requests.go
Here you can find how i use the library for Bearer Auth and other auth type:
https://github.com/alessiosavi/GoCloudant/blob/a8ad3a7990f04ea728bb327d6faea6af3e5455ca/cloudant.go
I have a standard piece of go http request that I am trying to get right in the code below.
func IssueSearchQuery(conf Config) (string, error) {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
var client = &http.Client{
Timeout: time.Second * 60,
Transport: tr,
}
URL := conf.URL + "/api/search"
v := url.Values{}
v.Set("query_expression", "select * from test")
req, err := http.NewRequest("POST", URL,
bytes.NewBufferString(v.Encode()))
req.SetBasicAuth(conf.User, conf.Password)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
bodyText, err := ioutil.ReadAll(resp.Body)
s := string(bodyText)
return s, err
}
In the above code, I am connecting to the server, basic authentication works, but the server responds complaining saying the required query_expression query parameter is missing. If I do curl -u user:pass -d "query_expression=select * from test" --insecure -XPOST 'https://ip:port/api/search' the server responds with the required result.
The code also works if I don't use url.Values{} and instead manually encode my query into the url like URL := conf.URL + "/api/ariel/searches?query_expression=select%20*20from%20test"
I am not able to figure out what I am doing wrong with my query parameters. Can someone help? Thanks!
You appear to be trying to set the POST body to the url-encoded values. Either set your Content-Type header to application/x-www-form-urlencoded, and put them in the body as you're doing (thanks #hobbs), or use your url-encoded values in the actual url rather than the body:
u, err := url.Parse(conf.URL + "/api/search")
v := url.Values{}
v.Set("query_expression", "select * from test")
u.RawQuery = v.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
Typically people expect POST data to be in the body, so just adding the req.Header.Set("Content-Type", "application/x-www-form-urlencoded") is probably the best.
I found, implementation of transport in http2 package doesn't support proxy/socks for http2 client. Is there an simple way to make it work?
Yes I seem to recall something a while ago about the http2 client not working with HTTP/SOCKS5 proxies. Anyway time has moved on and the below works fine (if that's what you're after). Note that if using a custom DialTLS within the Transport, proxied connections still aren't supported.
package main
import (
"log"
"net/http"
"net/url"
)
func main() {
var addressString = "https://www.facebook.com/"
var proxyString = "socks5://127.0.0.1:9150"
req, _ := http.NewRequest("GET", addressString, nil)
tr := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
proxyURI, err := url.Parse(proxyString)
return proxyURI, err
},
}
// The http client is equipped to handle http/2 connections
hc := &http.Client{Transport: tr}
resp, _ := hc.Do(req)
log.Print(resp)
}
If you're looking to communicate over sockets something like this should work:
socket := "<socket-path>"
// server
sock, err := net.Listen("unix", socket)
go http.Serve(s, nil)
//client
httpc := http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", socket)
},
},
}
Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.