Playing with golang's net/http package and SPDY. Something is really confusing me:
The *tls.Conn of TLSNextProto function can't be read at all. Any read attempt will get a "connection reset by peer" error.
Run the following program, and then access https://localhost:8080/ using Chrome with SPDY enabled.
Am I using the TLS connection object in a wrong way? Please help.
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
server := &http.Server{
Addr: ":8080",
TLSConfig: &tls.Config{
NextProtos: []string{"spdy/3"},
},
TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){
"spdy/3": func(s *http.Server, conn *tls.Conn, h http.Handler) {
buf := make([]byte, 1)
if n, err := conn.Read(buf); err != nil {
log.Panicf("%v|%v\n", n, err)
}
},
},
}
err := server.ListenAndServeTLS("/path/to/host.cert", "/path/to/host.key")
if err != nil {
log.Fatal(err)
}
}
OK. I got it. It is the certificate issue. If the certificate used by server.ListenAndServeTLS() is not signed by a CA trusted by the browser(Chrome), connection will be reset. For creating you own CA and cert, following http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/
Related
I have a dockerfile which runs tor -
FROM alpine:edge
RUN apk update && apk add tor
EXPOSE 9050
USER tor
CMD ["/usr/bin/tor"]
and ran it using command - docker run --name tor -p 11000:9050 tor
and checked connection using - telnet 127.0.0.1 11000 and it showed connected
Now I want to use tor as proxy while any request from go program. I tried -
package main
import (
"fmt"
"net/http"
"net/url"
"time"
)
func main() {
proxyUrl, err := url.Parse("socks5://127.0.0.1:11000")
if err != nil {
// TODO handle me
panic(err)
}
cl := http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
},
Timeout: 18000 * time.Millisecond,
}
resp, err := cl.Get("http://google.com")
if err != nil {
// TODO handle me
panic(err)
}
// TODO work with the response
fmt.Println(resp)
}
But running this program threw error -
panic: Get http://google.com: socks connect tcp 127.0.0.1:11000->google.com:80: read tcp 127.0.0.1:59630->127.0.0.1:11000: read: connection reset by peer
goroutine 1 [running]: <stacktrace>
exit status 2
I tried other approaches also, notably mentioned here and here but kept getting same error - read: connection reset by peer
Please help which part is incorrect here.
Thanks.
--------------------another approach that I tried ----------------
As mentioned in one of the links, I tried this code also -
const (
PROXY_ADDR = "127.0.0.1:11000"
URL = "http://facebookcorewwwi.onion"
)
func main() {
// create a socks5 dialer
dialer, err := proxy.SOCKS5("tcp", PROXY_ADDR, nil, proxy.Direct)
if err != nil {
fmt.Fprintln(os.Stderr, "can't connect to the proxy:", err)
os.Exit(1)
}
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
// do anything with ctx
return dialer.Dial(network, address)
}
// setup a http client
httpTransport := &http.Transport{
DialContext: dialContext,
}
httpClient := &http.Client{Transport: httpTransport}
// create a request
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
fmt.Fprintln(os.Stderr, "can't create request:", err)
os.Exit(2)
}
resp, err := httpClient.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot make get request: ", err)
os.Exit(2)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot read response body: ", err)
os.Exit(2)
}
fmt.Println("received response -> ", body)
}
but received error -
cannot make get request: Get http://facebookcorewwwi.onion: socks connect tcp 127.0.0.1:11000->facebookcorewwwi.onion:80: read tcp 127.0.0.1:59826->127.0.0.1:11000: read: connection reset by peer
exit status 2
Any help is appreciable.
After making sure tor is working properly on port 9050.
Try the following curl command to ensure tor is working properly.
curl --socks5 localhost:9050 --socks5-hostname localhost:9050 -s https://wtfismyip.com/json
Can you try this
package main
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/http"
"golang.org/x/net/proxy"
)
func main() {
proxyUrl := "127.0.0.1:9050"
dialer, err := proxy.SOCKS5("tcp", proxyUrl, nil, proxy.Direct)
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
return dialer.Dial(network, address)
}
transport := &http.Transport{DialContext: dialContext,
DisableKeepAlives: true}
cl := &http.Client{Transport: transport}
resp, err := cl.Get("https://wtfismyip.com/json")
if err != nil {
// TODO handle me
panic(err)
}
body, err := ioutil.ReadAll(resp.Body)
// TODO work with the response
if err != nil {
fmt.Println("body read failed")
}
fmt.Println(string(body))
}
As suggested in the answer/comment above, main problem was socks5 connection to dockerized tor container.
Found the solution here, i just had to modify dockerfile like -
FROM alpine:edge
RUN apk update && apk add tor
RUN cp /etc/tor/torrc.sample /etc/tor/torrc && echo "SocksPort 0.0.0.0:9050" > /etc/tor/torrc
EXPOSE 9050
CMD ["/usr/bin/tor"]
Other code pieces are working fine.
I am trying to connect to a Host using Random TLS Fingerprinting. I am using https://github.com/refraction-networking/utls (see my issue i created on https://github.com/refraction-networking/utls/issues/42)
My issue is now, how can i utilize a HTTP or SOCKS5 Proxy while opening that connection?
The Code im using right now is:
package main
import (
"bufio"
"fmt"
"log"
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/refraction-networking/utls"
)
var (
dialTimeout = time.Duration(15) * time.Second
)
var requestHostname = "google.com"
var requestAddr = "172.217.22.110:443"
// this example generates a randomized fingeprint, then re-uses it in a follow-up connection
func HttpGetConsistentRandomized(hostname string, addr , uri string) (*http.Response, error) {
config := tls.Config{ServerName: hostname}
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn := tls.UClient(tcpConn, &config, tls.HelloRandomized)
defer uTlsConn.Close()
err = uTlsConn.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
uTlsConn.Close()
// At this point uTlsConn.ClientHelloID holds a seed that was used to generate
// randomized fingerprint. Now we can establish second connection with same fp
tcpConn2, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn2 := tls.UClient(tcpConn2, &config, uTlsConn.ClientHelloID)
defer uTlsConn2.Close()
err = uTlsConn2.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
return httpGetOverConn(uTlsConn2, uTlsConn2.HandshakeState.ServerHello.AlpnProtocol, uri)
}
func main() {
var response *http.Response
var err error
response, err = HttpGetConsistentRandomized(requestHostname, requestAddr, "/2.0/ssocookie")
if err != nil {
fmt.Printf("#> HttpGetConsistentRandomized() failed: %+v\n", err)
} else {
//fmt.Printf("#> HttpGetConsistentRandomized() response: %+s\n", httputil.DumpResponse(response,true))
dump, err := httputil.DumpResponse(response, true)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+s\n", dump)
}
return
}
func httpGetOverConn(conn net.Conn, alpn string, uri string) (*http.Response, error) {
req := &http.Request{
Method: "GET",
URL: &url.URL{Host: "www." + requestHostname + uri},
Header: make(http.Header),
Host: "www." + requestHostname,
}
req.Proto = "HTTP/1.1"
req.ProtoMajor = 1
req.ProtoMinor = 1
err := req.Write(conn)
if err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(conn), req)
}
As Steffen said, you have to create a proxy dialer first, dial the proxy to create a net.Conn, then use that net.Conn when creating the uTLS Client, before handshaking. For brevity's sake, your custom dialTLS function would look something like:
import (
"crypto/tls"
"net"
"net/url"
"github.com/magisterquis/connectproxy"
"golang.org/x/net/proxy"
utls "github.com/refraction-networking/utls"
)
var proxyString = "http://127.0.0.1:8080"
dialTLS := func(network, addr string, _ *tls.Config) (net.Conn, error) {
proxyURI, _ := url.Parse(proxyString)
switch proxyURI.Scheme {
case "socks5":
proxyDialer, err = proxy.SOCKS5("tcp", proxyString, nil, proxy.Direct)
case "http":
proxyDialer, err = connectproxy.New(proxyURI, proxy.Direct)
}
conn, err := proxyDialer.Dial("tcp", addr)
uconn := utls.UClient(conn, cfg, &utls.HelloRandomizedALPN)
...
}
Two suggestions:
Use the "connectproxy" module referenced above if you intend to tunnel through a HTTP CONNECT proxy.
Make life easier for yourself and take a look at the Meek pluggable transport source for Tor. There's a 'utls.go' module which takes care of everything for you, including setting up either a http or http2 transport depending on the negotiated ALPN protocol. It only supports SOCKS but you could easily adapt it to handle HTTP proxies.
A HTTP proxy and SOCKS proxy work be having some initial proxy specific handshake after the TCP connect. After this handshake is done they provide a normal TCP socket which then can be used for doing the TLS handshake etc. Thus, all you need is to replace your
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
with a proxy specific method to setup the TCP connection. This can be done by using SOCKS5 in x/net/proxy to create the appropriate Dialer or similar using the HTTP CONNECT method is done in connectproxy.
I found, implementation of transport in http2 package doesn't support proxy/socks for http2 client. Is there an simple way to make it work?
Yes I seem to recall something a while ago about the http2 client not working with HTTP/SOCKS5 proxies. Anyway time has moved on and the below works fine (if that's what you're after). Note that if using a custom DialTLS within the Transport, proxied connections still aren't supported.
package main
import (
"log"
"net/http"
"net/url"
)
func main() {
var addressString = "https://www.facebook.com/"
var proxyString = "socks5://127.0.0.1:9150"
req, _ := http.NewRequest("GET", addressString, nil)
tr := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
proxyURI, err := url.Parse(proxyString)
return proxyURI, err
},
}
// The http client is equipped to handle http/2 connections
hc := &http.Client{Transport: tr}
resp, _ := hc.Do(req)
log.Print(resp)
}
If you're looking to communicate over sockets something like this should work:
socket := "<socket-path>"
// server
sock, err := net.Listen("unix", socket)
go http.Serve(s, nil)
//client
httpc := http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", socket)
},
},
}
I am playing with the net package, and i want to make a simple proxy.
First i make a listener on localhost, then i dial the remote address
remote, err := net.Dial("tcp", "google.com:80")
if err != nil {
log.Fatal(err)
}
defer remote.Close()
fmt.Fprint(remote, "GET / HTTP/1.0\r\n\r\n")
How can i pipe the response to the browser? Or do i need to work with the default webserver and copy the response body? I really want to try it with net package or something
thx
To copy the connection from the remote is used 2 goroutines with io.Copy
func copyContent(from, to net.Conn, done chan bool) {
_, err := io.Copy(from, to)
if err != nil {
done <- true
}
done <- true
}
// in the main func
done := make(chan bool, 2)
go copyContent(conn, remote, done)
go copyContent(remote, conn, done)
<-done
<-done
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/