I am trying to create an HTTP client that can send self-signed HTTP requests through a proxy server.
I tried this code but I am not sure if there is a problem here, will the following code work?
func CreateProxyClient(serverProxy string, sid string, portProxy int) (*Client, error) {
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
proxyUrl, _ := url.Parse(serverProxy+":"+strconv.Itoa(portProxy))
tr := &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
}
var netClient = &http.Client{
Timeout: time.Second * 10,
Transport: tr,
}
return &Client{netClient, serverProxy, sid}, nil
}
"Is there a problem"? Only if you consider blindly trusting the certificate a problem (that's why it's called InsecureSkipVerify).
The better option is to configure the client to trust the specific certificate that the server is using, so you get MITM protection in addition to encryption.
To do this, get a copy of the server's certificate via a trusted channel (e.g. copy it from the server's filesystem), then add it to the client's CA pool (this will also trust all certificates signed by the server's cert, if applicable).
Here is an example for the test certificate in net/http, which is used by httptest.NewTLSServer:
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"log"
"net/http"
"net/http/httptest"
)
// cert is used by httptest.NewTLSServer.
//
// In a real application you're going to want to load the certificate from
// disk, rather than hard-coding it. Otherwise you have to recompile the program
// when the certificate is updated.
var cert = []byte(`-----BEGIN CERTIFICATE-----
MIICEzCCAXygAwIBAgIQMIMChMLGrR+QvmQvpwAU6zANBgkqhkiG9w0BAQsFADAS
MRAwDgYDVQQKEwdBY21lIENvMCAXDTcwMDEwMTAwMDAwMFoYDzIwODQwMTI5MTYw
MDAwWjASMRAwDgYDVQQKEwdBY21lIENvMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCB
iQKBgQDuLnQAI3mDgey3VBzWnB2L39JUU4txjeVE6myuDqkM/uGlfjb9SjY1bIw4
iA5sBBZzHi3z0h1YV8QPuxEbi4nW91IJm2gsvvZhIrCHS3l6afab4pZBl2+XsDul
rKBxKKtD1rGxlG4LjncdabFn9gvLZad2bSysqz/qTAUStTvqJQIDAQABo2gwZjAO
BgNVHQ8BAf8EBAMCAqQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUw
AwEB/zAuBgNVHREEJzAlggtleGFtcGxlLmNvbYcEfwAAAYcQAAAAAAAAAAAAAAAA
AAAAATANBgkqhkiG9w0BAQsFAAOBgQCEcetwO59EWk7WiJsG4x8SY+UIAA+flUI9
tyC4lNhbcF2Idq9greZwbYCqTTTr2XiRNSMLCOjKyI7ukPoPjo16ocHj+P3vZGfs
h1fIw3cSS2OolhloGw/XM6RWPWtPAlGykKLciQrBru5NAPvCMsb/I1DAceTiotQM
fblo6RBxUQ==
-----END CERTIFICATE-----`)
func main() {
pool, err := x509.SystemCertPool()
if err != nil {
log.Fatal(err)
}
if !pool.AppendCertsFromPEM(cert) {
log.Fatal("Cannot append self-signed cert to CA pool")
}
c := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: pool,
},
},
}
s := httptest.NewTLSServer(nil)
res, err := c.Get(s.URL)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.Status)
}
Try it on the playground: https://play.golang.org/p/HsI2RyOd5qd
Related
I am building a web server that must accept HTTP requests from a client, but must also accept requests over a raw TCP socket from peers. Since HTTP runs over TCP, I am trying to route the HTTP requests by the TCP server rather than running two separate services.
Is there an easy way to read in the data with net.Conn.Read(), determine if it is an HTTP GET/POST request and pass it off to the built in HTTP handler or Gorilla mux? Right now my code looks like this and I am building the http routing logic myself:
func ListenConn() {
listen, _ := net.Listen("tcp", ":8080")
defer listen.Close()
for {
conn, err := listen.Accept()
if err != nil {
logger.Println("listener.go", "ListenConn", err.Error())
}
go HandleConn(conn)
}
}
func HandleConn(conn net.Conn) {
defer conn.Close()
// determines if it is an http request
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
ln := scanner.Bytes()
fmt.Println(ln)
if strings.Fields(string(ln))[0] == "GET" {
http.GetRegistrationCode(conn, strings.Fields(string(ln))[1])
return
}
... raw tcp handler code
}
}
It is not a good idea to mix HTTP and raw TCP traffic.
Think about all firewalls and routers between your application and clients. They all designed to enable safe HTTP(s) delivery. What they will do with your tcp traffic coming to the same port as valid HTTP?
As a solution you can split your traffic to two different ports in the same application.
With ports separation you can route your HTTP and TCP traffic independently and configure appropriate network security for every channel.
Sample code to listen for 2 different ports:
package main
import (
"fmt"
"net"
"net/http"
"os"
)
type httpHandler struct {
}
func (m *httpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
fmt.Println("HTTP request")
}
func main() {
// http
go func() {
http.ListenAndServe(":8001", &httpHandler{})
}()
// tcp
l, err := net.Listen("tcp", "localhost:8002")
if err != nil {
fmt.Println("Error listening:", err.Error())
os.Exit(1)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
go handleRequest(conn)
}
}
// Handles incoming requests.
func handleRequest(conn net.Conn) {
// read/write from connection
fmt.Println("TCP connection")
conn.Close()
}
open http://localhost:8001/ in browser and run command line echo -n "test" | nc localhost 8002 to test listeners
I want to connect to a server through proxy server that I have.
I am searching for something that is similar to Python's HTTPConnection.set_tunnel, is there something like this in golang?
----edit-----
I'm trying to create a connection to a server that allows self signed certificates & transfers through proxy, will this code work properly?
func CreateProxyClient(serverProxy string, sid string, portProxy int) (*Client, error) {
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
proxyUrl, _ := url.Parse(serverProxy+":"+strconv.Itoa(portProxy))
tr := &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
}
var netClient = &http.Client{
Timeout: time.Second * 10,
Transport: tr,
}
return &Client{netClient, serverProxy, sid}, nil
}
You can set environment variable HTTP_PROXY for HTTP or HTTPS_PROXY for HTTPS so the default http transport will use it.
Also as an alternative you can create http.Transport by yourself with Proxy field set to http.ProxyURL function call or use you custom implementation.
Example:
proxyURL, _ := url.Parse("http://proxy.example.com:port")
http.DefaultTransport = &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
// request using proxy
resp, _ := http.Get("https://google.com"))
I found, implementation of transport in http2 package doesn't support proxy/socks for http2 client. Is there an simple way to make it work?
Yes I seem to recall something a while ago about the http2 client not working with HTTP/SOCKS5 proxies. Anyway time has moved on and the below works fine (if that's what you're after). Note that if using a custom DialTLS within the Transport, proxied connections still aren't supported.
package main
import (
"log"
"net/http"
"net/url"
)
func main() {
var addressString = "https://www.facebook.com/"
var proxyString = "socks5://127.0.0.1:9150"
req, _ := http.NewRequest("GET", addressString, nil)
tr := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
proxyURI, err := url.Parse(proxyString)
return proxyURI, err
},
}
// The http client is equipped to handle http/2 connections
hc := &http.Client{Transport: tr}
resp, _ := hc.Do(req)
log.Print(resp)
}
If you're looking to communicate over sockets something like this should work:
socket := "<socket-path>"
// server
sock, err := net.Listen("unix", socket)
go http.Serve(s, nil)
//client
httpc := http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", socket)
},
},
}
How do I disable common name validation inside of a go http client. I am doing mutual TLS with a common CA and hence common name validation means nothing.
The tls docs say,
// ServerName is used to verify the hostname on the returned
// certificates unless InsecureSkipVerify is given. It is also included
// in the client's handshake to support virtual hosting unless it is
// an IP address.
ServerName string
I don't want to do InsecureSkipVerify but I don't want to validate the common name.
You would pass a tls.Config struct with your own VerifyPeerCertificate function, and then you would check the certificate yourself.
VerifyPeerCertificate func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error
If normal verification fails then the handshake will abort before
considering this callback. If normal verification is disabled by
setting InsecureSkipVerify then this callback will be considered but
the verifiedChains argument will always be nil.
You can look here for an example of how to verify a certificate. Iif you look here, you'll see that part of even this verification process includes checking the hostname, but luckily you'll see that it skips it if it's set to the empty string.
So, basically you write your own VerifyPeerCertificate function, convert the rawCerts [][]byte, which I think would look something like:
customVerify := func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
roots := x509.NewCertPool()
for _, rawCert := range rawCerts {
cert, _ := x509.ParseCertificate(rawCert)
roots.AddCert(cert)
}
opts := x509.VerifyOptions{
Roots: roots,
}
_, err := cert.Verify(opts)
return err
}
conf := tls.Config{
//...
VerifyPeerCertificate: customVerify,
}
Normal https post like this
pool := x509.NewCertPool()
caStr, err := ioutil.ReadFile(serverCAFile)
if err != nil {
return nil, fmt.Errorf("read server ca file fail")
}
pool.AppendCertsFromPEM(caStr)
tr := &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: pool,
},
}
client := &http.Client{Transport: tr}
client.Post(url, bodyType, body)
But if your url is use ip(ex. https://127.0.0.1:8080/api/test) or you URL is not match certificate common name, and you want to only ignore certificate common name check, should do like this
pool := x509.NewCertPool()
caStr, err := ioutil.ReadFile(serverCAFile)
if err != nil {
return nil, fmt.Errorf("read server ca file fail")
}
block, _ := pem.Decode(caStr)
if block == nil {
return nil, fmt.Errorf("Decode ca file fail")
}
if block.Type != "CERTIFICATE" || len(block.Headers) != 0 {
return nil, fmt.Errorf("Decode ca block file fail")
}
cert, err := x509.ParseCertificate(block.Bytes)
if err != nil {
return nil, fmt.Errorf("ParseCertificate ca block file fail")
}
pool.AddCert(cert)
tr := &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: pool,
ServerName: cert.Subject.CommonName, //manual set ServerName
},
}
client := &http.Client{Transport: tr}
client.Post(url, bodyType, body)
Playing with golang's net/http package and SPDY. Something is really confusing me:
The *tls.Conn of TLSNextProto function can't be read at all. Any read attempt will get a "connection reset by peer" error.
Run the following program, and then access https://localhost:8080/ using Chrome with SPDY enabled.
Am I using the TLS connection object in a wrong way? Please help.
package main
import (
"crypto/tls"
"log"
"net/http"
)
func main() {
server := &http.Server{
Addr: ":8080",
TLSConfig: &tls.Config{
NextProtos: []string{"spdy/3"},
},
TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){
"spdy/3": func(s *http.Server, conn *tls.Conn, h http.Handler) {
buf := make([]byte, 1)
if n, err := conn.Read(buf); err != nil {
log.Panicf("%v|%v\n", n, err)
}
},
},
}
err := server.ListenAndServeTLS("/path/to/host.cert", "/path/to/host.key")
if err != nil {
log.Fatal(err)
}
}
OK. I got it. It is the certificate issue. If the certificate used by server.ListenAndServeTLS() is not signed by a CA trusted by the browser(Chrome), connection will be reset. For creating you own CA and cert, following http://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/