I found, implementation of transport in http2 package doesn't support proxy/socks for http2 client. Is there an simple way to make it work?
Yes I seem to recall something a while ago about the http2 client not working with HTTP/SOCKS5 proxies. Anyway time has moved on and the below works fine (if that's what you're after). Note that if using a custom DialTLS within the Transport, proxied connections still aren't supported.
package main
import (
"log"
"net/http"
"net/url"
)
func main() {
var addressString = "https://www.facebook.com/"
var proxyString = "socks5://127.0.0.1:9150"
req, _ := http.NewRequest("GET", addressString, nil)
tr := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
proxyURI, err := url.Parse(proxyString)
return proxyURI, err
},
}
// The http client is equipped to handle http/2 connections
hc := &http.Client{Transport: tr}
resp, _ := hc.Do(req)
log.Print(resp)
}
If you're looking to communicate over sockets something like this should work:
socket := "<socket-path>"
// server
sock, err := net.Listen("unix", socket)
go http.Serve(s, nil)
//client
httpc := http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
return net.Dial("unix", socket)
},
},
}
Related
I am developing an application in Go which uses the net/http package to make a number of external http requests/calls which I would like to monitor the http calls' latency, response times, status code etc.
As there are a number of endpoints which my applications calls, and there could be more that will be added in the future, I would like to consider a solution which will obtain the metrics for all such external http calls.
Any suggestions, guidance or examples on this is much appreciated.
I am thinking of some way in which a middleware/wrapper is added onto the Do/Raw call within the net/http package which I can create and modify a copy of but I'm not exactly sure how to do that.
Here's a very (!) basic example that abstracts http.Client's Do method with a Prometheus Counter that Inc's on every Do and labels the Counter with the host's name, the request method and the response code:
package main
import (
"log"
"net/http"
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
type Client struct {
Client *http.Client
Counter *prometheus.CounterVec
}
func NewClient(counter *prometheus.CounterVec) *Client {
return &Client{
Client: &http.Client{},
Counter: counter,
}
}
func (c *Client) Do(rqst *http.Request) (*http.Response, error) {
resp, err := c.Client.Do(rqst)
host := rqst.URL.Host
code := strconv.Itoa(resp.StatusCode)
c.Counter.With(prometheus.Labels{
"host": host,
"response_code": code,
}).Inc()
return resp, err
}
func main() {
metric := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "foo",
Help: "Number of HTTP requests.",
},
[]string{
"host",
"response_code",
},
)
prometheus.MustRegister(metric)
c := NewClient(metric)
go func() {
for {
method := http.MethodPost
url := "https://httpbin.org/post"
rqst, _ := http.NewRequest(method, url, nil)
_, _ = c.Do(rqst)
time.Sleep(15 * time.Second)
}
}()
log.Println("Starting server")
http.Handle("/metrics", promhttp.Handler())
log.Fatal(http.ListenAndServe(":8080", nil))
}
And:
# HELP foo Number of HTTP requests.
# TYPE foo counter
foo{host="httpbin.org",method="POST",response_code="200"} 6```
I'm trying to implement a http forwarding server that supports domain blocking. I've tried
go io.Copy(dst, src)
go io.Copy(src, dst)
and it works like a charm on tcp forwarding. Then I've tried to do request line parsing with something similar to
go func(){
reader := io.TeeReader(src, dst)
textReader := textproto.NewReader(bufio.NewReader(reader))
requestLine, _ = textReader.ReadLine()
// ...
ioutil.ReadAll(reader)
}
It works fine, but I was getting worried about bad performance(with ioutil.ReadAll). So I've written the code below.
func (f *Forwarder) handle(src, dst net.Conn) {
defer dst.Close()
defer src.Close()
done := make(chan struct{})
go func() {
textReader := bufio.NewReader(src)
requestLine, _ = textReader.ReadString('\n')
// parse request line and apply domain blocking
dst.Write([]byte(requestLine))
io.Copy(dst, src)
done <- struct{}{}
}()
go func() {
textReader := bufio.NewReader(dst)
s.statusLine, _ = textReader.ReadString('\n')
src.Write([]byte(s.statusLine))
io.Copy(src, dst)
done <- struct{}{}
}()
<-done
<-done
}
Unfortunately, it doesn't work at all. Requests get to print out, but not for responses. I've stuck here and don't know what's wrong.
TCP forwarding is to realize that the tunnel proxy does not need to parse data. The reverse proxy can use the standard library.
The tunnel proxy is implemented to separate the http and https protocols. The client generally uses the tunnel to send https and sends the Connect method. Sending http is the Get method. For the https request service, only dail creates the connection tcp conversion, and the http request is implemented using a reverse proxy.
func(w http.ResponseWriter, r *http.Request) {
// check url host
if r.URL.Host != "" {
if r.Method == eudore.MethodConnect {
// tunnel proxy
conn, err := net.Dial("tcp", r.URL.Host)
if err != nil {
w.WriteHeader(502)
return
}
client, _, err := w.Hijack()
if err != nil {
w.WriteHeader(502)
conn.Close()
return
}
client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))
go func() {
io.Copy(client, conn)
client.Close()
conn.Close()
}()
go func() {
io.Copy(conn, client)
client.Close()
conn.Close()
}()
} else {
// reverse proxy
httputil.NewSingleHostReverseProxy(r.URL).ServeHTTP(w, r)
}
}
}
Implementing a reverse proxy will parse the client request, and the proxy will send the request to the target server.
Reverse proxy conversion request, not tested :
func(w http.ResponseWriter, r *http.Request) {
// set host
r.URL.Scheme = "http"
r.URL.Path = "example.com"
// send
resp,err := http.DefaultClient.Do(r)
if err != nil {
w.WriteHeader(502)
return
}
// write respsonse
defer resp.Body.Close()
w.WriteHeader(resp.StatusCode)
h := w.Header()
for k,v := range resp.Header {
h[k]=v
}
io.Copy(w, resp.Body)
}
However, the direct forwarding request does not process the hop-to-hop header. The hop-to-hop header is clearly stated in the rfc. The hop-to-hop header is the transmission information between two connections. For example, the client to the proxy and the proxy to the server are two. And the client to the server is end-to-end.
Please use the standard library directly for the reverse proxy, it has already handled the hop-to-hop header and Upgrade for you.
exmample NewSingleHostReverseProxy with filter:
package main
import (
"net/http"
"strings"
"net/http/httputil"
"net/url"
)
func main() {
addr, _ := url.Parse("http://localhost:8088")
proxy := httputil.NewSingleHostReverseProxy(addr)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/") {
proxy.ServeHTTP(w, r)
} else {
w.WriteHeader(404)
}
})
// Listen Server
}
I am trying to connect to a Host using Random TLS Fingerprinting. I am using https://github.com/refraction-networking/utls (see my issue i created on https://github.com/refraction-networking/utls/issues/42)
My issue is now, how can i utilize a HTTP or SOCKS5 Proxy while opening that connection?
The Code im using right now is:
package main
import (
"bufio"
"fmt"
"log"
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/refraction-networking/utls"
)
var (
dialTimeout = time.Duration(15) * time.Second
)
var requestHostname = "google.com"
var requestAddr = "172.217.22.110:443"
// this example generates a randomized fingeprint, then re-uses it in a follow-up connection
func HttpGetConsistentRandomized(hostname string, addr , uri string) (*http.Response, error) {
config := tls.Config{ServerName: hostname}
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn := tls.UClient(tcpConn, &config, tls.HelloRandomized)
defer uTlsConn.Close()
err = uTlsConn.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
uTlsConn.Close()
// At this point uTlsConn.ClientHelloID holds a seed that was used to generate
// randomized fingerprint. Now we can establish second connection with same fp
tcpConn2, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn2 := tls.UClient(tcpConn2, &config, uTlsConn.ClientHelloID)
defer uTlsConn2.Close()
err = uTlsConn2.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
return httpGetOverConn(uTlsConn2, uTlsConn2.HandshakeState.ServerHello.AlpnProtocol, uri)
}
func main() {
var response *http.Response
var err error
response, err = HttpGetConsistentRandomized(requestHostname, requestAddr, "/2.0/ssocookie")
if err != nil {
fmt.Printf("#> HttpGetConsistentRandomized() failed: %+v\n", err)
} else {
//fmt.Printf("#> HttpGetConsistentRandomized() response: %+s\n", httputil.DumpResponse(response,true))
dump, err := httputil.DumpResponse(response, true)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+s\n", dump)
}
return
}
func httpGetOverConn(conn net.Conn, alpn string, uri string) (*http.Response, error) {
req := &http.Request{
Method: "GET",
URL: &url.URL{Host: "www." + requestHostname + uri},
Header: make(http.Header),
Host: "www." + requestHostname,
}
req.Proto = "HTTP/1.1"
req.ProtoMajor = 1
req.ProtoMinor = 1
err := req.Write(conn)
if err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(conn), req)
}
As Steffen said, you have to create a proxy dialer first, dial the proxy to create a net.Conn, then use that net.Conn when creating the uTLS Client, before handshaking. For brevity's sake, your custom dialTLS function would look something like:
import (
"crypto/tls"
"net"
"net/url"
"github.com/magisterquis/connectproxy"
"golang.org/x/net/proxy"
utls "github.com/refraction-networking/utls"
)
var proxyString = "http://127.0.0.1:8080"
dialTLS := func(network, addr string, _ *tls.Config) (net.Conn, error) {
proxyURI, _ := url.Parse(proxyString)
switch proxyURI.Scheme {
case "socks5":
proxyDialer, err = proxy.SOCKS5("tcp", proxyString, nil, proxy.Direct)
case "http":
proxyDialer, err = connectproxy.New(proxyURI, proxy.Direct)
}
conn, err := proxyDialer.Dial("tcp", addr)
uconn := utls.UClient(conn, cfg, &utls.HelloRandomizedALPN)
...
}
Two suggestions:
Use the "connectproxy" module referenced above if you intend to tunnel through a HTTP CONNECT proxy.
Make life easier for yourself and take a look at the Meek pluggable transport source for Tor. There's a 'utls.go' module which takes care of everything for you, including setting up either a http or http2 transport depending on the negotiated ALPN protocol. It only supports SOCKS but you could easily adapt it to handle HTTP proxies.
A HTTP proxy and SOCKS proxy work be having some initial proxy specific handshake after the TCP connect. After this handshake is done they provide a normal TCP socket which then can be used for doing the TLS handshake etc. Thus, all you need is to replace your
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
with a proxy specific method to setup the TCP connection. This can be done by using SOCKS5 in x/net/proxy to create the appropriate Dialer or similar using the HTTP CONNECT method is done in connectproxy.
Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.
Why can't I run both HTTP and HTTPS from the same golang program?
Here is the code where the two servers are initiated.. The server which is initiated first will run - the second won't.. If they are switched arround the other will run and the other won't..
No errors are returned when running the program, but the requests http://www.localhost or https://secure.localhost times out
// Start HTTP
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
Here is the complete code
package main
import (
"net/http"
"fmt"
"log"
"os"
"io"
"runtime"
// go get github.com/gorilla/mux
"github.com/gorilla/mux"
)
const (
HOST = "localhost"
)
func Handler_404(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "Oops, something went wrong!")
}
func Handler_www(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "Hello world :)")
}
func Handler_api(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "This is the API")
}
func Handler_secure(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "This is Secure")
}
func redirect(r *mux.Router, from string, to string){
r.Host(from).Subrouter().HandleFunc("/", func (w http.ResponseWriter, r *http.Request){
http.Redirect(w, r, to, 301)
})
}
func main(){
port := 9000
ssl_port := 443
runtime.GOMAXPROCS(runtime.NumCPU())
http_r := mux.NewRouter()
https_r := mux.NewRouter()
// HTTP 404
http_r.NotFoundHandler = http.HandlerFunc(Handler_404)
// Redirect "http://HOST" => "http://www.HOST"
redirect(http_r, HOST, fmt.Sprintf("http://www.%s:%d", HOST, port))
// Redirect "http://secure.HOST" => "https://secure.HOST"
redirect(http_r, "secure."+HOST, fmt.Sprintf("https://secure.%s", HOST))
www := http_r.Host("www."+HOST).Subrouter()
www.HandleFunc("/", Handler_www)
api := http_r.Host("api."+HOST).Subrouter()
api.HandleFunc("/", Handler_api)
secure := https_r.Host("secure."+HOST).Subrouter()
secure.HandleFunc("/", Handler_secure)
// Start HTTP
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
}
ListenAndServe and ListenAndServeTLS open the listening socket and then loop forever serving client connections. These functions only return on an error.
The main goroutine never gets to the starting the TLS server because the main goroutine is busy waiting for HTTP connections in ListenAndServe.
To fix the problem, start the HTTP server in a new goroutine:
// Start HTTP
go func() {
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
}()
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
As previously said, both ListenAndServe and ListenAndServeTLS are blocking. That being said, I would agree that examples above are in fact resolving your issue as the point is to be in goroutine BUT same examples are not quite following go idioms.
You should be using error channels here as you want to capture ALL errors that are sent to you instead of having just one error returned back. Here's fully working sample that starts HTTP as HTTPS servers and return errors as channel that's later on used just to display errors.
package main
import (
"log"
"net/http"
)
func Run(addr string, sslAddr string, ssl map[string]string) chan error {
errs := make(chan error)
// Starting HTTP server
go func() {
log.Printf("Staring HTTP service on %s ...", addr)
if err := http.ListenAndServe(addr, nil); err != nil {
errs <- err
}
}()
// Starting HTTPS server
go func() {
log.Printf("Staring HTTPS service on %s ...", addr)
if err := http.ListenAndServeTLS(sslAddr, ssl["cert"], ssl["key"], nil); err != nil {
errs <- err
}
}()
return errs
}
func sampleHandler(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.Write([]byte("This is an example server.\n"))
}
func main() {
http.HandleFunc("/", sampleHandler)
errs := Run(":8080", ":10443", map[string]string{
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem",
})
// This will run forever until channel receives error
select {
case err := <-errs:
log.Printf("Could not start serving service due to (error: %s)", err)
}
}
Hope this helps! :)
func serveHTTP(mux *http.ServeMux, errs chan<- error) {
errs <- http.ListenAndServe(":80", mux)
}
func serveHTTPS(mux *http.ServeMux, errs chan<- error) {
errs <- http.ListenAndServeTLS(":443", "fullchain.pem", "privkey.pem", mux)
}
func main() {
mux := http.NewServeMux()
// setup routes for mux // define your endpoints
errs := make(chan error, 1) // a channel for errors
go serveHTTP(mux, errs) // start the http server in a thread
go serveHTTPS(mux, errs) // start the https server in a thread
log.Fatal(<-errs) // block until one of the servers writes an error
}
The ListenAndServe (and ListenAndServeTLS) functions do not return to their caller (unless an error is encountered). You can test this by trying to print something in between the two calls.