I got http: server gave HTTP response to HTTPS client in Golang httptest, when testing a get request of https url using httptest server below.
It works fine when I use URL start with "http://"
func testingHTTPClient(handler http.Handler) (*http.Client, func()) {
s := httptest.NewServer(handler)
cli := &http.Client{
Transport: &http.Transport{
DialContext: func(_ context.Context, network, _ string) (net.Conn, error) {
return net.Dial(network, s.Listener.Addr().String())
},
},
}
return cli, s.Close
}
refer from
Code snippet
How to stub requests to remote hosts with Go
Turns out that instead of
s := httptest.NewServer(handler),
I should use
s := httptest.NewTLSServer(handler)
to get a https server.
Related
I'm trying to implement a http forwarding server that supports domain blocking. I've tried
go io.Copy(dst, src)
go io.Copy(src, dst)
and it works like a charm on tcp forwarding. Then I've tried to do request line parsing with something similar to
go func(){
reader := io.TeeReader(src, dst)
textReader := textproto.NewReader(bufio.NewReader(reader))
requestLine, _ = textReader.ReadLine()
// ...
ioutil.ReadAll(reader)
}
It works fine, but I was getting worried about bad performance(with ioutil.ReadAll). So I've written the code below.
func (f *Forwarder) handle(src, dst net.Conn) {
defer dst.Close()
defer src.Close()
done := make(chan struct{})
go func() {
textReader := bufio.NewReader(src)
requestLine, _ = textReader.ReadString('\n')
// parse request line and apply domain blocking
dst.Write([]byte(requestLine))
io.Copy(dst, src)
done <- struct{}{}
}()
go func() {
textReader := bufio.NewReader(dst)
s.statusLine, _ = textReader.ReadString('\n')
src.Write([]byte(s.statusLine))
io.Copy(src, dst)
done <- struct{}{}
}()
<-done
<-done
}
Unfortunately, it doesn't work at all. Requests get to print out, but not for responses. I've stuck here and don't know what's wrong.
TCP forwarding is to realize that the tunnel proxy does not need to parse data. The reverse proxy can use the standard library.
The tunnel proxy is implemented to separate the http and https protocols. The client generally uses the tunnel to send https and sends the Connect method. Sending http is the Get method. For the https request service, only dail creates the connection tcp conversion, and the http request is implemented using a reverse proxy.
func(w http.ResponseWriter, r *http.Request) {
// check url host
if r.URL.Host != "" {
if r.Method == eudore.MethodConnect {
// tunnel proxy
conn, err := net.Dial("tcp", r.URL.Host)
if err != nil {
w.WriteHeader(502)
return
}
client, _, err := w.Hijack()
if err != nil {
w.WriteHeader(502)
conn.Close()
return
}
client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))
go func() {
io.Copy(client, conn)
client.Close()
conn.Close()
}()
go func() {
io.Copy(conn, client)
client.Close()
conn.Close()
}()
} else {
// reverse proxy
httputil.NewSingleHostReverseProxy(r.URL).ServeHTTP(w, r)
}
}
}
Implementing a reverse proxy will parse the client request, and the proxy will send the request to the target server.
Reverse proxy conversion request, not tested :
func(w http.ResponseWriter, r *http.Request) {
// set host
r.URL.Scheme = "http"
r.URL.Path = "example.com"
// send
resp,err := http.DefaultClient.Do(r)
if err != nil {
w.WriteHeader(502)
return
}
// write respsonse
defer resp.Body.Close()
w.WriteHeader(resp.StatusCode)
h := w.Header()
for k,v := range resp.Header {
h[k]=v
}
io.Copy(w, resp.Body)
}
However, the direct forwarding request does not process the hop-to-hop header. The hop-to-hop header is clearly stated in the rfc. The hop-to-hop header is the transmission information between two connections. For example, the client to the proxy and the proxy to the server are two. And the client to the server is end-to-end.
Please use the standard library directly for the reverse proxy, it has already handled the hop-to-hop header and Upgrade for you.
exmample NewSingleHostReverseProxy with filter:
package main
import (
"net/http"
"strings"
"net/http/httputil"
"net/url"
)
func main() {
addr, _ := url.Parse("http://localhost:8088")
proxy := httputil.NewSingleHostReverseProxy(addr)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/") {
proxy.ServeHTTP(w, r)
} else {
w.WriteHeader(404)
}
})
// Listen Server
}
When using rs.cors to enable Cors on a go web service, returning http status codes other than 200 (for instance, returning http.StatusTooManyRequests) results in a CORS error:
Access to XMLHttpRequest at 'http://localsrv:3021/test?status=toomany' from origin 'http://localhost:8080' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Using rs.cors is not a requirement, but I was unable to successfully get CORS working in some cases without it.
Here is some sample code:
package main
import (
"log"
"net/http"
"github.com/gorilla/mux"
"github.com/rs/cors"
)
func main() {
rtr := mux.NewRouter()
rtr.HandleFunc("/test", test).Methods("GET")
handler := cors.Default().Handler(rtr)
if err := http.ListenAndServe(":3021",
(limitMiddleware(handler))); err != nil {
log.Fatalf("unable to start server: %s", err.Error())
}
}
func limitMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
params := r.URL.Query()
if params["status"][0] == "toomany" {
http.Error(w, http.StatusText(http.StatusTooManyRequests), http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}
func test(w http.ResponseWriter, req *http.Request) {
log.Println("test()")
}
JavaScript request that should generate a StatusTooManyRequests ends up with a CORS error instead of the HTTP 429.
axios.get("http://okami:3021/test?status=toomany")
.then(res => {
console.log("stuff http://okami:3021/test result", res);
}); //axios
Is there a way to use rs.cors so that CORS kicks in when there is a CORS issue, and http error codes can return when there is not a CORS issue?
Basically, unless the system returns HTTP 200, CORS kicks in.
This should be simple, but I'm scratching my head on this one.
Let me know if it would be better to abandon rs.cors and work on manually enabling cors instead and I can start another question for that!
The function returned by limitMiddleware executes before the cors middleware (the next argument). Reorder your handlers:
rtr := mux.NewRouter()
rtr.HandleFunc("/test", test).Methods("GET")
handler := cors.Default().Handler(limitMiddleware(rtr))
http.ListenAndServe(":3021", handler)
You need to specify Cors, Please try this code inside your main(){ ... }
port := ":9000"
headers := handlers.AllowedHeaders([]string{"X-Requested-With", "Content-Type", "Authorization"})
methods := handlers.AllowedMethods([]string{"GET", "POST", "PUT", "DELETE"})
origins := handlers.AllowedOrigins([]string{"*"})
err = http.ListenAndServe(port, handlers.CORS(headers, methods, origins))
if err != nil {
log.Errorf("Error starting server: %s\n", err)
os.Exit(1)
}
Is there a way to use rs.cors so that CORS kicks in when there is a CORS issue, and http error codes can return when there is not a CORS issue?
Unless you handle cors, you will get that as error, so the fix of this issue is to fix cors.
Using rs.cors is not a requirement, but I was unable to successfully get CORS working in some cases without it.
you can handle cors by setting http.ResponseWriter Header value. you can do it so
func getGoogle(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
}
or replace "*" with your front end url.
I am building a web server that must accept HTTP requests from a client, but must also accept requests over a raw TCP socket from peers. Since HTTP runs over TCP, I am trying to route the HTTP requests by the TCP server rather than running two separate services.
Is there an easy way to read in the data with net.Conn.Read(), determine if it is an HTTP GET/POST request and pass it off to the built in HTTP handler or Gorilla mux? Right now my code looks like this and I am building the http routing logic myself:
func ListenConn() {
listen, _ := net.Listen("tcp", ":8080")
defer listen.Close()
for {
conn, err := listen.Accept()
if err != nil {
logger.Println("listener.go", "ListenConn", err.Error())
}
go HandleConn(conn)
}
}
func HandleConn(conn net.Conn) {
defer conn.Close()
// determines if it is an http request
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
ln := scanner.Bytes()
fmt.Println(ln)
if strings.Fields(string(ln))[0] == "GET" {
http.GetRegistrationCode(conn, strings.Fields(string(ln))[1])
return
}
... raw tcp handler code
}
}
It is not a good idea to mix HTTP and raw TCP traffic.
Think about all firewalls and routers between your application and clients. They all designed to enable safe HTTP(s) delivery. What they will do with your tcp traffic coming to the same port as valid HTTP?
As a solution you can split your traffic to two different ports in the same application.
With ports separation you can route your HTTP and TCP traffic independently and configure appropriate network security for every channel.
Sample code to listen for 2 different ports:
package main
import (
"fmt"
"net"
"net/http"
"os"
)
type httpHandler struct {
}
func (m *httpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
fmt.Println("HTTP request")
}
func main() {
// http
go func() {
http.ListenAndServe(":8001", &httpHandler{})
}()
// tcp
l, err := net.Listen("tcp", "localhost:8002")
if err != nil {
fmt.Println("Error listening:", err.Error())
os.Exit(1)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
go handleRequest(conn)
}
}
// Handles incoming requests.
func handleRequest(conn net.Conn) {
// read/write from connection
fmt.Println("TCP connection")
conn.Close()
}
open http://localhost:8001/ in browser and run command line echo -n "test" | nc localhost 8002 to test listeners
I am trying to create an HTTP client that can send self-signed HTTP requests through a proxy server.
I tried this code but I am not sure if there is a problem here, will the following code work?
func CreateProxyClient(serverProxy string, sid string, portProxy int) (*Client, error) {
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
proxyUrl, _ := url.Parse(serverProxy+":"+strconv.Itoa(portProxy))
tr := &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
}
var netClient = &http.Client{
Timeout: time.Second * 10,
Transport: tr,
}
return &Client{netClient, serverProxy, sid}, nil
}
"Is there a problem"? Only if you consider blindly trusting the certificate a problem (that's why it's called InsecureSkipVerify).
The better option is to configure the client to trust the specific certificate that the server is using, so you get MITM protection in addition to encryption.
To do this, get a copy of the server's certificate via a trusted channel (e.g. copy it from the server's filesystem), then add it to the client's CA pool (this will also trust all certificates signed by the server's cert, if applicable).
Here is an example for the test certificate in net/http, which is used by httptest.NewTLSServer:
package main
import (
"crypto/tls"
"crypto/x509"
"fmt"
"log"
"net/http"
"net/http/httptest"
)
// cert is used by httptest.NewTLSServer.
//
// In a real application you're going to want to load the certificate from
// disk, rather than hard-coding it. Otherwise you have to recompile the program
// when the certificate is updated.
var cert = []byte(`-----BEGIN CERTIFICATE-----
MIICEzCCAXygAwIBAgIQMIMChMLGrR+QvmQvpwAU6zANBgkqhkiG9w0BAQsFADAS
MRAwDgYDVQQKEwdBY21lIENvMCAXDTcwMDEwMTAwMDAwMFoYDzIwODQwMTI5MTYw
MDAwWjASMRAwDgYDVQQKEwdBY21lIENvMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCB
iQKBgQDuLnQAI3mDgey3VBzWnB2L39JUU4txjeVE6myuDqkM/uGlfjb9SjY1bIw4
iA5sBBZzHi3z0h1YV8QPuxEbi4nW91IJm2gsvvZhIrCHS3l6afab4pZBl2+XsDul
rKBxKKtD1rGxlG4LjncdabFn9gvLZad2bSysqz/qTAUStTvqJQIDAQABo2gwZjAO
BgNVHQ8BAf8EBAMCAqQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUw
AwEB/zAuBgNVHREEJzAlggtleGFtcGxlLmNvbYcEfwAAAYcQAAAAAAAAAAAAAAAA
AAAAATANBgkqhkiG9w0BAQsFAAOBgQCEcetwO59EWk7WiJsG4x8SY+UIAA+flUI9
tyC4lNhbcF2Idq9greZwbYCqTTTr2XiRNSMLCOjKyI7ukPoPjo16ocHj+P3vZGfs
h1fIw3cSS2OolhloGw/XM6RWPWtPAlGykKLciQrBru5NAPvCMsb/I1DAceTiotQM
fblo6RBxUQ==
-----END CERTIFICATE-----`)
func main() {
pool, err := x509.SystemCertPool()
if err != nil {
log.Fatal(err)
}
if !pool.AppendCertsFromPEM(cert) {
log.Fatal("Cannot append self-signed cert to CA pool")
}
c := &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
RootCAs: pool,
},
},
}
s := httptest.NewTLSServer(nil)
res, err := c.Get(s.URL)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.Status)
}
Try it on the playground: https://play.golang.org/p/HsI2RyOd5qd
I want to connect to a server through proxy server that I have.
I am searching for something that is similar to Python's HTTPConnection.set_tunnel, is there something like this in golang?
----edit-----
I'm trying to create a connection to a server that allows self signed certificates & transfers through proxy, will this code work properly?
func CreateProxyClient(serverProxy string, sid string, portProxy int) (*Client, error) {
http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
proxyUrl, _ := url.Parse(serverProxy+":"+strconv.Itoa(portProxy))
tr := &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
}
var netClient = &http.Client{
Timeout: time.Second * 10,
Transport: tr,
}
return &Client{netClient, serverProxy, sid}, nil
}
You can set environment variable HTTP_PROXY for HTTP or HTTPS_PROXY for HTTPS so the default http transport will use it.
Also as an alternative you can create http.Transport by yourself with Proxy field set to http.ProxyURL function call or use you custom implementation.
Example:
proxyURL, _ := url.Parse("http://proxy.example.com:port")
http.DefaultTransport = &http.Transport{
Proxy: http.ProxyURL(proxyURL),
}
// request using proxy
resp, _ := http.Get("https://google.com"))