When sending Post to a https rest api endpoint in my local network I'm receiving the error:
wsarecv: An existing connection was forcibly closed by the remote host.
However, when I use Postman on the same server it works fine.
The problem is specific to my Go http configuration, I'm just not sure what the problem is.
I've tried disabling TLS on the http client but the problem persists:
// disable SSLVerify
func getHttpClient() *http.Client {
tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}} // disable SSL integrity check
return &http.Client{Transport: tr}
}
edit: correction, it's being routed over a proxy, probably Bluecoat, that's performing some MitM operations. But still can't figure out why it works over Postman, but not with Golang.
I am trying to implement a HTTP server in Golang that receives requests from an Amazon ELB that uses the proxy protocol. Now I'd like to know what the original IP address is and so I was thinking about using this package.
Now this package talks raws HTTP as far as I can tell but my server implements a higher level HTTP server with a router.
I am having trouble translating between them. My question is this: how do I use this library and still use a router like gorilla/mux? Now there's nothing special about this package, it just talks at a lower level than HTTP.
Example:
// Listen on TCP port 2000 on all interfaces.
l, err := net.Listen("tcp", ":2000")
// proxyproxy is the library that maintains the source ip address for me
proxyList := &proxyproto.Listener{Listener: list}
if err != nil {
log.Fatal(err)
}
defer proxyList.Close()
for {
// Wait for a connection.
conn, err := proxyList.Accept()
if err != nil {
log.Fatal(err)
}
// Handle the connection in a new goroutine.
// The loop then returns to accepting, so that
// multiple connections may be served concurrently.
go func(c net.Conn) {
// how do I connect my router
}(conn)
}
The usual way of finding out the actual client IP over HTTP is by using some HTTP headers such as:
X-Forwarded-For
X-Real-IP
Actually, Amazon ELB seems to support the X-Forwarded-For header:
http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/x-forwarded-headers.html
If you are using Gorilla, they have a middleware that seems to take care of that for you:
https://godoc.org/github.com/gorilla/handlers#ProxyHeaders
I was going through this code of gRPC server. Can anyone tell me the need for reflection used here
Code :
func main() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
// Register reflection service on gRPC server.
reflection.Register(s)
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Server reflection is not necessary to run the helloworld example.
The helloworld example is also used as a server reflection example, that's why you see the reflection registering code there.
More about server reflection:
Server reflection is a service defined to provides information about publicly-accessible gRPC services on a gRPC server. Tutorial available here: https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md
server-based-reflection is something that you will not need to build your day-to-day gRPC APIs.
This is a special instruction which exposes all the publicly accessible gRPC services on a gRPC server.
What this means essentially is that anyone can request your gRPC server to emit out details of the RPC service methods, request-response structures.
Where is this used?
This is used at places where you want to. dynamically call gRPC APIs. By dynamically I mean, the client does not need to hold the proto data-structures and register the RPC client stub.
grpCurl - curl gRPC services
gRPC transcoding - exposing json APIs on gRPC servers
In Ruby/Rack, I'm able to get the scheme of the current request URL from scheme#request. However, in Go, http.Request.URL.Scheme returns an empty string:
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%#v\n", r.URL.Scheme) // Always shows empty string
}
How do I get scheme of the current request URL?
To serve HTTP and HTTPS you will need to call both serve functions
http.ListenAndServe() and http.ListenAndServeTLS() with the same handler
because if you are using only 1 of them like the example of the question then you are only listing on 1 protocol http.ListenAndServe() for http, and http.ListenAndServeTLS() for HTTPS, if you will try to contact the server with a different protocol it will not go through,
and because HTTPS is HTTP over TLS the *http.Request has a TLS property that will give you back a *tls.ConnectionState with info about the TLS that was used on this request, then if you want to know how the client contact the server you can check on the request TLS property,
if the request were made with HTTPS it would not be nil,
if the request was made with HTTP then the TLS property will be nil,
because the only way that a request was made with TLS is with the HTTPS protocol
func handler(w http.ResponseWriter, r *http.Request) {
if r.TLS == nil {
// the scheme was HTTP
} else {
// the scheme was HTTPS
}
}
func main() {
http.HandleFunc("/", handler)
go func(){
log.Fatal(http.ListenAndServeTLS(":8443","localhost.crt", "localhost.key", nil))
}()
log.Fatal(http.ListenAndServe(":8080", nil))
}
A quick grep shows that r.URL.Scheme is never set to anything other than the empty string anywhere in net/http. Personally I think it should be, as far as possible, but apparently I have a minority opinion.
If you opened a TLS listener yourself with http.ListenAndServeTLS() then presumably you know the scheme is https already. You can use a trivial middleware handler that fills in r.URL.Scheme in this case.
func AlwaysHTTPS(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.URL.Scheme = "https"
next.ServeHTTP(w, r)
})
}
If you're running behind a web server, then it may pass the request protocol in a header such as X-Forwarded-Proto. In this case, you can use a handler like gorilla's handlers.ProxyHeaders() to fill in the missing fields.
An example using gorilla mux:
package main
import (
"log"
"net/http"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
)
func main() {
r := mux.NewRouter()
r.Use(handlers.ProxyHeaders)
http.Handle("/", r)
log.Fatal(http.ListenAndServe("[::]:8009", nil))
}
From its comments:
ProxyHeaders inspects common reverse proxy headers and sets the corresponding fields in the HTTP request struct. These are X-Forwarded-For and X-Real-IP for the remote (client) IP address, X-Forwarded-Proto or X-Forwarded-Scheme for the scheme (http|https) and the RFC7239 Forwarded header, which may include both client IPs and schemes.
NOTE: This middleware should only be used when behind a reverse proxy like nginx, HAProxy or Apache. Reverse proxies that don't (or are configured not to) strip these headers from client requests, or where these headers are accepted "as is" from a remote client (e.g. when Go is not behind a proxy), can manifest as a vulnerability if your application uses these headers for validating the 'trustworthiness' of a request.
Since you use ListenAndServe and not ListenAndServeTLS, your scheme can be safely assumed as http. If you use both tls and non tls versions, you can use r.TLS and check it for null to know whether TLS was established. If your go app is running behind a reverse proxy, then you have to check the documentation on the web server which is forwarding the requests to your app, to learn how to configure it to pass this information as headers. Here's a link describing nginx configuration which accomplishes that. You can easily find configuration guides for other webservers as well.
Better yet, configure HSTS on your main web server, so that you don't have to worry about insecure connections altogether. There are very few, if any, legitimate uses to non-TLS http. For nginx you can find this article useful. And again for other web servers you will easily find configuration guides.
If you're unsure if your site/application needs https I recommend reading this.
localhost is a special case for URL formation. It is going to be empty anyway if your client is localhost.
net.http package doc:
As a special case, if req.URL.Host is "localhost" (with or without a port number), then a nil URL and nil error will be returned.
The way to get required url/uri information is to get it from http.Request directly. For example:
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%s\n", r.Host)
}
http header X-Forwarded-Proto will have http or https
It is because, you're accessing the HTTP server so:
GET / HTTP/1.1
Host: localhost:8080
in this cases, based on parsing what you get is raw URL from Go's http.Request.URL. why you are getting this is because you are accessing the URL from a relative path, hence the lack of a Host or Scheme in the URL object.
If you do want to get the HTTP host, you may have to access the Host attribute of the http.Request struct. See http://golang.org/pkg/http/#Request
as it is not directly available, but you can still be able to assemble it:
u := r.URL
// The scheme can be http/https because that's depends on protocol your server handles.
u.Scheme = "http"
I'm trying to connect to a proxy server to start an HTTP CONNECT tunnel. The proxy server uses authentication. This code however fails to work:
conn, err := net.Dial("tcp", "[user:pass#111.222.333.444]:5555")
Even though the host exists, I get an error:
"dial tcp: lookup user:pass#111.222.333.444: no such host"
The string format I'm using was described in this post. Can't seem to get it to work though.
https://stackoverflow.com/a/8858209/6767074
I eventually found the problem. The net.Dial() method wasn't the one concerned with proxy authentication.
I just had to fill in the "Proxy-Authorization" header of the request before calling for am HTTP response. So my TCP address became:
conn, err := net.Dial("tcp", "111.222.333.444:5555")