How do I use nginx as a reverse proxy alongside Golang? - http

I want to use Golang as my server side language, but everything I've read points to nginx as the web server rather than relying on net/http (not that it's bad, but it just seems preferable overall, not the point of this post though).
I've found a few articles on using fastcgi with Golang, but I have no luck in finding anything on reverse proxies and HTTP and whatnot, other than this benchmark which doesn't go into enough detail unfortunately.
Are there any tutorials/guides available on how this operates?
For example there is a big post on Stackoverflow detailing it with Node, but I cannot find a similar one for go.

That's not needed at all anymore unless you're using nginx for caching, Golang 1.6+ is more than good enough to server http and https directly.
However if you're insisting, and I will secretly judge you and laugh at you, here's the work flow:
Your go app listens on a local port, say "127.0.0.1:8080"
nginx listens on 0.0.0.0:80 and 0.0.0.0:443 and proxies all requests to 127.0.0.1:8080.
Be judged.
The nginx setup in Node.js + Nginx - What now? is exactly the same setup you would use for Go, or any other standalone server for that matter that isn't cgi/fastcgi.

I use Nginx in production very effectively, using Unix sockets instead of TCP for the FastCGI connection. This code snippet comes from Manners, but you can adapt it for the normal Go api quite easily.
func isUnixNetwork(addr string) bool {
return strings.HasPrefix(addr, "/") || strings.HasPrefix(addr, ".")
}
func listenToUnix(bind string) (listener net.Listener, err error) {
_, err = os.Stat(bind)
if err == nil {
// socket exists and is "already in use";
// presume this is from earlier run and therefore delete it
err = os.Remove(bind)
if err != nil {
return
}
} else if !os.IsNotExist(err) {
return
}
listener, err = net.Listen("unix", bind)
return
}
func listen(bind string) (listener net.Listener, err error) {
if isUnixNetwork(bind) {
logger.Printf("Listening on unix socket %s\n", bind)
return listenToUnix(bind)
} else if strings.Contains(bind, ":") {
logger.Printf("Listening on tcp socket %s\n", bind)
return net.Listen("tcp", bind)
} else {
return nil, fmt.Errorf("error while parsing bind arg %v", bind)
}
}
Take a look around about line 252, which is where the switching happens between HTTP over a TCP connection and FastCGI over Unix-domain sockets.
With Unix sockets, you have to adjust your startup scripts to ensure that the sockets are created in an orderly way with the correct ownership and permissions. If you get that right, the rest is easy.
To answer other remarks about why you would want to use Nginx, it always depends on your use-case. I have Nginx-hosted static/PHP websites; it is convenient to use it as a reverse-proxy on the same server in such cases.

Related

How to determine exactly when a golang server is listening on a port [duplicate]

When I look at the net/http server interface, I don't see an obvious way to get notified and react when the http.Server comes up and starts listening:
ListenAndServe(":8080", nil)
The function doesn't return until the server actually shuts down. I also looked at the Server type, but there doesn't appear to be anything that lets me tap into that timing. Some function or a channel would have been great but I don't see any.
Is there any way that will let me detect that event, or am I left to just sleeping "enough" to fake it?
ListenAndServe is a helper function that opens a listening socket and then serves connections on that socket. Write the code directly in your application to signal when the socket is open:
l, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
// Signal that server is open for business.
if err := http.Serve(l, rootHandler); err != nil {
// handle error
}
If the signalling step does not block, then http.Serve will easily consume any backlog on the listening socket.
Related question: https://stackoverflow.com/a/32742904/5728991

Using Go standard libs, why do I leak TCP connections constantly in this two-tier architecture?

In this situation, I'm using all standard Go libraries -- net/http, most importantly.
The application consists of two layers. The first layer is the basic web application. The web application serves out the UI, and proxies a bunch of API calls back to the second layer based on username -- so, it's effectively a load balancer with consistent hashing -- each user is allocated to one of these second-layer nodes, and any requests pertaining to that user must be sent to that particular node.
Quick details
These API endpoints in the first layer effectively read in a JSON body, check the username, use that to figure out which of the layer 2 nodes to send the JSON body to, and then it sends it there. This is done using a global http.Client that has timeouts set on it, as appropriate.
The server side does a defer request.Body.Close() in each of the handlers after ensuring no error comes back from decoder.Decode(&obj) calls that unmarshal the JSON. If there is any codepath where that could happen, it isn't one that's likely to get followed very often.
Symptoms
On the node in the second layer (the application server) I get log lines like this because it's leaking sockets presumably and sucking up all the FDs:
2019/07/15 16:16:59 http: Accept error: accept tcp [::]:8100: accept4: too many open files; retrying in 1s
2019/07/15 16:17:00 http: Accept error: accept tcp [::]:8100: accept4: too many open files; retrying in 1s
And, when I do lsof 14k lines are output, of which 11,200 are TCP sockets. When I look into the contents of lsof, I see that nearly all these TCP sockets are in connection state CLOSE_WAIT, and are between my application server (second layer node) and the web server (the first layer node).
Interestingly, nothing seems to go wrong with the web application server (layer 1) during this timeframe.
Why does this happen?
I've seen lots of explanations, but most either point out that you need to specify custom defaults on a custom http.Client and not use the default, or they tell you to make sure to close the request bodies after reading from them in the layer 2 handlers.
Given all this information, does anyone have any idea what I can do to at least put this to bed once and for all? Everything I search on the internet is user error, and while I certainly hope that's the case here, I worry that I've nailed down every last quirk of the Go standard library I can find.
Been having trouble nailing down exactly how long it takes for this to happen -- the last time it happened, it was up for 3 days before I started to see this error, and at that point obviously nothing recovers until I kill and restart the process.
Any help would be hugely appreciated!
EDIT: example of client-side code
Here is an example of what I'm doing in the web application (layer 1) to call the layer 2 node:
var webHttpClient = &http.Client{
Transport: &http.Transport{
MaxIdleConnsPerHost: MaxIdleConnections,
},
Timeout: time.Second * 20,
}
// ...
uri := fmt.Sprintf("http://%s/%s", tsUri, "pms/all-venue-balances")
req, e := http.NewRequest("POST", uri, bytes.NewBuffer(b))
resp, err := webHttpClient.Do(req)
if err != nil {
log.Printf("Submit rebal error 3: %v\n", err)
w.WriteHeader(500)
return
}
defer resp.Body.Close()
body, _ := ioutil.ReadAll(resp.Body)
w.WriteHeader(200)
w.Write(body)

Dial tcp I/O timeout on simultaneous requests

I am building a tool in Go that needs to make a very large number of simultaneous HTTP requests to many different servers. My initial prototype in Python had no problem doing a few hundred simultaneous requests.
However, I have found that in Go this almost always results in a Get http://www.google.com: dial tcp 216.58.205.228:80: i/o timeout for some if the number of simultaneous requests exceeds ~30-40.
I've tested on macOS, openSUSE, different hardware, in different networks and with different domain lists, and changing the DNS server as described in other Stackoverflow answers does not work either.
The interesting thing is that the failed requests do not even produce a packet, as can be seen when checking with Wireshark.
Is there anything that I am doing wrong or is that a bug in Go?
Minimum reproducible program below:
package main
import (
"fmt"
"net/http"
"sync"
)
func main() {
domains := []string{/* large domain list here, eg from https://moz.com/top500 */}
limiter := make(chan string, 50) // Limits simultaneous requests
wg := sync.WaitGroup{} // Needed to not prematurely exit before all requests have been finished
for i, domain := range domains {
wg.Add(1)
limiter <- domain
go func(i int, domain string) {
defer func() { <-limiter }()
defer wg.Done()
resp, err := http.Get("http://"+domain)
if err != nil {
fmt.Printf("%d %s failed: %s\n", i, domain, err)
return
}
fmt.Printf("%d %s: %s\n", i, domain, resp.Status)
}(i, domain)
}
wg.Wait()
}
Two particular error messages are happening, a net.DNSError that does not make any sense and a non-descript poll.TimeoutError:
&url.Error{Op:"Get", URL:"http://harvard.edu", Err:(*net.OpError)(0xc00022a460)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc000aca200)}
&net.DNSError{Err:"no such host", Name:"harvard.edu", Server:"", IsTimeout:false, IsTemporary:false}
&url.Error{Op:"Get", URL:"http://latimes.com", Err:(*net.OpError)(0xc000d92730)}
&net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*poll.TimeoutError)(0x14779a0)}
&poll.TimeoutError{}
Update:
Running the requests with a seperate http.Client as well as http.Transport and net.Dialer does not make any difference as can be seen when running code from this playground.
I think many of your net.DNSErrors are actually too many open files errors in disguise. You can see this by running your sample code with the netgo tag (recommendation from here) (go run -tags netgo main.go) which will emit errors like:
…dial tcp: lookup buzzfeed.com on 192.168.1.1:53: dial udp 192.168.1.1:53: socket: too many open files
instead of
…dial tcp: lookup buzzfeed.com: no such host
Make sure you're closing the request's response body (resp.Body.Close()). You can find more about this specific problem at What's the best way to handle "too many open files"? and How to set ulimit -n from a golang program?. (On my machine (macOS), increasing file limits manually seemed to help, but I don't think it's a good solution since it doesn't really scale, and I'm not sure how many open files you'd need overall.)
As suggested by #liam-kelly, I think the i/o timeout error is coming from a DNS server or some other security mechanism. Setting a custom (bad) DNS server IP gives me the same error.

Check for Internet connection from application

I need to check if the user is connected to internet before I can proceed.
I am hitting the endpoint using HttpClient as follows:
client := &http.Client{}
req, _ := http.NewRequest("GET", url, nil)
req.SetBasicAuth(username, password)
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
ui.Failed("Check your internet connection")
}
1) I need to show clear messages to the user if the user is not connected to the internet in this case, display "Please check your internet connection"
2) In case the server is not responding, and receives 504 bad gateway,
display "504 Bad gateway"
Can someone help how to proceed and distinguish between these two scenarios and I would like to display only simple messages and not the entire error messages received from the server.
Checking for an established Internet connection isn't as simple as making a single HTTP request to an arbitray URL, like Ivan de la Beldad suggests. This can fail for any number of reasons, none of which will necessarily stop you from doing what you actually intend to do with the connection. To name a few:
clients3.google.com may be deliberatly blocked by the local network (firewall, corporate or school proxy) or any en-route network (think Great Firewall of China)
clients3.google.com may be unreachable for some clients due to network outages
clients3.google.com itself may block the client for whatever reason (perhaps unintentionally)
clients3.google.com may have an outage
port 80 and 443 may work fine, but all other ports are blocked by shitty hotel/coffee shop wifi
shitty hotel/coffee shop wifi presents fake TLS certificates to clients, so HTTPS requests will fail in many cases
So instead of relying on a single arbitrary HTTP request it is much better to send some kind of liveliness probe to whatever service(s) you actually want to use.
If your app wants to communicate with an API, see if there is a health or status endpoint that you can call. If there isn't, look for some kind of cheap no-op. And try not to tell users to simply "check their Internet connection". Try to at least explain why your app concludes that there might be an issue "We can't connect to Twitter right now. If you are connected to the Internet try again in a few minutes" is much better.
On the off-chance that you really only want to check if the Internet itself is available, I would suggest making a DNS query to several DNS servers on the Internet. DNS is not likely to be blocked through local policies and cheaper than HTTP requests. Pick your DNS queries wisely and be prepared for NXDOMAIN responses.
To check if you're connected to internet you can use this:
func connected() (ok bool) {
_, err := http.Get("http://clients3.google.com/generate_204")
if err != nil {
return false
}
return true
}
And to get the status code you can have it from res.StatusCode.
The final result would be something like that:
if !connected() {
ui.Failed("Check your internet connection")
}
client := &http.Client{}
req, _ := http.NewRequest("GET", url, nil)
req.SetBasicAuth(username, password)
res, _ := client.Do(req)
if res.StatusCode == 504 {
ui.Failed("504 Bad gateway")
}
(I'm ignoring other errors that obviusly you should check)

How to use next available port in http.ListenAndServe

I wrote a simple web server to listen on port 8080. But I don't want to use a hard coded port number. What I want is that my server listen on any available port. And I want to know that on what port number my web server is listening.
My code is given bellow:
package main
import (
"net/http"
)
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
You may use port 0 to indicate you're not specifying an exact port but you want a free, available port selected by the system:
http.ListenAndServe(":0", nil)
The problem with this is that you won't be able to find out what port was assigned. So you need to create the net.Listener yourself (using the net.Listen() function), and manually pass it to http.Serve():
listener, err := net.Listen("tcp", ":0")
if err != nil {
panic(err)
}
fmt.Println("Using port:", listener.Addr().(*net.TCPAddr).Port)
panic(http.Serve(listener, nil))
Example output:
Using port: 42039
As you can see, you can access the assigned port from the net.Listener, from its net.Addr address (acquired by its Addr() method). net.Addr does not directly give access to the port, but since we created the net.Listener using tcp network stream, the net.Addr will be of dynamic type *net.TCPAddr (which we can acquire with a type assertion), which is a struct and has a field Port int.
Note that if you don't need the port in your application (e.g. you just want to display it for yourself), you don't need the type assertion, you can just print listener.Addr() (which will contain the port at the end):
fmt.Println("Address:", listener.Addr())
Example output:
Address: [::]:42039
Also don't forget to handle returned errors (http.ListenAndServe() in this case). In my example I simply passed it to panic() because http.LitenAndServe() and http.Serve() block if everything goes well (so they only return if there's an error, which I pass to panic()).
I landed on this page because I wanted to start a new server on a random port for testing purposes. I later learned about the httptest package, https://golang.org/pkg/net/http/httptest/ , which does a much better job of managing temporary HTTP server for testing.
Package https://pkg.go.dev/net/http/httptest supports HTTP server on random port.
package main
import (
"fmt"
"net/http"
"net/http/httptest"
)
func main() {
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
})
server := httptest.NewServer(handler)
defer server.Close()
fmt.Println("Server started", server.URL)
}
Example output
Server started http://127.0.0.1:54926

Resources