I am looking at the grpc example, but I don't understand this example. Can someone explain it to me?
the example located: https://github.com/grpc/grpc-go/blob/master/examples/features/cancellation/server/main.go#L52
func (s *server) BidirectionalStreamingEcho(stream pb.Echo_BidirectionalStreamingEchoServer) error {
for {
in, err := stream.Recv()
if err != nil {
fmt.Printf("server: error receiving from stream: %v\n", err)
if err == io.EOF {
return nil
}
return err
}
fmt.Printf("echoing message %q\n", in.Message)
stream.Send(&pb.EchoResponse{Message: in.Message})
}
}
Can someone explain the process of this execution, how is it called? thx.
the client call the method
stream, err := c.BidirectionalStreamingEcho(ctx)
so server will start the call BidirectionalStreamingEcho
Related
I'm writing a post endpoint and in my error handling I want to return a 400.
log.Println("Artifact post requested")
// Check for valid JSON Body
body, err := getJSONBody(r)
if err != nil {
log.Println("we got an error")
log.Println(err)
w.WriteHeader(http.StatusBadRequest)
return
}
func getJSONBody(r *http.Request) ([]byte, error) {
if r.Body == nil {
fmt.Println("body has issues")
return nil, errors.New("Received an empty body")
}
defer r.Body.Close()
// Read body
body, readErr := ioutil.ReadAll(r.Body)
if readErr != nil {
return nil, readErr
}
return body, nil
}
But when I run unit tests I get a 200 instead of 400. Any ideas why this may happen?
Test output
It was something wrong with the Unit tests, making multiple requests failed but making a single request passed (got the 400 I was looking for).
I am trying to set up a way to communicate between a central server running Go an a fleet of IoT devices (also running Go).
For each device, it connects to to the central server through a persistent TCPConn. These devices are behind a router(s). The central server saves that connection and sends/receives messages through it. Right now, this is fully functional and works.
However, now the message passing is getting complicated enough that the utility provided by HTTP rather than pure TCP is becoming necessary.
I have attempted to write a version of http.Transport that returns said connection. However, I am unable to provide and return a valid connection from the the Dial/DialContext functions.
IoT Device
func main() {
http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
})
tcpAddr, err := net.ResolveTCPAddr("tcp", "###.###.###.###:8533")
if err != nil {
panic(err)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
panic(err)
}
err = conn.SetKeepAlive(true)
if err != nil {
panic(err)
}
err = conn.SetKeepAlivePeriod(time.Second * 10)
if err != nil {
panic(err)
}
fmt.Println("Listening")
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
Central Server
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp", port)
if err != nil {
panic(err)
}
listener, err := net.ListenTCP("tcp", tcpAddr)
if err != nil {
panic(err)
}
conn, err := listener.AcceptTCP()
if err != nil {
panic(err)
}
fmt.Println("Received conn, attempting to send HTTP through connection")
dialFunc := func(network, addr string) (net.Conn, error) {
return conn, nil
}
t := http.Transport{
Dial: dialFunc,
}
client := http.Client{
Transport: &t,
}
fmt.Println("Making request")
res, err := client.Get("http://www.shouldNotMatter.com:8080/foo") // HANGS HERE
if err != nil {
fmt.Println(err)
}
fmt.Println("Received response")
defer res.Body.Close()
if res.StatusCode == http.StatusOK {
bodyBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
bodyString := string(bodyBytes)
fmt.Println(bodyString)
} else {
fmt.Println(res)
}
}
Upon using a debugger to see where it hangs, it seems that it gets stuck in a select statement during the pconn roundtrip. Line 2420 in https://golang.org/src/net/http/transport.go?s=3397:10477
Create a type that returns an existing connection from a dial method:
type connDialer struct {
c net.Conn
}
func (cd connDialer) Dial(network, addr string) (net.Conn, error) {
return cd.c, nil
}
Use the dial method value in a transport:
client := http.Client{Transport: &http.Transport{Dial: connDialer{c}.Dial}}
where c is the existing net.Conn.
Try it on the playground (it works for at one request. It will fail when the client dials a second connection).
The overall approach is fragile. Consider using WebSockets, gRPC or other protocols that are designed to support bi-directional communication.
You missed the code on the client code. The client makes an idle connection to the server and does nothing with it so the connection will definitely hang up. You need to pass the connection to the HTTP server. This can be achieved by using net/http.Serve and passing a net.Listener to it.
type connListener struct {
conn net.Conn
ch chan struct{}
}
func (c connListener) Accept() (Conn, error) {
if c.conn != nil {
conn := c.conn
c.conn = nil
return conn, nil
}
<-c.ch
return nil, errors.New("listener closed")
}
func (c connListener) Close() error {
close(c.ch)
return nil
}
func (c connListener) Addr() net.Addr {
return c.conn.LocalAddr()
}
// call it like this
http.Serve(connListener{conn, make(chan struct{})}, nil)
BTW are you having the client connect to the server and then reverse the connection, making the client behave like an HTTP server and the server behave like the HTTP client? You might want to google "reverse http" for some information on this.
I am trying to implement rpc and http server in my system.
So to listen servers at the same time I have run 2 goroutines
Here is a code
func main() {
// Recovering all errors during the process
defer errorHandler()
wg.Add(2)
go RPCConnect()
fmt.Println("Listening for RPC 127.0.0.1:" + config.rpcPort)
go HTTPConnect()
fmt.Println("Listening for HTTP 127.0.0.1:" + config.httpPort)
wg.Wait()
}
func RPCConnect() {
err := rpc.Register(pool)
if err != nil {
panic(err)
}
rpc.HandleHTTP()
listener, e := net.Listen("tcp", ":"+config.rpcPort)
if e != nil {
panic(e)
}
err = http.Serve(listener, nil)
if err != nil {
panic(err)
}
}
func HTTPConnect() {
var httpPool HTTPPool
r := mux.NewRouter()
r.HandleFunc("/create", httpPool.Create).Methods("POST")
r.HandleFunc("/generate", httpPool.Generate).Methods("POST")
r.HandleFunc("/list", httpPool.List).Methods("GET")
r.HandleFunc("/delete", httpPool.Delete).Methods("POST")
err := http.ListenAndServe("localhost:"+config.httpPort, r)
if err != nil {
panic(err)
}
}
I dont know is this a best way or not . Can somebody tell me more simple and flexible method?
I apologize in advance if the question is not relevant
Suggested approach is pretty well.
If you want to use only one port instead of two, there are several third-party tools to do it:
1) https://github.com/soheilhy/cmux
2) https://github.com/grpc-ecosystem/grpc-gateway
I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.
In the following code is it also necessary to close the response body in the error case:
res, err := http.Get(url)
if err != nil {
log.Printf("Error: %s\n", err)
}
defer res.Body.Close()
General concept is that when a function (or method) has multi return values one being an error, error should be checked first and only proceed if the error is nil. Functions should return zero values for other (non-error) values if there is an error. If the function behaves differently, it should be documented. http.Get() does not document such deviation.
So it should be handled like this:
res, err := http.Get(url)
if err != nil {
log.Printf("Error: %s\n", err)
return
}
defer res.Body.Close()
// Read/work with body
Notes:
As JimB confirms too, if a non-nil error is returned, even if the response is non-nil, we don't have to close it. In case of a redirection error the non-nil response may hold context and further information about where following the redirect failed. See details below:
http.Get() honors the general concept "most of the time": it returns nil response if there is an error:
return nil, someError
However checking client.go, unexported method Client.doFollowingRedirects(), currently line #427:
if redirectFailed {
// Special case for Go 1 compatibility: return both the response
// and an error if the CheckRedirect function failed.
// See https://golang.org/issue/3795
return resp, urlErr
}
So due to a backward compatibility issue it may return a non-nil response and a non-nil error at the same time, if redirection fails.
On the other hand trying to call resp.Body.Close() if resp is nil will cause a run-time panic.
So if we want to close response body in this case, it could look like this (can only be closed if resp is not nil):
res, err := http.Get(url)
if err != nil {
log.Printf("Error: %s\n", err)
}
if res != nil {
defer res.Body.Close()
// Read/work with body
}
Or:
res, err := http.Get(url)
if err != nil {
log.Printf("Error: %s\n", err)
}
if res == nil {
return
}
defer res.Body.Close()
// Read/work with body
The doc of http.Response guarantees that Response.Body will not be nil even if there is no response data:
// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body.
But if the error is not nil, you don't have to close the non-nil response body.