I'm implementing a HTTP long poll client in Go for receiving updates from a server.
What's the difference in normal HTTP client and Long-poll client in Go?
It is just the timeout?
My code looks like this:
request, err := http.NewRequest(get, url, nil)
if err != nil {
fmt.Print("Error")
return
}
request.Header.Set(consulTokenHeader, aclToken)
httpClient := &http.Client{} //introduce timeout?
response, err := httpClient.Do(request)
if err != nil {
log.Fatal(err)
}
Related
Problem Statement
I have a client (which dials to the server) and server (that listens for incoming requests) written in golang and with the RPC calls defined. I am trying to initiate an HTTP request on the server side which would in turn execute the RPC call for streaming and send a JSON response back to the user
Challenge
I was able to handle both grpc and HTTP requests on different ports but having issues with passing parameters from the HTTP request onto the RPC call on the server side
Server Code
log.Println("Listening for connections from client ........")
lis, err := net.Listen("tcp", ":9000")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := testApi.Server{}
grpcServer := grpc.NewServer()
testApi.RegisterTestApiServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %s", err)
}
func main() {
go runGrpc()
log.Printf("*------ Waiting for requests from users ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
test_id, _ := strconv.Atoi(vars["test_id"])
log.Println("Test id request from user : ", test_id)
func (s * Server) ConnectAndStream(channelStream TestApiService_ConnectAndStreamServer) error {
// Question: This Id has to come from http request above- test_id
var id int32 = 1234566
// id := a.ConnectAndExchange
log.Println("Id from sam user ", id)
// var id int32 = 1234566
for i := 1; i <= 2; i++ {
id += 1
log.Println("Speed Server is sending data : ", id)
channelStream.Send(&Input{Id: id})
}
for i := 1; i <= 2; i++ {
log.Println("now time to receive")
client_response, err := channelStream.Recv()
log.Println("Response from samd client : ", client_response.Id)
if err != nil {
log.Println("Error while receiving from samd : ", err)
}
}
return nil
}
I am stuck with being able to pass the test_id from the curl request to the RPC call as above. Any input is greatly appreciated
Note
Client - Dials in and connects to the server and starts receiving and sending data (bi-directional streaming)
Both the Http and GRPC client are part of the same server application. So why call the RPC method from the Http handler? The Http handler should have access to the same backend functionality.
Your question is slightly unclear but if you are trying to have your client establish a GRPC connection to the server via the HTTP handler this will not work. The GRPC connection established in this situation is between the server and its self.
Edit - thanks for the clarification. Now I understand better the flow that you are trying to achieve. Your http handler method can make the outgoing grpc call to the server and return the response back via the http.ResponseWriter
For simplicity I have used the hello world example on https://github.com/grpc/grpc-go/tree/master/examples/helloworld
Running the code sample below and hitting http://localhost:1000/exchangeId/Test will show the output
Starting
*------ Waiting for http requests from users on port 1000 ------*
server listening at 127.0.0.1:1001
Test id request from user : Test
Server Received: Test
Greeting: Hello Test
Code sample:
import (
"context"
"log"
"net"
"net/http"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
"github.com/gorilla/mux"
)
var (
grpcserver = "localhost:1001"
)
func main() {
log.Print("Starting")
go StartGrpcServer()
log.Printf("*------ Waiting for http requests from users on port 1000 ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":1000", router))
}
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Server Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
func StartGrpcServer() {
lis, err := net.Listen("tcp", grpcserver)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
test_id := vars["test_id"]
log.Println("Test id request from user : ", test_id)
// Set up a connection to the server.
conn, err := grpc.Dial(grpcserver, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
resp, err := c.SayHello(ctx, &pb.HelloRequest{Name: test_id})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", resp.GetMessage())
w.Write([]byte(resp.GetMessage()))
}
What I try to achieve:
I want to recieve Http request from client then hijack the connection to monitor it on server side (Checking health of connection). Also I want to send Http response on that hijacked connection. In that order: recieve Http request, get request body, hijack connection, return response to client, monitor connection health on server side.
What I've already achieved:
Here is the code of http request handler:
func (c *HandlerCtx) HijackHandler(w http.ResponseWriter, req *http.Request){
// Fetching request body skipped
// Hijacking the connection
h, _ := w.(http.Hijacker)
conn, br, err := h.Hijack()
if err != nil {
fmt.Println(err.Error())
return
}
responseBody := "Http response from hijacked connection"
hr := http.Response{
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header, 0),
Body: ioutil.NopCloser(bytes.NewBufferString(responseBody)),
ContentLength: int64(len(responseBody)),
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: req,
TLS: nil,
}
// Writing response
err = hr.Write(br)
if err != nil{
fmt.Println(err.Error())
return
}
// Sending EOF to allow io.ReadAll(resp.Body) without blocking
if v, ok := conn.(interface{ CloseWrite() error }); ok {
err = v.CloseWrite()
if err != nil {
fmt.Println(err.Error())
}
}
// Monitor connection health
}
This is the client code:
func main(){
// Body skipped for testing purposes
resp, err := http.Post("http://127.0.0.1:8085/hello", "application/json", nil)
if err != nil{
fmt.Println(err.Error())
return
}
b, err := io.ReadAll(resp.Body)
if err != nil{
fmt.Println(err.Error())
return
}
}
Now I recieve unexpected EOF from client after calling err = v.CloseWrite() but when I don't CloseWrite the client code stuck on io.ReadAll(resp.Body)
Is there any way to force client to read that http response? Please help me find solution.
I am trying to set up a way to communicate between a central server running Go an a fleet of IoT devices (also running Go).
For each device, it connects to to the central server through a persistent TCPConn. These devices are behind a router(s). The central server saves that connection and sends/receives messages through it. Right now, this is fully functional and works.
However, now the message passing is getting complicated enough that the utility provided by HTTP rather than pure TCP is becoming necessary.
I have attempted to write a version of http.Transport that returns said connection. However, I am unable to provide and return a valid connection from the the Dial/DialContext functions.
IoT Device
func main() {
http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
})
tcpAddr, err := net.ResolveTCPAddr("tcp", "###.###.###.###:8533")
if err != nil {
panic(err)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
panic(err)
}
err = conn.SetKeepAlive(true)
if err != nil {
panic(err)
}
err = conn.SetKeepAlivePeriod(time.Second * 10)
if err != nil {
panic(err)
}
fmt.Println("Listening")
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
Central Server
func main() {
tcpAddr, err := net.ResolveTCPAddr("tcp", port)
if err != nil {
panic(err)
}
listener, err := net.ListenTCP("tcp", tcpAddr)
if err != nil {
panic(err)
}
conn, err := listener.AcceptTCP()
if err != nil {
panic(err)
}
fmt.Println("Received conn, attempting to send HTTP through connection")
dialFunc := func(network, addr string) (net.Conn, error) {
return conn, nil
}
t := http.Transport{
Dial: dialFunc,
}
client := http.Client{
Transport: &t,
}
fmt.Println("Making request")
res, err := client.Get("http://www.shouldNotMatter.com:8080/foo") // HANGS HERE
if err != nil {
fmt.Println(err)
}
fmt.Println("Received response")
defer res.Body.Close()
if res.StatusCode == http.StatusOK {
bodyBytes, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
bodyString := string(bodyBytes)
fmt.Println(bodyString)
} else {
fmt.Println(res)
}
}
Upon using a debugger to see where it hangs, it seems that it gets stuck in a select statement during the pconn roundtrip. Line 2420 in https://golang.org/src/net/http/transport.go?s=3397:10477
Create a type that returns an existing connection from a dial method:
type connDialer struct {
c net.Conn
}
func (cd connDialer) Dial(network, addr string) (net.Conn, error) {
return cd.c, nil
}
Use the dial method value in a transport:
client := http.Client{Transport: &http.Transport{Dial: connDialer{c}.Dial}}
where c is the existing net.Conn.
Try it on the playground (it works for at one request. It will fail when the client dials a second connection).
The overall approach is fragile. Consider using WebSockets, gRPC or other protocols that are designed to support bi-directional communication.
You missed the code on the client code. The client makes an idle connection to the server and does nothing with it so the connection will definitely hang up. You need to pass the connection to the HTTP server. This can be achieved by using net/http.Serve and passing a net.Listener to it.
type connListener struct {
conn net.Conn
ch chan struct{}
}
func (c connListener) Accept() (Conn, error) {
if c.conn != nil {
conn := c.conn
c.conn = nil
return conn, nil
}
<-c.ch
return nil, errors.New("listener closed")
}
func (c connListener) Close() error {
close(c.ch)
return nil
}
func (c connListener) Addr() net.Addr {
return c.conn.LocalAddr()
}
// call it like this
http.Serve(connListener{conn, make(chan struct{})}, nil)
BTW are you having the client connect to the server and then reverse the connection, making the client behave like an HTTP server and the server behave like the HTTP client? You might want to google "reverse http" for some information on this.
I'm using httputil.ReverseProxy with an http.RoundTripper of my own implementation that uses an ssh.Channel as a transport. My RoundTrip method looks approximately like this:
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
// defer ch.Close()
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
return http.ReadResponse(bufio.NewReader(ch), req)
}
func (c SSHConnection) GetChannel() (ssh.Channel, error) {
ch, req, err := c.Conn.OpenChannel("forwarded-tcpip", msg)
if err != nil {
return nil, err
}
go ssh.DiscardRequests(req)
return ch, nil
}
Notice the commented-out defer ch.Close(). Initially I naively closed the connection here, but the response body would sometimes be empty, due to a race between the HTTP proxy's reading of the body and this closing of the SSH channel.
Assuming, for now, that I don't care to do keep-alive, when can I close the ssh.Channel? If I don't, every request starts a new goroutine (because of go ssh.DiscardRequests(req)), so I leak a goroutine on every HTTP requests until the underlying SSH connection is closed.
An http.RoundTripper should not close the connection until after the response body has been fully consumed, or at the request of the server.
The simplest option is to fully buffer the response and close the connection immediately. In some cases this may actually be the most efficient, if the traffic mostly consists of small, independent requests.
The next option is to hook the closing of the response body to close the channel.
type Body struct {
io.ReadCloser
channel ssh.Channel
}
func (b *Body) Close() error {
b.channel.Close()
return b.ReadCloser.Close()
}
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
resp, err := http.ReadResponse(bufio.NewReader(ch), req)
if err != nil {
ch.Close()
return nil, err
}
resp.Body = &Body{
ReadCloser: resp.Body,
channel: ch,
}
return resp, err
}
Lastly, to make the most efficient use of the ssh channels, you could use an existing Transport with a net.Dialer which makes the ssh connection, and wraps the channel in a net.Conn interface.
I am using go http client to make get requests and the client has been initialised with a cookiejar however the response cookie array is empty. Has anyone got any idea what I am doing wrong?
jar, err := cookiejar.New(nil)
if err != nil {
log.Fatal(err)
}
s.http_client = &http.Client{Jar: jar}
resp, _ := s.http_client.Get(s.url)
fmt.Println(resp.Cookies()) returns an empty array although I can see cookies returned in firefox.
You create a cookiejar, and you can use it as seen in "how to follow location with cookie":
jar, err := cookiejar.New(&options)
if err != nil {
log.Fatal(err)
}
client := http.Client{Jar: jar} // <============
resp, err := client.Get("http://dubbelboer.com/302cookie.php")
if err != nil {
log.Fatal(err)
}
data, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
(introduced with Go1.1 as in this answer)
An http.Client struct has:
// Jar specifies the cookie jar.
// If Jar is nil, cookies are not sent in requests and ignored
// in responses.
Jar CookieJar
As 3of3 mentions, you don't need a cookiejar to fetch a cookie:
for _, cookie := range r.Cookies() {
fmt.Fprint(w, cookie.Name)
}
Check if the cookiejar is still empty after having read the full response body.