This question already has answers here:
golang http+jsonrpc access from web page
(2 answers)
Closed 5 years ago.
How can I use JSON-RPC over HTTP based on this specification in Go?
Go provides JSON-RPC codec in net/rpc/jsonrpc but this codec use network connection as input so you cannot use it with go RPC HTTP handler. I attach sample code that uses TCP for JSON-RPC:
func main() {
cal := new(Calculator)
server := rpc.NewServer()
server.Register(cal)
listener, e := net.Listen("tcp", ":1234")
if e != nil {
log.Fatal("listen error:", e)
}
for {
if conn, err := listener.Accept(); err != nil {
log.Fatal("accept error: " + err.Error())
} else {
log.Printf("new connection established\n")
go server.ServeCodec(jsonrpc.NewServerCodec(conn))
}
}
}
The built-in RPC HTTP handler uses the gob codec on a hijacked HTTP connection. Here's how to do the same with the JSONRPC.
Write an HTTP handler that runs the JSONRPC server with a hijacked connection.
func serveJSONRPC(w http.ResponseWriter, req *http.Request) {
if req.Method != "CONNECT" {
http.Error(w, "method must be connect", 405)
return
}
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
http.Error(w, "internal server error", 500)
return
}
defer conn.Close()
io.WriteString(conn, "HTTP/1.0 Connected\r\n\r\n")
jsonrpc.ServeConn(conn)
}
Register this handler with the HTTP server. For example:
http.HandleFunc("/rpcendpoint", serveJSONRPC)
EDIT: OP has since updated question to make it clear that they want GET/POST instead of connect.
Related
Problem Statement
I have a client (which dials to the server) and server (that listens for incoming requests) written in golang and with the RPC calls defined. I am trying to initiate an HTTP request on the server side which would in turn execute the RPC call for streaming and send a JSON response back to the user
Challenge
I was able to handle both grpc and HTTP requests on different ports but having issues with passing parameters from the HTTP request onto the RPC call on the server side
Server Code
log.Println("Listening for connections from client ........")
lis, err := net.Listen("tcp", ":9000")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := testApi.Server{}
grpcServer := grpc.NewServer()
testApi.RegisterTestApiServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %s", err)
}
func main() {
go runGrpc()
log.Printf("*------ Waiting for requests from users ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
test_id, _ := strconv.Atoi(vars["test_id"])
log.Println("Test id request from user : ", test_id)
func (s * Server) ConnectAndStream(channelStream TestApiService_ConnectAndStreamServer) error {
// Question: This Id has to come from http request above- test_id
var id int32 = 1234566
// id := a.ConnectAndExchange
log.Println("Id from sam user ", id)
// var id int32 = 1234566
for i := 1; i <= 2; i++ {
id += 1
log.Println("Speed Server is sending data : ", id)
channelStream.Send(&Input{Id: id})
}
for i := 1; i <= 2; i++ {
log.Println("now time to receive")
client_response, err := channelStream.Recv()
log.Println("Response from samd client : ", client_response.Id)
if err != nil {
log.Println("Error while receiving from samd : ", err)
}
}
return nil
}
I am stuck with being able to pass the test_id from the curl request to the RPC call as above. Any input is greatly appreciated
Note
Client - Dials in and connects to the server and starts receiving and sending data (bi-directional streaming)
Both the Http and GRPC client are part of the same server application. So why call the RPC method from the Http handler? The Http handler should have access to the same backend functionality.
Your question is slightly unclear but if you are trying to have your client establish a GRPC connection to the server via the HTTP handler this will not work. The GRPC connection established in this situation is between the server and its self.
Edit - thanks for the clarification. Now I understand better the flow that you are trying to achieve. Your http handler method can make the outgoing grpc call to the server and return the response back via the http.ResponseWriter
For simplicity I have used the hello world example on https://github.com/grpc/grpc-go/tree/master/examples/helloworld
Running the code sample below and hitting http://localhost:1000/exchangeId/Test will show the output
Starting
*------ Waiting for http requests from users on port 1000 ------*
server listening at 127.0.0.1:1001
Test id request from user : Test
Server Received: Test
Greeting: Hello Test
Code sample:
import (
"context"
"log"
"net"
"net/http"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
"github.com/gorilla/mux"
)
var (
grpcserver = "localhost:1001"
)
func main() {
log.Print("Starting")
go StartGrpcServer()
log.Printf("*------ Waiting for http requests from users on port 1000 ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":1000", router))
}
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Server Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
func StartGrpcServer() {
lis, err := net.Listen("tcp", grpcserver)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
test_id := vars["test_id"]
log.Println("Test id request from user : ", test_id)
// Set up a connection to the server.
conn, err := grpc.Dial(grpcserver, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
resp, err := c.SayHello(ctx, &pb.HelloRequest{Name: test_id})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", resp.GetMessage())
w.Write([]byte(resp.GetMessage()))
}
I'm trying to implement a http forwarding server that supports domain blocking. I've tried
go io.Copy(dst, src)
go io.Copy(src, dst)
and it works like a charm on tcp forwarding. Then I've tried to do request line parsing with something similar to
go func(){
reader := io.TeeReader(src, dst)
textReader := textproto.NewReader(bufio.NewReader(reader))
requestLine, _ = textReader.ReadLine()
// ...
ioutil.ReadAll(reader)
}
It works fine, but I was getting worried about bad performance(with ioutil.ReadAll). So I've written the code below.
func (f *Forwarder) handle(src, dst net.Conn) {
defer dst.Close()
defer src.Close()
done := make(chan struct{})
go func() {
textReader := bufio.NewReader(src)
requestLine, _ = textReader.ReadString('\n')
// parse request line and apply domain blocking
dst.Write([]byte(requestLine))
io.Copy(dst, src)
done <- struct{}{}
}()
go func() {
textReader := bufio.NewReader(dst)
s.statusLine, _ = textReader.ReadString('\n')
src.Write([]byte(s.statusLine))
io.Copy(src, dst)
done <- struct{}{}
}()
<-done
<-done
}
Unfortunately, it doesn't work at all. Requests get to print out, but not for responses. I've stuck here and don't know what's wrong.
TCP forwarding is to realize that the tunnel proxy does not need to parse data. The reverse proxy can use the standard library.
The tunnel proxy is implemented to separate the http and https protocols. The client generally uses the tunnel to send https and sends the Connect method. Sending http is the Get method. For the https request service, only dail creates the connection tcp conversion, and the http request is implemented using a reverse proxy.
func(w http.ResponseWriter, r *http.Request) {
// check url host
if r.URL.Host != "" {
if r.Method == eudore.MethodConnect {
// tunnel proxy
conn, err := net.Dial("tcp", r.URL.Host)
if err != nil {
w.WriteHeader(502)
return
}
client, _, err := w.Hijack()
if err != nil {
w.WriteHeader(502)
conn.Close()
return
}
client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))
go func() {
io.Copy(client, conn)
client.Close()
conn.Close()
}()
go func() {
io.Copy(conn, client)
client.Close()
conn.Close()
}()
} else {
// reverse proxy
httputil.NewSingleHostReverseProxy(r.URL).ServeHTTP(w, r)
}
}
}
Implementing a reverse proxy will parse the client request, and the proxy will send the request to the target server.
Reverse proxy conversion request, not tested :
func(w http.ResponseWriter, r *http.Request) {
// set host
r.URL.Scheme = "http"
r.URL.Path = "example.com"
// send
resp,err := http.DefaultClient.Do(r)
if err != nil {
w.WriteHeader(502)
return
}
// write respsonse
defer resp.Body.Close()
w.WriteHeader(resp.StatusCode)
h := w.Header()
for k,v := range resp.Header {
h[k]=v
}
io.Copy(w, resp.Body)
}
However, the direct forwarding request does not process the hop-to-hop header. The hop-to-hop header is clearly stated in the rfc. The hop-to-hop header is the transmission information between two connections. For example, the client to the proxy and the proxy to the server are two. And the client to the server is end-to-end.
Please use the standard library directly for the reverse proxy, it has already handled the hop-to-hop header and Upgrade for you.
exmample NewSingleHostReverseProxy with filter:
package main
import (
"net/http"
"strings"
"net/http/httputil"
"net/url"
)
func main() {
addr, _ := url.Parse("http://localhost:8088")
proxy := httputil.NewSingleHostReverseProxy(addr)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/api/") {
proxy.ServeHTTP(w, r)
} else {
w.WriteHeader(404)
}
})
// Listen Server
}
I am trying to convert below nodejs code to Go. I have to establish keep alive http request to PouchDB server's _changes?feed=continuous. However, I'm not able to achieve it in Go.
var http = require('http')
var agent = new http.Agent({
keepAlive: true
});
var options = {
host: 'localhost',
port: '3030',
method: 'GET',
path: '/downloads/_changes?feed=continuous&include_docs=true',
agent
};
var req = http.request(options, function(response) {
response.on('data', function(data) {
let val = data.toString()
if(val == '\n')
console.log('newline')
else {
console.log(JSON.parse(val))
//to close the connection
//agent.destroy()
}
});
response.on('end', function() {
// Data received completely.
console.log('end');
});
response.on('error', function(err) {
console.log(err)
})
});
req.end();
Below is the Go code
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
result, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
I am getting status 200 Ok, but no data gets printed, its stuck. On the other hand if I use longpoll option ie. http://localhost:3030/downloads/_changes?feed=longpoll then I am receiving data.
Your code is working "as expected" and what you wrote in Go is not equivalent to code shown in Node.js. Go code blocks on ioutil.ReadAll(resp.Body) because connection is kept open by CouchDB server. Once server closes the connection your client code will print out result as ioutil.ReadAll() will be able to read all data down to EOF.
From CouchDB documentation about continuous feed:
A continuous feed stays open and connected to the database until explicitly closed and changes are sent to the client as they happen, i.e. in near real-time. As with the longpoll feed type you can set both the timeout and heartbeat intervals to ensure that the connection is kept open for new changes and updates.
You can try experiment and add &timeout=1 to URL which will force CouchDB to close connection after 1s. Your Go code then should print the whole response.
Node.js code works differently, event data handler is called every time server sends some data. If you want to achieve same and process partial updates as they come (before connection is closed) you cannot use ioutil.ReadAll() as that waits for EOF (and thus blocks in your case) but something like resp.Body.Read() to process partial buffers. Here is very simplified snippet of code that demonstrates that and should give you basic idea:
package main
import (
"fmt"
"net/http"
"net/url"
"strings"
)
func main() {
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
defer resp.Body.Close()
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
buf := make([]byte, 1024)
for {
l, err := resp.Body.Read(buf)
if l == 0 && err != nil {
break // this is super simplified
}
// here you can send off data to e.g. channel or start
// handler goroutine...
fmt.Printf("%s", buf[:l])
}
fmt.Println()
}
In real world application you probably want to make sure your buf holds something that looks like a valid message and then pass it to channel or handler goroutine for further processing.
Finally, I was able to resolve the issue. The issue was related to DisableCompression flag. https://github.com/golang/go/issues/16488 this issue gave me some hint.
By setting DisableCompression: true fixed the issue.
client := &http.Client{Transport: &http.Transport{
DisableCompression: true,
}}
I am assuming client := &http.Client{} sends DisableCompression : false by default and pouchdb server is sending compressed json, Hence received data was compressed and resp.Body.Read was not able to read.
I am building a web server that must accept HTTP requests from a client, but must also accept requests over a raw TCP socket from peers. Since HTTP runs over TCP, I am trying to route the HTTP requests by the TCP server rather than running two separate services.
Is there an easy way to read in the data with net.Conn.Read(), determine if it is an HTTP GET/POST request and pass it off to the built in HTTP handler or Gorilla mux? Right now my code looks like this and I am building the http routing logic myself:
func ListenConn() {
listen, _ := net.Listen("tcp", ":8080")
defer listen.Close()
for {
conn, err := listen.Accept()
if err != nil {
logger.Println("listener.go", "ListenConn", err.Error())
}
go HandleConn(conn)
}
}
func HandleConn(conn net.Conn) {
defer conn.Close()
// determines if it is an http request
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
ln := scanner.Bytes()
fmt.Println(ln)
if strings.Fields(string(ln))[0] == "GET" {
http.GetRegistrationCode(conn, strings.Fields(string(ln))[1])
return
}
... raw tcp handler code
}
}
It is not a good idea to mix HTTP and raw TCP traffic.
Think about all firewalls and routers between your application and clients. They all designed to enable safe HTTP(s) delivery. What they will do with your tcp traffic coming to the same port as valid HTTP?
As a solution you can split your traffic to two different ports in the same application.
With ports separation you can route your HTTP and TCP traffic independently and configure appropriate network security for every channel.
Sample code to listen for 2 different ports:
package main
import (
"fmt"
"net"
"net/http"
"os"
)
type httpHandler struct {
}
func (m *httpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
fmt.Println("HTTP request")
}
func main() {
// http
go func() {
http.ListenAndServe(":8001", &httpHandler{})
}()
// tcp
l, err := net.Listen("tcp", "localhost:8002")
if err != nil {
fmt.Println("Error listening:", err.Error())
os.Exit(1)
}
defer l.Close()
for {
conn, err := l.Accept()
if err != nil {
fmt.Println("Error accepting: ", err.Error())
os.Exit(1)
}
go handleRequest(conn)
}
}
// Handles incoming requests.
func handleRequest(conn net.Conn) {
// read/write from connection
fmt.Println("TCP connection")
conn.Close()
}
open http://localhost:8001/ in browser and run command line echo -n "test" | nc localhost 8002 to test listeners
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/