What I try to achieve:
I want to recieve Http request from client then hijack the connection to monitor it on server side (Checking health of connection). Also I want to send Http response on that hijacked connection. In that order: recieve Http request, get request body, hijack connection, return response to client, monitor connection health on server side.
What I've already achieved:
Here is the code of http request handler:
func (c *HandlerCtx) HijackHandler(w http.ResponseWriter, req *http.Request){
// Fetching request body skipped
// Hijacking the connection
h, _ := w.(http.Hijacker)
conn, br, err := h.Hijack()
if err != nil {
fmt.Println(err.Error())
return
}
responseBody := "Http response from hijacked connection"
hr := http.Response{
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header, 0),
Body: ioutil.NopCloser(bytes.NewBufferString(responseBody)),
ContentLength: int64(len(responseBody)),
TransferEncoding: nil,
Close: false,
Uncompressed: false,
Trailer: nil,
Request: req,
TLS: nil,
}
// Writing response
err = hr.Write(br)
if err != nil{
fmt.Println(err.Error())
return
}
// Sending EOF to allow io.ReadAll(resp.Body) without blocking
if v, ok := conn.(interface{ CloseWrite() error }); ok {
err = v.CloseWrite()
if err != nil {
fmt.Println(err.Error())
}
}
// Monitor connection health
}
This is the client code:
func main(){
// Body skipped for testing purposes
resp, err := http.Post("http://127.0.0.1:8085/hello", "application/json", nil)
if err != nil{
fmt.Println(err.Error())
return
}
b, err := io.ReadAll(resp.Body)
if err != nil{
fmt.Println(err.Error())
return
}
}
Now I recieve unexpected EOF from client after calling err = v.CloseWrite() but when I don't CloseWrite the client code stuck on io.ReadAll(resp.Body)
Is there any way to force client to read that http response? Please help me find solution.
Related
Problem Statement
I have a client (which dials to the server) and server (that listens for incoming requests) written in golang and with the RPC calls defined. I am trying to initiate an HTTP request on the server side which would in turn execute the RPC call for streaming and send a JSON response back to the user
Challenge
I was able to handle both grpc and HTTP requests on different ports but having issues with passing parameters from the HTTP request onto the RPC call on the server side
Server Code
log.Println("Listening for connections from client ........")
lis, err := net.Listen("tcp", ":9000")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := testApi.Server{}
grpcServer := grpc.NewServer()
testApi.RegisterTestApiServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %s", err)
}
func main() {
go runGrpc()
log.Printf("*------ Waiting for requests from users ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
test_id, _ := strconv.Atoi(vars["test_id"])
log.Println("Test id request from user : ", test_id)
func (s * Server) ConnectAndStream(channelStream TestApiService_ConnectAndStreamServer) error {
// Question: This Id has to come from http request above- test_id
var id int32 = 1234566
// id := a.ConnectAndExchange
log.Println("Id from sam user ", id)
// var id int32 = 1234566
for i := 1; i <= 2; i++ {
id += 1
log.Println("Speed Server is sending data : ", id)
channelStream.Send(&Input{Id: id})
}
for i := 1; i <= 2; i++ {
log.Println("now time to receive")
client_response, err := channelStream.Recv()
log.Println("Response from samd client : ", client_response.Id)
if err != nil {
log.Println("Error while receiving from samd : ", err)
}
}
return nil
}
I am stuck with being able to pass the test_id from the curl request to the RPC call as above. Any input is greatly appreciated
Note
Client - Dials in and connects to the server and starts receiving and sending data (bi-directional streaming)
Both the Http and GRPC client are part of the same server application. So why call the RPC method from the Http handler? The Http handler should have access to the same backend functionality.
Your question is slightly unclear but if you are trying to have your client establish a GRPC connection to the server via the HTTP handler this will not work. The GRPC connection established in this situation is between the server and its self.
Edit - thanks for the clarification. Now I understand better the flow that you are trying to achieve. Your http handler method can make the outgoing grpc call to the server and return the response back via the http.ResponseWriter
For simplicity I have used the hello world example on https://github.com/grpc/grpc-go/tree/master/examples/helloworld
Running the code sample below and hitting http://localhost:1000/exchangeId/Test will show the output
Starting
*------ Waiting for http requests from users on port 1000 ------*
server listening at 127.0.0.1:1001
Test id request from user : Test
Server Received: Test
Greeting: Hello Test
Code sample:
import (
"context"
"log"
"net"
"net/http"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
"github.com/gorilla/mux"
)
var (
grpcserver = "localhost:1001"
)
func main() {
log.Print("Starting")
go StartGrpcServer()
log.Printf("*------ Waiting for http requests from users on port 1000 ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":1000", router))
}
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Server Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
func StartGrpcServer() {
lis, err := net.Listen("tcp", grpcserver)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
test_id := vars["test_id"]
log.Println("Test id request from user : ", test_id)
// Set up a connection to the server.
conn, err := grpc.Dial(grpcserver, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
resp, err := c.SayHello(ctx, &pb.HelloRequest{Name: test_id})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", resp.GetMessage())
w.Write([]byte(resp.GetMessage()))
}
Here's what I have right now:
func (client Client) PressButton(i int, k int, msg Message) error {
if client.Token == "" {
return fmt.Errorf("no token")
}
i--
k--
url := "https://discord.com/api/v9/interactions"
time.Sleep(2 * time.Second)
data := map[string]interface{}{"component_type": msg.Components[i].Buttons[k].Type, "custom_id": msg.Components[i].Buttons[k].CustomID, "hash": msg.Components[i].Buttons[k].Hash}
values := map[string]interface{}{"application_id": "270904126974590976", "channel_id": msg.ChannelID, "type": "3", "data": data, "guild_id": msg.GuildID, "message_flags": 0, "message_id": msg.ID,}
json_data, err := json.Marshal(values)
if err != nil {
return fmt.Errorf("error while encoding button click as json: %v", err)
}
req, err := http.NewRequest("POST", url, strings.NewReader(string(json_data)))
if err != nil {
return fmt.Errorf("error while creating http request: %v", err)
}
req.Header.Set("authorization", client.Token)
req.Header.Set("Content-Type", "application/json")
httpClient := &http.Client{}
resp, err := httpClient.Do(req)
if err != nil {
return fmt.Errorf("error while sending http request: %v", err)
}
if resp.StatusCode != 204 {
switch resp.StatusCode {
case http.StatusUnauthorized:
return ErrUnauthorized
case http.StatusForbidden:
return ErrForbidden
case http.StatusNotFound:
return ErrNotFound
case http.StatusTooManyRequests:
return ErrTooManyRequests
case http.StatusInternalServerError:
return ErrIntervalServer
default:
return fmt.Errorf("unexpected status code while clicking button: %v", resp.StatusCode)
}
}
return nil
}
This works perfectly 90% of the times. But sometimes I get an error 400. I looked it up, I tried every recommended solution, I have confirmed that the payload and the URL are absolutely right moreover it wouldn't work most of the times if they weren't. I've also tried changing the headers among other things.
Here's an example payload that was sent (Before Marshaling) :
map[application_id:12345 channel_id:872747211551801384 data:map[component_type:2 custom_id:c34117ee-a332-4d7c-888f-47081e40092f:0 hash:]
(Changed the application id)
Response status is 400, here's the response body:
{"code": 50035, "errors": {"data": {"_errors": [{"code": "COMPONENT_VALIDATION_FAILED", "message": "Component validation failed"}]}}, "message": "Invalid Form Body"}
I've absolutely scavanged the internet and coding servers on discord to understand why this is happening.
The hash is always an empty field now, earlier it used to have a value for my use-case. I just left it in so it's more generalized.
I'm implementing a HTTP long poll client in Go for receiving updates from a server.
What's the difference in normal HTTP client and Long-poll client in Go?
It is just the timeout?
My code looks like this:
request, err := http.NewRequest(get, url, nil)
if err != nil {
fmt.Print("Error")
return
}
request.Header.Set(consulTokenHeader, aclToken)
httpClient := &http.Client{} //introduce timeout?
response, err := httpClient.Do(request)
if err != nil {
log.Fatal(err)
}
I am trying to connect to a Host using Random TLS Fingerprinting. I am using https://github.com/refraction-networking/utls (see my issue i created on https://github.com/refraction-networking/utls/issues/42)
My issue is now, how can i utilize a HTTP or SOCKS5 Proxy while opening that connection?
The Code im using right now is:
package main
import (
"bufio"
"fmt"
"log"
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/refraction-networking/utls"
)
var (
dialTimeout = time.Duration(15) * time.Second
)
var requestHostname = "google.com"
var requestAddr = "172.217.22.110:443"
// this example generates a randomized fingeprint, then re-uses it in a follow-up connection
func HttpGetConsistentRandomized(hostname string, addr , uri string) (*http.Response, error) {
config := tls.Config{ServerName: hostname}
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn := tls.UClient(tcpConn, &config, tls.HelloRandomized)
defer uTlsConn.Close()
err = uTlsConn.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
uTlsConn.Close()
// At this point uTlsConn.ClientHelloID holds a seed that was used to generate
// randomized fingerprint. Now we can establish second connection with same fp
tcpConn2, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn2 := tls.UClient(tcpConn2, &config, uTlsConn.ClientHelloID)
defer uTlsConn2.Close()
err = uTlsConn2.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
return httpGetOverConn(uTlsConn2, uTlsConn2.HandshakeState.ServerHello.AlpnProtocol, uri)
}
func main() {
var response *http.Response
var err error
response, err = HttpGetConsistentRandomized(requestHostname, requestAddr, "/2.0/ssocookie")
if err != nil {
fmt.Printf("#> HttpGetConsistentRandomized() failed: %+v\n", err)
} else {
//fmt.Printf("#> HttpGetConsistentRandomized() response: %+s\n", httputil.DumpResponse(response,true))
dump, err := httputil.DumpResponse(response, true)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+s\n", dump)
}
return
}
func httpGetOverConn(conn net.Conn, alpn string, uri string) (*http.Response, error) {
req := &http.Request{
Method: "GET",
URL: &url.URL{Host: "www." + requestHostname + uri},
Header: make(http.Header),
Host: "www." + requestHostname,
}
req.Proto = "HTTP/1.1"
req.ProtoMajor = 1
req.ProtoMinor = 1
err := req.Write(conn)
if err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(conn), req)
}
As Steffen said, you have to create a proxy dialer first, dial the proxy to create a net.Conn, then use that net.Conn when creating the uTLS Client, before handshaking. For brevity's sake, your custom dialTLS function would look something like:
import (
"crypto/tls"
"net"
"net/url"
"github.com/magisterquis/connectproxy"
"golang.org/x/net/proxy"
utls "github.com/refraction-networking/utls"
)
var proxyString = "http://127.0.0.1:8080"
dialTLS := func(network, addr string, _ *tls.Config) (net.Conn, error) {
proxyURI, _ := url.Parse(proxyString)
switch proxyURI.Scheme {
case "socks5":
proxyDialer, err = proxy.SOCKS5("tcp", proxyString, nil, proxy.Direct)
case "http":
proxyDialer, err = connectproxy.New(proxyURI, proxy.Direct)
}
conn, err := proxyDialer.Dial("tcp", addr)
uconn := utls.UClient(conn, cfg, &utls.HelloRandomizedALPN)
...
}
Two suggestions:
Use the "connectproxy" module referenced above if you intend to tunnel through a HTTP CONNECT proxy.
Make life easier for yourself and take a look at the Meek pluggable transport source for Tor. There's a 'utls.go' module which takes care of everything for you, including setting up either a http or http2 transport depending on the negotiated ALPN protocol. It only supports SOCKS but you could easily adapt it to handle HTTP proxies.
A HTTP proxy and SOCKS proxy work be having some initial proxy specific handshake after the TCP connect. After this handshake is done they provide a normal TCP socket which then can be used for doing the TLS handshake etc. Thus, all you need is to replace your
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
with a proxy specific method to setup the TCP connection. This can be done by using SOCKS5 in x/net/proxy to create the appropriate Dialer or similar using the HTTP CONNECT method is done in connectproxy.
I'm using httputil.ReverseProxy with an http.RoundTripper of my own implementation that uses an ssh.Channel as a transport. My RoundTrip method looks approximately like this:
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
// defer ch.Close()
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
return http.ReadResponse(bufio.NewReader(ch), req)
}
func (c SSHConnection) GetChannel() (ssh.Channel, error) {
ch, req, err := c.Conn.OpenChannel("forwarded-tcpip", msg)
if err != nil {
return nil, err
}
go ssh.DiscardRequests(req)
return ch, nil
}
Notice the commented-out defer ch.Close(). Initially I naively closed the connection here, but the response body would sometimes be empty, due to a race between the HTTP proxy's reading of the body and this closing of the SSH channel.
Assuming, for now, that I don't care to do keep-alive, when can I close the ssh.Channel? If I don't, every request starts a new goroutine (because of go ssh.DiscardRequests(req)), so I leak a goroutine on every HTTP requests until the underlying SSH connection is closed.
An http.RoundTripper should not close the connection until after the response body has been fully consumed, or at the request of the server.
The simplest option is to fully buffer the response and close the connection immediately. In some cases this may actually be the most efficient, if the traffic mostly consists of small, independent requests.
The next option is to hook the closing of the response body to close the channel.
type Body struct {
io.ReadCloser
channel ssh.Channel
}
func (b *Body) Close() error {
b.channel.Close()
return b.ReadCloser.Close()
}
func (c SSHConnection) RoundTrip(req *http.Request) (*http.Response, error) {
ch, err := c.GetChannel()
if err != nil {
return nil, errors.New("couldn't open forwarded-tcpip channel: " + err.Error())
}
err = req.Write(ch)
if err != nil {
return nil, errors.New("couldn't send request: " + err.Error())
}
resp, err := http.ReadResponse(bufio.NewReader(ch), req)
if err != nil {
ch.Close()
return nil, err
}
resp.Body = &Body{
ReadCloser: resp.Body,
channel: ch,
}
return resp, err
}
Lastly, to make the most efficient use of the ssh channels, you could use an existing Transport with a net.Dialer which makes the ssh connection, and wraps the channel in a net.Conn interface.