I'm currently attempting to debug an issue with issuing a HTTP POST via GoLang 1.6.
The connection is failing with an 'EOF' error.
The test client code: https://gist.github.com/fatmcgav/35d1a4fbd74c7208f445c487f756a5e1
package main
import (
"bytes"
"crypto/tls"
"encoding/json"
"fmt"
"net/http"
"net/http/httputil"
"os"
)
func main() {
const body = "{\"auth\":" +
"{\"identity\":" +
"{\"methods\":[\"password\"]," +
"\"password\":" +
"{\"user\":{\"name\":\"xxx\"," +
"\"password\":\"xxx\"," +
"\"domain\":{\"name\":\"xxx_domain\"}" +
"}" +
"}" +
"},\"scope\":{\"project\":" +
"{\"domain\": " +
"{\"name\":\"xxx_domain\"},\"name\":\"xxx\"}}}}"
// Unmarshall JSON
var m map[string]interface{}
json.Unmarshal([]byte(body), &m)
// fmt.Println("%v", m)
var req *http.Request
var resp *http.Response
var dump []byte
var err error
client := &http.Client{}
url := "https://identity.xxx/v3/auth/tokens"
fmt.Println("Requesting auth token against URL: %s", url)
rendered, err := json.Marshal(m)
if err != nil {
fmt.Println("Error marshalling body: %q", err)
os.Exit(2)
}
req, err = http.NewRequest("POST", url, bytes.NewReader(rendered))
if err != nil {
fmt.Println("Got an error: %q", err)
os.Exit(2)
}
// Need to Close connection
req.Close = true
fmt.Println("Setting Insecure TLS mode")
// Configure custom TLS settings.
config := &tls.Config{InsecureSkipVerify: true}
transport := &http.Transport{TLSClientConfig: config,
DisableKeepAlives: true,
DisableCompression: true,
}
client.Transport = transport
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
dump, err = httputil.DumpRequestOut(req, true)
fmt.Printf("Outgoing requst = \n%q\n", dump)
resp, err = client.Do(req)
if err != nil {
fmt.Println("Error encountered: %q", err)
}
dump, err = httputil.DumpResponse(resp, true)
fmt.Printf("Response = \n%q\n", dump)
}
and resultant output is here:
~/golang/scripts go build req-test.go && ./req-test
Requesting auth token against URL: %s https://identity.xxx/v3/auth/tokens
Setting Insecure TLS mode
Outgoing requst =
"POST /v3/auth/tokens HTTP/1.1\r\nHost: identity.xxx\r\nUser-Agent: Go-http-client/1.1\r\nConnection: close\r\nContent-Length: 262\r\nAccept: application/json\r\nContent-Type: application/json\r\nAccept-Encoding: gzip\r\n\r\n{\"auth\":{\"identity\":{\"methods\":[\"password\"],\"password\":{\"user\":{\"domain\":{\"name\":\"BusinessSupport_domain\"},\"name\":\"xxx\",\"password\":\"xxx\"}}},\"scope\":{\"project\":{\"domain\":{\"name\":\"xxx_domain\"},\"name\":\"xxx\"}}}}"
Error encountered: %q Post https://identity.xxx/v3/auth/tokens: EOF
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x13659e]
goroutine 1 [running]:
panic(0x338980, 0xc82000a170)
/usr/local/opt/go/libexec/src/runtime/panic.go:464 +0x3e6
net/http/httputil.DumpResponse(0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/opt/go/libexec/src/net/http/httputil/dump.go:285 +0x42e
main.main()
/Users/gavinw/golang/scripts/req-test.go:77 +0xdc
For obvious reasons, I've anonymised some of the details..
Any pointers on what I'm doing wrong..
Cheers
Gavin
Related
I have a dockerfile which runs tor -
FROM alpine:edge
RUN apk update && apk add tor
EXPOSE 9050
USER tor
CMD ["/usr/bin/tor"]
and ran it using command - docker run --name tor -p 11000:9050 tor
and checked connection using - telnet 127.0.0.1 11000 and it showed connected
Now I want to use tor as proxy while any request from go program. I tried -
package main
import (
"fmt"
"net/http"
"net/url"
"time"
)
func main() {
proxyUrl, err := url.Parse("socks5://127.0.0.1:11000")
if err != nil {
// TODO handle me
panic(err)
}
cl := http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
},
Timeout: 18000 * time.Millisecond,
}
resp, err := cl.Get("http://google.com")
if err != nil {
// TODO handle me
panic(err)
}
// TODO work with the response
fmt.Println(resp)
}
But running this program threw error -
panic: Get http://google.com: socks connect tcp 127.0.0.1:11000->google.com:80: read tcp 127.0.0.1:59630->127.0.0.1:11000: read: connection reset by peer
goroutine 1 [running]: <stacktrace>
exit status 2
I tried other approaches also, notably mentioned here and here but kept getting same error - read: connection reset by peer
Please help which part is incorrect here.
Thanks.
--------------------another approach that I tried ----------------
As mentioned in one of the links, I tried this code also -
const (
PROXY_ADDR = "127.0.0.1:11000"
URL = "http://facebookcorewwwi.onion"
)
func main() {
// create a socks5 dialer
dialer, err := proxy.SOCKS5("tcp", PROXY_ADDR, nil, proxy.Direct)
if err != nil {
fmt.Fprintln(os.Stderr, "can't connect to the proxy:", err)
os.Exit(1)
}
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
// do anything with ctx
return dialer.Dial(network, address)
}
// setup a http client
httpTransport := &http.Transport{
DialContext: dialContext,
}
httpClient := &http.Client{Transport: httpTransport}
// create a request
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
fmt.Fprintln(os.Stderr, "can't create request:", err)
os.Exit(2)
}
resp, err := httpClient.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot make get request: ", err)
os.Exit(2)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot read response body: ", err)
os.Exit(2)
}
fmt.Println("received response -> ", body)
}
but received error -
cannot make get request: Get http://facebookcorewwwi.onion: socks connect tcp 127.0.0.1:11000->facebookcorewwwi.onion:80: read tcp 127.0.0.1:59826->127.0.0.1:11000: read: connection reset by peer
exit status 2
Any help is appreciable.
After making sure tor is working properly on port 9050.
Try the following curl command to ensure tor is working properly.
curl --socks5 localhost:9050 --socks5-hostname localhost:9050 -s https://wtfismyip.com/json
Can you try this
package main
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/http"
"golang.org/x/net/proxy"
)
func main() {
proxyUrl := "127.0.0.1:9050"
dialer, err := proxy.SOCKS5("tcp", proxyUrl, nil, proxy.Direct)
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
return dialer.Dial(network, address)
}
transport := &http.Transport{DialContext: dialContext,
DisableKeepAlives: true}
cl := &http.Client{Transport: transport}
resp, err := cl.Get("https://wtfismyip.com/json")
if err != nil {
// TODO handle me
panic(err)
}
body, err := ioutil.ReadAll(resp.Body)
// TODO work with the response
if err != nil {
fmt.Println("body read failed")
}
fmt.Println(string(body))
}
As suggested in the answer/comment above, main problem was socks5 connection to dockerized tor container.
Found the solution here, i just had to modify dockerfile like -
FROM alpine:edge
RUN apk update && apk add tor
RUN cp /etc/tor/torrc.sample /etc/tor/torrc && echo "SocksPort 0.0.0.0:9050" > /etc/tor/torrc
EXPOSE 9050
CMD ["/usr/bin/tor"]
Other code pieces are working fine.
I am trying to connect to a Host using Random TLS Fingerprinting. I am using https://github.com/refraction-networking/utls (see my issue i created on https://github.com/refraction-networking/utls/issues/42)
My issue is now, how can i utilize a HTTP or SOCKS5 Proxy while opening that connection?
The Code im using right now is:
package main
import (
"bufio"
"fmt"
"log"
"net"
"net/http"
"net/http/httputil"
"net/url"
"time"
"github.com/refraction-networking/utls"
)
var (
dialTimeout = time.Duration(15) * time.Second
)
var requestHostname = "google.com"
var requestAddr = "172.217.22.110:443"
// this example generates a randomized fingeprint, then re-uses it in a follow-up connection
func HttpGetConsistentRandomized(hostname string, addr , uri string) (*http.Response, error) {
config := tls.Config{ServerName: hostname}
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn := tls.UClient(tcpConn, &config, tls.HelloRandomized)
defer uTlsConn.Close()
err = uTlsConn.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
uTlsConn.Close()
// At this point uTlsConn.ClientHelloID holds a seed that was used to generate
// randomized fingerprint. Now we can establish second connection with same fp
tcpConn2, err := net.DialTimeout("tcp", addr, dialTimeout)
if err != nil {
return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
}
uTlsConn2 := tls.UClient(tcpConn2, &config, uTlsConn.ClientHelloID)
defer uTlsConn2.Close()
err = uTlsConn2.Handshake()
if err != nil {
return nil, fmt.Errorf("uTlsConn.Handshake() error: %+v", err)
}
return httpGetOverConn(uTlsConn2, uTlsConn2.HandshakeState.ServerHello.AlpnProtocol, uri)
}
func main() {
var response *http.Response
var err error
response, err = HttpGetConsistentRandomized(requestHostname, requestAddr, "/2.0/ssocookie")
if err != nil {
fmt.Printf("#> HttpGetConsistentRandomized() failed: %+v\n", err)
} else {
//fmt.Printf("#> HttpGetConsistentRandomized() response: %+s\n", httputil.DumpResponse(response,true))
dump, err := httputil.DumpResponse(response, true)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%+s\n", dump)
}
return
}
func httpGetOverConn(conn net.Conn, alpn string, uri string) (*http.Response, error) {
req := &http.Request{
Method: "GET",
URL: &url.URL{Host: "www." + requestHostname + uri},
Header: make(http.Header),
Host: "www." + requestHostname,
}
req.Proto = "HTTP/1.1"
req.ProtoMajor = 1
req.ProtoMinor = 1
err := req.Write(conn)
if err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(conn), req)
}
As Steffen said, you have to create a proxy dialer first, dial the proxy to create a net.Conn, then use that net.Conn when creating the uTLS Client, before handshaking. For brevity's sake, your custom dialTLS function would look something like:
import (
"crypto/tls"
"net"
"net/url"
"github.com/magisterquis/connectproxy"
"golang.org/x/net/proxy"
utls "github.com/refraction-networking/utls"
)
var proxyString = "http://127.0.0.1:8080"
dialTLS := func(network, addr string, _ *tls.Config) (net.Conn, error) {
proxyURI, _ := url.Parse(proxyString)
switch proxyURI.Scheme {
case "socks5":
proxyDialer, err = proxy.SOCKS5("tcp", proxyString, nil, proxy.Direct)
case "http":
proxyDialer, err = connectproxy.New(proxyURI, proxy.Direct)
}
conn, err := proxyDialer.Dial("tcp", addr)
uconn := utls.UClient(conn, cfg, &utls.HelloRandomizedALPN)
...
}
Two suggestions:
Use the "connectproxy" module referenced above if you intend to tunnel through a HTTP CONNECT proxy.
Make life easier for yourself and take a look at the Meek pluggable transport source for Tor. There's a 'utls.go' module which takes care of everything for you, including setting up either a http or http2 transport depending on the negotiated ALPN protocol. It only supports SOCKS but you could easily adapt it to handle HTTP proxies.
A HTTP proxy and SOCKS proxy work be having some initial proxy specific handshake after the TCP connect. After this handshake is done they provide a normal TCP socket which then can be used for doing the TLS handshake etc. Thus, all you need is to replace your
tcpConn, err := net.DialTimeout("tcp", addr, dialTimeout)
with a proxy specific method to setup the TCP connection. This can be done by using SOCKS5 in x/net/proxy to create the appropriate Dialer or similar using the HTTP CONNECT method is done in connectproxy.
I want to set the Request struct field Response by reference in the function request in order to use it further below in the main function. Unfortunately, I get the following error:
{Status: StatusCode:0 Proto: ProtoMajor:0 ProtoMinor:0 Header:map[] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:<nil> TLS:<nil>}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4922d1]
goroutine 1 [running]:
io.copyBuffer(0x72dbc0, 0xc0000a8008, 0x0, 0x0, 0xc000196000, 0x8000, 0x8000, 0xb9, 0x0, 0x0)
/usr/lib/go/src/io/io.go:402 +0x101
io.Copy(...)
/usr/lib/go/src/io/io.go:364
main.main()
/home/y/code/scmc/foo.go:49 +0x20d
exit status 2
The code looks as follows:
package main
import (
"fmt"
"io"
"net/http"
"os"
)
type Request struct {
Method string
Url string
Reader io.Reader
Response *http.Response
}
func request(r Request) error {
request, err := http.NewRequest(r.Method, r.Url, r.Reader)
if err != nil {
return err
}
client := &http.Client{}
response, err := client.Do(request)
if err != nil {
return err
}
if r.Response != nil {
r.Response = response
}
return nil
}
func main() {
var r http.Response
if err := request(Request{
Method: "GET",
Url: "http://google.com",
Response: &r,
}); err != nil {
panic(err)
}
fmt.Printf("%+v\n", r)
if _, err := io.Copy(os.Stdout, r.Body); err != nil {
panic(err)
}
}
What am I doing wrong?
client.Do() returns a pointer: *http.Response. If you want to "transfer" a *http.Response pointer value out of your function, you need a pointer value to this type which you may set. That pointer value has to be of type **http.Response (note: pointer to pointer):
type Request struct {
Method string
Url string
Reader io.Reader
Response **http.Response
}
Inside your request() function you need to set the pointed value:
if r.Response != nil {
*r.Response = response
}
And when calling request():
var r *http.Response
if err := request(Request{
Method: "GET",
Url: "http://google.com",
Response: &r,
}); err != nil {
panic(err)
}
For analogy:
If you'd want to transfer out an int value:
func do(i *int) {
*i= 10
}
// Calling it:
var i int
do(&i)
To transfer out an *int value:
func do(i **int) {
x := 10
*i = &x
}
// Calling it:
var i *int
do(&i)
trying to send more than one of the same HTTP request generated by http.ReadRequest, I'm getting the following error:
Post http://192.168.x.x:8000/dir1: http: invalid Read on closed Body
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5ec4cc]
My code takes a file, and puts it in a bufio.NewReader. It then uses http.ReadRequest on what it had read, which is a HTTP POST request. finally - it opens another file, "wordlist.txt", which contains directories - and iterates over the original HTTP URI each time with a different directory from the list.
For some reason - the second requests causes to program to error out (http: invalid Read on closed Body). Heres a working example (just change the paths to yours):
main.go
package main
import (
"bufio"
"fmt"
"log"
"net/http"
"net/url"
"os"
)
func main() {
//Read HTTP request from text file
rdr, err := os.Open("/opt/gohttp/tests/req2.txt")
if err != nil {
log.Println(err)
}
//Parse to a web request to work with using ReadRequest
req, err := http.ReadRequest(bufio.NewReader(rdr))
if err != nil {
log.Println(err)
}
// fix for "requestURI can't be sent in client requests"
req.RequestURI = ""
//Open wordlist
file, err := os.Open("/opt/gohttp/tests/wordlist.txt")
if err != nil {
log.Println(err)
}
defer file.Close()
//Scan wordlist line by line and append each to the directory
scanner := bufio.NewScanner(file)
for scanner.Scan() {
var i string = scanner.Text()
fmt.Println(i)
//this part appends the wordlist word to the request sent e.g. /dir1
u, err := url.Parse("http://" + req.Host + "/" + i) //todo: https? other protocols?
if err != nil {
log.Println("scanner parse error: ", err)
}
req.URL = u
//Send the request
fmt.Println("Sending request.." + "\n")
client := &http.Client{}
response, err := client.Do(req)
if err != nil {
log.Println(err) // invalid read on closed body error
}
fmt.Println(*response)
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
}
example HTTP request file
POST /hi HTTP/1.1
Host: 192.168.x.x:8000
Content-Length: 41
{"email":"aa","password":"secret"}
example wordlist file
0
1
dir1
dir2
dir3
application output
go run tests/main.go
0
Sending request..
{404 Not Found 404 HTTP/1.1 1 1...}
1
Sending request..
2019/01/28 04:27:52 Post http://192.168.x.x:8000/1: http: invalid Read on closed Body
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5ec4cc]
goroutine 1 [running]:
main.main()
/opt/gohttp/tests/main.go:53 +0x35c
exit status 2
Please note that the first HTTP request is sent successfuly, and only the second one returns the error above. How can I fix this?
I'm trying to detect sending failures by inspecting the error returned by golang TCPConn.Write, but it's nil. I also tried using TCPConn.SetWriteDeadline without success.
That's how things happen:
the server starts
a client connects
the server sends a message and the client receives it
the client shuts down
the server sends one more message: no error
the server sends the third message: only now the error appears
Question: why only the second message to a non-existing client results in an error? How should the case be handled properly?
The code follows:
package main
import (
"net"
"os"
"bufio"
"fmt"
"time"
)
func AcceptConnections(listener net.Listener, console <- chan string) {
msg := ""
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
for {
if msg == "" {
msg = <- console
fmt.Printf("read from console: %s", msg)
}
err = conn.SetWriteDeadline(time.Now().Add(time.Second))
if err != nil {
fmt.Printf("SetWriteDeadline failed: %v\n", err)
}
_, err = conn.Write([]byte(msg))
if err != nil {
// expecting an error after sending a message
// to a non-existing client endpoint
fmt.Printf("failed sending a message to network: %v\n", err)
break
} else {
fmt.Printf("msg sent: %s", msg)
msg = ""
}
}
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
The server console looks like this:
listening on 127.0.0.1:6666
client connected
hi there!
read from console: hi there!
msg sent: hi there!
this one should fail
read from console: this one should fail
msg sent: this one should fail
this one actually fails
read from console: this one actually fails
failed sending a message to network: write tcp 127.0.0.1:51194: broken pipe
The client looks like this:
package main
import (
"net"
"os"
"io"
//"bufio"
//"fmt"
)
func cp(dst io.Writer, src io.Reader, errc chan<- error) {
// -reads from src and writes to dst
// -blocks until EOF
// -EOF is not an error
_, err := io.Copy(dst, src)
// push err to the channel when io.Copy returns
errc <- err
}
func StartCommunication(conn net.Conn) {
//create a channel for errors
errc := make(chan error)
//read connection and print to console
go cp(os.Stdout, conn, errc)
//read user input and write to connection
go cp(conn, os.Stdin, errc)
//wait until nil or an error arrives
err := <- errc
if err != nil {
println("cp error: ", err.Error())
}
}
func main() {
servAddr := "localhost:6666"
tcpAddr, err := net.ResolveTCPAddr("tcp", servAddr)
if err != nil {
println("ResolveTCPAddr failed:", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
println("net.DialTCP failed:", err.Error())
os.Exit(1)
}
defer conn.Close()
StartCommunication(conn)
}
EDIT: Following JimB's suggestion I came up with a working example. Messages don't get lost any more and are re-sent in a new connection. I'm not quite sure though how safe is it to use a shared variable (connWrap.IsFaulted) between different go routines.
package main
import (
"net"
"os"
"bufio"
"fmt"
)
type Connection struct {
IsFaulted bool
Conn net.Conn
}
func StartWritingToNetwork(connWrap * Connection, errChannel chan <- error, msgStack chan string) {
for {
msg := <- msgStack
if connWrap.IsFaulted {
//put it back for another connection
msgStack <- msg
return
}
_, err := connWrap.Conn.Write([]byte(msg))
if err != nil {
fmt.Printf("failed sending a message to network: %v\n", err)
connWrap.IsFaulted = true
msgStack <- msg
errChannel <- err
return
} else {
fmt.Printf("msg sent: %s", msg)
}
}
}
func StartReadingFromNetwork(connWrap * Connection, errChannel chan <- error){
network := bufio.NewReader(connWrap.Conn)
for (!connWrap.IsFaulted) {
line, err := network.ReadString('\n')
if err != nil {
fmt.Printf("failed reading from network: %v\n", err)
connWrap.IsFaulted = true
errChannel <- err
} else {
fmt.Printf("%s", line)
}
}
}
func AcceptConnections(listener net.Listener, console chan string) {
errChannel := make(chan error)
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
connWrap := Connection{false, conn}
go StartReadingFromNetwork(&connWrap, errChannel)
go StartWritingToNetwork(&connWrap, errChannel, console)
//block until an error occurs
<- errChannel
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
This isn't Go specific, and is a artifact of the underlying TCP socket showing through.
A decent diagram of the TCP termination steps is at the bottom of this page:
http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm
The simple version is that when the client closes its socket, it sends a FIN, and receives an ACK from the server. It then waits for the server to do the same. Instead of sending a FIN though, you're sending more data, which is discarded, and the client socket now assumes that any more data coming from you is invalid, so the next time you send you get an RST, which is what bubbles up into the error you see.
Going back to your program, you need to handle this somehow. Generally you can think of whomever is in charge of initiating a send, is also in charge of initiating termination, hence your server should assume that it can continue to send until it closes the connection, or encounters an error. If you need to more reliably detect the client closing, you need to have some sort of client response in the protocol. That way recv can be called on the socket and return 0, which alerts you to the closed connection.
In go, this will return an EOF error from the connection's Read method (or from within the Copy in your case). SetWriteDeadline doesn't work because a small write will go though and get dropped silently, or the client will eventually respond with an RST, giving you an error.