I am playing with the net package, and i want to make a simple proxy.
First i make a listener on localhost, then i dial the remote address
remote, err := net.Dial("tcp", "google.com:80")
if err != nil {
log.Fatal(err)
}
defer remote.Close()
fmt.Fprint(remote, "GET / HTTP/1.0\r\n\r\n")
How can i pipe the response to the browser? Or do i need to work with the default webserver and copy the response body? I really want to try it with net package or something
thx
To copy the connection from the remote is used 2 goroutines with io.Copy
func copyContent(from, to net.Conn, done chan bool) {
_, err := io.Copy(from, to)
if err != nil {
done <- true
}
done <- true
}
// in the main func
done := make(chan bool, 2)
go copyContent(conn, remote, done)
go copyContent(remote, conn, done)
<-done
<-done
Related
I am trying to convert below nodejs code to Go. I have to establish keep alive http request to PouchDB server's _changes?feed=continuous. However, I'm not able to achieve it in Go.
var http = require('http')
var agent = new http.Agent({
keepAlive: true
});
var options = {
host: 'localhost',
port: '3030',
method: 'GET',
path: '/downloads/_changes?feed=continuous&include_docs=true',
agent
};
var req = http.request(options, function(response) {
response.on('data', function(data) {
let val = data.toString()
if(val == '\n')
console.log('newline')
else {
console.log(JSON.parse(val))
//to close the connection
//agent.destroy()
}
});
response.on('end', function() {
// Data received completely.
console.log('end');
});
response.on('error', function(err) {
console.log(err)
})
});
req.end();
Below is the Go code
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
result, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
I am getting status 200 Ok, but no data gets printed, its stuck. On the other hand if I use longpoll option ie. http://localhost:3030/downloads/_changes?feed=longpoll then I am receiving data.
Your code is working "as expected" and what you wrote in Go is not equivalent to code shown in Node.js. Go code blocks on ioutil.ReadAll(resp.Body) because connection is kept open by CouchDB server. Once server closes the connection your client code will print out result as ioutil.ReadAll() will be able to read all data down to EOF.
From CouchDB documentation about continuous feed:
A continuous feed stays open and connected to the database until explicitly closed and changes are sent to the client as they happen, i.e. in near real-time. As with the longpoll feed type you can set both the timeout and heartbeat intervals to ensure that the connection is kept open for new changes and updates.
You can try experiment and add &timeout=1 to URL which will force CouchDB to close connection after 1s. Your Go code then should print the whole response.
Node.js code works differently, event data handler is called every time server sends some data. If you want to achieve same and process partial updates as they come (before connection is closed) you cannot use ioutil.ReadAll() as that waits for EOF (and thus blocks in your case) but something like resp.Body.Read() to process partial buffers. Here is very simplified snippet of code that demonstrates that and should give you basic idea:
package main
import (
"fmt"
"net/http"
"net/url"
"strings"
)
func main() {
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
defer resp.Body.Close()
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
buf := make([]byte, 1024)
for {
l, err := resp.Body.Read(buf)
if l == 0 && err != nil {
break // this is super simplified
}
// here you can send off data to e.g. channel or start
// handler goroutine...
fmt.Printf("%s", buf[:l])
}
fmt.Println()
}
In real world application you probably want to make sure your buf holds something that looks like a valid message and then pass it to channel or handler goroutine for further processing.
Finally, I was able to resolve the issue. The issue was related to DisableCompression flag. https://github.com/golang/go/issues/16488 this issue gave me some hint.
By setting DisableCompression: true fixed the issue.
client := &http.Client{Transport: &http.Transport{
DisableCompression: true,
}}
I am assuming client := &http.Client{} sends DisableCompression : false by default and pouchdb server is sending compressed json, Hence received data was compressed and resp.Body.Read was not able to read.
I'm writing a simple webserver in golang that gets/creates/edits/deletes a simple text file. I've written the function handlers and I'd like to test them by sending a request to the appropriate url and checking to see what happens. My code is as below:
func createHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
name := vars["name"]
body, _ := ioutil.ReadAll(r.Body)
fmt.Fprint(w, name)
ioutil.WriteFile(name, []byte(body), 0644)
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/textFiles/{name}", createHandler).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", r))
var url = "http://localhost:8080/textFiles/testFile.txt"
var text = []byte(`{"title":"this is an example."}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(text))
if err != nil {
panic(err)
}
client := &http.Client{}
client.Do(req)
}
Once this code is run, however, no new file is created. I've been googling but I can't find anything on this type of problem, where I'm sending a request to the server that I'm building within the same file. Help appreciated.
The client code is not executed. The callhttp.ListenAndServe(":8080", r) runs the server. The function only returns when there was an error running the server. If the function does return, then log.Fatal will exit the process.
One fix is to run the server in a goroutine. This will allow main goroutine to continue executing to the client code.
go func() {
log.Fatal(http.ListenAndServe(":8080", r))
}()
This may not fix the problem because there's no guarantee that server will run before the client makes the request. Fix this issue by creating the listening socket in the main function and running the server in a goroutine.
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatal(err)
}
go func() {
log.Fatal(http.Serve(ln, r))
}()
... client code as before
If the goal if this code is testing, then use httptest.Server. The example in the documentation show show to use the test server.
I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.
I work on a tcp server in golang ,recently I found a problem, when the client connected to the server, use netstat -nat|grep -i "55555" to check , it tells me the connection has established and Recv-Q has message , but Accept() need take about several seconds to return . I'm confused about it, could anyone tell me why or give any advice how to find the problem?
Here is my main code
tcpAddr, err := net.ResolveTCPAddr("tcp4", server.Host)
if err != nil {
log.Error(err.Error())
os.Exit(-1)
}
listener, err := net.ListenTCP("tcp", tcpAddr)
if err != nil {
log.Error(err.Error())
os.Exit(-1)
}
for {
conn, err := listener.Accept()
if err != nil {
continue
}
log.Debug("Accept a new connection")
go handleClient(conn)//deal the connection
}
P.S.
In handleClient, it's too long, I can't paste it here, only one more thing, I set tcp keep alive to the connection , use this package github.com/felixge/tcpkeepalive
kaConn, err := tcpkeepalive.EnableKeepAlive(connection)
if err != nil {
log.Debug("EnableKeepAlive err ", err)
}else{
kaConn.SetKeepAliveIdle(30*time.Second)
kaConn.SetKeepAliveCount(4)
kaConn.SetKeepAliveInterval(5*time.Second)
}
Update :
When I stop tcp keep alive, it works and Accept return instantly .
But even if not stop tcp keep alive, Accept return instantly when I use a script to test.Is it to do with the client?
I'm writing a simple proxy in go that relays an HTTP request to one or more backend servers. Right now I'm still using only one backend server, but performance is not good and I'm sure I am doing something wrong. Probably related to how I send the HTTP request to another server: if I comment the call to send() then the server goes blazing fast, yielding more than 14 krps. While with the call to send() performance drops to less than 1 krps and drops even lower with time. This on a MacBook Pro.
The code is based on trivial code samples; I have created the client and reused it following the recommendation in the docs. Tests are done with Apache ab:
$ ab -n 10000 -c 10 -k "http://127.0.0.1:8080/test"
The backend server running on port 55455 does not change between experiments; Apache or nginx can be used. My custom web server yields more than 7 krps when measured directly, without proxy:
$ ab -n 10000 -c 10 -k "http://127.0.0.1:55455/test"
I would expect the proxied version to behave just as well as the non-proxied version, and sustain the performance over time.
The complete sample code follows.
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
tr := &http.Transport{
DisableCompression: true,
DisableKeepAlives: false,
}
client := &http.Client{Transport: tr}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
send(r, client)
fmt.Fprintf(w, "OK")
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
func send(r *http.Request, client *http.Client) int {
req, err := http.NewRequest("GET", "http://localhost:55455" + r.URL.Path, nil)
if err != nil {
log.Fatal(err)
return 0
}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
return 0
}
if resp == nil {
return 0
}
return 1
}
Eventually the code should send the request to multiple servers and process their answers, returning an int with the result. But I'm stuck at this step of just making the call.
What am I doing horribly wrong?
As the comment suggests, you should be returning (and dealing with) type error instead of ints, and to reiterate, don't use AB. The biggest thing that stands out to me is
You should set the MaxIdleConnsPerHost in your Transport. This represents how many connections will persist (keep-alive) even if they have nothing to do at the moment.
You have to read and close the body of the response in order for it to be returned to the pool.
...
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
defer resp.Body.Close()
...