I'd like to write a function that takes a SSH session and runs a command, providing an io.Reader that reads its output as it is recieved.
package main
import (
"golang.org/x/crypto/ssh"
"io"
)
func StreamOutput(session *ssh.Session, command string) (output io.Reader, err error) {
if session != nil {
defer session.Close()
}
if err != nil {
return output, err
}
// connect to both outputs
outReader, err := session.StdoutPipe()
if err != nil {
return output, err
}
errReader, err := session.StderrPipe()
if err != nil {
return output, err
}
output = io.MultiReader(outReader, errReader)
err = session.Start(command)
// return the readers that are to be read from
return output, err
When I test this code using an ioutil.ReadAll on output, however, it only prints the appropriate output intermittently, even after calling session.Wait(). Why might this be? Is there a better way to return the functions output (channels, perhaps?)? Thanks!
The problem was that I had the following code in my function:
if session != nil {
defer session.Close()
}
That way, the output would only print if the command completed and sent back its stdout before the function returned (which was infrequent). I fixed my code simply by removing these lines.
Related
I am trying to push my data to apache server using GoLang. Suppose my apache server is temporarily stopped. Then will my http request retry automatically.
I am using this statement
resp, err := http.DefaultClient.Do(req)
if err != nil {
return errors.Wrap(err, "http request error")
}
I am unable to proceed further coz what is think is my execution is stuck here. And I am repeatedly getting this error.
No, you will need to implement your own retry method, this is a basic example that could give you an idea:
https://play.golang.org/p/_o5AgePDEXq
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
var (
err error
response *http.Response
retries int = 3
)
for retries > 0 {
response, err = http.Get("https://non-existent")
// response, err = http.Get("https://google.com/robots.txt")
if err != nil {
log.Println(err)
retries -= 1
} else {
break
}
}
if response != nil {
defer response.Body.Close()
data, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
fmt.Printf("data = %s\n", data)
}
}
You can use more cleaner implementation of retrying by using retryablehttp
https://github.com/hashicorp/go-retryablehttp which handles most of the conditions
This is provides customisation for retry policy, backoffs etc.
I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.
I'm very new to Go, so I may be misunderstanding something foundational about Go's async/stream handling, but here goes...
I'm trying to write some tests using ginkgo on a function I wrote that processes streams.
The processing side reads in newline-delimited text from a File until it encounters a special delimiter line at which point it tries to parse the text as JSON. The code looks like this:
func ParseConfig(inStream *os.File) (Config, error){
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break;
}
header += line
}
// parse JSON here and return
}
My test looks something like this
Describe("ParseConfig()", func() {
It("should pass for a valid header", func(){
_, err := io.WriteString(stream, "{\"Key\": \"key\", \"File\": \"file\"}\n|||\n")
Expect(err).NotTo(HaveOccurred())
conf, err := parser.ParseConfig(stream)
Expect(err).NotTo(HaveOccurred())
Expect(conf.Key).To(Equal("key"))
})
})
Unfortunately, this yields a JSON parsing error, as it's trying to parse an empty string. I'm assuming that my problem is that I'm sending the string on the stream before I've told the ParseConfig() function to listen on that string for data? But I'm not entirely clear how I could refactor this to use proper go routines to first listen for data then send it.
Some of the potential solutions I saw were around the use of "channels" (with which I'm unfamiliar) but I was worried that this one need might not be worth a major refactor to introduce a whole new paradigm of concurrency.
Thanks!
Not sure if I understood correctly, but your ParseConfig should probably take an io.Reader instead of a *os.File. That way you can test it directly without worrying about concurrency.
file t_test.go:
package main
import (
"strings"
"testing"
"github.com/onsi/ginkgo"
"github.com/onsi/gomega"
)
var _ = ginkgo.Describe("ParseConfig()", func() {
ginkgo.It("should pass for a valid header", func() {
// really don't know what you were doing with your 'stream' variable
// This is a test, you should forge a test scenario and pass it to your config function
stream := strings.NewReader(`{"Key": "key", "File": "file"}` + "\n|||\n")
conf, err := ParseConfig(stream)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(conf.Key).To(gomega.Equal("key"))
})
})
func TestParseConfig(t *testing.T) {
ginkgo.RunSpecs(t, "Parse Config")
}
file main.go
package main
import (
"bufio"
"encoding/json"
"io"
"log"
"os"
)
type Config struct {
Key string
File string
}
func ParseConfig(inStream io.Reader) (*Config, error) {
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break
}
header += line
}
c := &Config{}
// parse JSON here and return
if err := json.Unmarshal([]byte(header), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
f, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
ParseConfig(f)
}
I'm writing a simple web app in Go and I want my responses to be streamed to the client (i.e. not buffered and sent in blocks once the request is fully processed) :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
From the client point of view, the two lines will be sent at the same time. Any suggestions are appreciated :)
Edit after #dystroy answer
It's possible to flush after each write I personally make, but in my use case it's not enough:
cmd := exec.Command("a long command that outputs lots of lines")
cmd.Stdout = res //where res is a http.ResponseWritter
cmd.Stderr = res
err := cmd.Run()
I want the output of my cmd to be flushed as well. Anyway to "autoflush" the ResponseWritter ?
Solution
I found help on golang's mailing list. There is 2 way to achieve this: using hijacker that allow to take over the underlying TCP connection of HTTP, or piping the stdout and stderr of the command in a go routine that will write and flush :
pipeReader, pipeWriter := io.Pipe()
cmd.Stdout = pipeWriter
cmd.Stderr = pipeWriter
go writeCmdOutput(res, pipeReader)
err := cmd.Run()
pipeWriter.Close()
//---------------------
func writeCmdOutput(res http.ResponseWriter, pipeReader *io.PipeReader) {
buffer := make([]byte, BUF_LEN)
for {
n, err := pipeReader.Read(buffer)
if err != nil {
pipeReader.Close()
break
}
data := buffer[0:n]
res.Write(data)
if f, ok := res.(http.Flusher); ok {
f.Flush()
}
//reset buffer
for i := 0; i < n; i++ {
buffer[i] = 0
}
}
}
Last update
Even nicer: http://play.golang.org/p/PpbPyXbtEs
As implied in the documentation, some ResponseWriter may implement the Flusher interface.
This means you can do something like this :
func handle(res http.ResponseWriter, req *http.Request) {
fmt.Fprintf(res, "sending first line of data")
if f, ok := res.(http.Flusher); ok {
f.Flush()
} else {
log.Println("Damn, no flush");
}
sleep(10) //not real code
fmt.Fprintf(res, "sending second line of data")
}
Be careful that buffering can occur in many other places in the network or client side.
Sorry if I've misunderstood your question, but would something like the below do the trick?
package main
import (
"bytes"
"fmt"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
body := make([]byte, int(r.ContentLength))
b := bytes.NewBuffer(body)
if _, err := b.ReadFrom(r.Body); err != nil {
fmt.Fprintf(w, "%s", err)
}
if _, err := b.WriteTo(w); err != nil {
fmt.Fprintf(w, "%s", err)
}
}
func main() {
http.HandleFunc("/", handler)
if err := http.ListenAndServe(":8080", nil); err != nil {
panic(err)
}
}
$ curl --data "param1=value1¶m2=value2" http://localhost:8080
returns:
param1=value1¶m2=value2
You could always append whatever data you wanted to body, or read more bytes into the buffer from elsewhere before writing it all out again.
I'm trying to detect sending failures by inspecting the error returned by golang TCPConn.Write, but it's nil. I also tried using TCPConn.SetWriteDeadline without success.
That's how things happen:
the server starts
a client connects
the server sends a message and the client receives it
the client shuts down
the server sends one more message: no error
the server sends the third message: only now the error appears
Question: why only the second message to a non-existing client results in an error? How should the case be handled properly?
The code follows:
package main
import (
"net"
"os"
"bufio"
"fmt"
"time"
)
func AcceptConnections(listener net.Listener, console <- chan string) {
msg := ""
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
for {
if msg == "" {
msg = <- console
fmt.Printf("read from console: %s", msg)
}
err = conn.SetWriteDeadline(time.Now().Add(time.Second))
if err != nil {
fmt.Printf("SetWriteDeadline failed: %v\n", err)
}
_, err = conn.Write([]byte(msg))
if err != nil {
// expecting an error after sending a message
// to a non-existing client endpoint
fmt.Printf("failed sending a message to network: %v\n", err)
break
} else {
fmt.Printf("msg sent: %s", msg)
msg = ""
}
}
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
The server console looks like this:
listening on 127.0.0.1:6666
client connected
hi there!
read from console: hi there!
msg sent: hi there!
this one should fail
read from console: this one should fail
msg sent: this one should fail
this one actually fails
read from console: this one actually fails
failed sending a message to network: write tcp 127.0.0.1:51194: broken pipe
The client looks like this:
package main
import (
"net"
"os"
"io"
//"bufio"
//"fmt"
)
func cp(dst io.Writer, src io.Reader, errc chan<- error) {
// -reads from src and writes to dst
// -blocks until EOF
// -EOF is not an error
_, err := io.Copy(dst, src)
// push err to the channel when io.Copy returns
errc <- err
}
func StartCommunication(conn net.Conn) {
//create a channel for errors
errc := make(chan error)
//read connection and print to console
go cp(os.Stdout, conn, errc)
//read user input and write to connection
go cp(conn, os.Stdin, errc)
//wait until nil or an error arrives
err := <- errc
if err != nil {
println("cp error: ", err.Error())
}
}
func main() {
servAddr := "localhost:6666"
tcpAddr, err := net.ResolveTCPAddr("tcp", servAddr)
if err != nil {
println("ResolveTCPAddr failed:", err.Error())
os.Exit(1)
}
conn, err := net.DialTCP("tcp", nil, tcpAddr)
if err != nil {
println("net.DialTCP failed:", err.Error())
os.Exit(1)
}
defer conn.Close()
StartCommunication(conn)
}
EDIT: Following JimB's suggestion I came up with a working example. Messages don't get lost any more and are re-sent in a new connection. I'm not quite sure though how safe is it to use a shared variable (connWrap.IsFaulted) between different go routines.
package main
import (
"net"
"os"
"bufio"
"fmt"
)
type Connection struct {
IsFaulted bool
Conn net.Conn
}
func StartWritingToNetwork(connWrap * Connection, errChannel chan <- error, msgStack chan string) {
for {
msg := <- msgStack
if connWrap.IsFaulted {
//put it back for another connection
msgStack <- msg
return
}
_, err := connWrap.Conn.Write([]byte(msg))
if err != nil {
fmt.Printf("failed sending a message to network: %v\n", err)
connWrap.IsFaulted = true
msgStack <- msg
errChannel <- err
return
} else {
fmt.Printf("msg sent: %s", msg)
}
}
}
func StartReadingFromNetwork(connWrap * Connection, errChannel chan <- error){
network := bufio.NewReader(connWrap.Conn)
for (!connWrap.IsFaulted) {
line, err := network.ReadString('\n')
if err != nil {
fmt.Printf("failed reading from network: %v\n", err)
connWrap.IsFaulted = true
errChannel <- err
} else {
fmt.Printf("%s", line)
}
}
}
func AcceptConnections(listener net.Listener, console chan string) {
errChannel := make(chan error)
for {
conn, err := listener.Accept()
if err != nil {
panic(err)
}
fmt.Printf("client connected\n")
connWrap := Connection{false, conn}
go StartReadingFromNetwork(&connWrap, errChannel)
go StartWritingToNetwork(&connWrap, errChannel, console)
//block until an error occurs
<- errChannel
}
}
func ReadConsole(network chan <- string) {
console := bufio.NewReader(os.Stdin)
for {
line, err := console.ReadString('\n')
if err != nil {
panic(err)
} else {
network <- line
}
}
}
func main() {
listener, err := net.Listen("tcp", "localhost:6666")
if err != nil {
panic(err)
}
println("listening on " + listener.Addr().String())
consoleToNetwork := make(chan string)
go AcceptConnections(listener, consoleToNetwork)
ReadConsole(consoleToNetwork)
}
This isn't Go specific, and is a artifact of the underlying TCP socket showing through.
A decent diagram of the TCP termination steps is at the bottom of this page:
http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm
The simple version is that when the client closes its socket, it sends a FIN, and receives an ACK from the server. It then waits for the server to do the same. Instead of sending a FIN though, you're sending more data, which is discarded, and the client socket now assumes that any more data coming from you is invalid, so the next time you send you get an RST, which is what bubbles up into the error you see.
Going back to your program, you need to handle this somehow. Generally you can think of whomever is in charge of initiating a send, is also in charge of initiating termination, hence your server should assume that it can continue to send until it closes the connection, or encounters an error. If you need to more reliably detect the client closing, you need to have some sort of client response in the protocol. That way recv can be called on the socket and return 0, which alerts you to the closed connection.
In go, this will return an EOF error from the connection's Read method (or from within the Copy in your case). SetWriteDeadline doesn't work because a small write will go though and get dropped silently, or the client will eventually respond with an RST, giving you an error.