I am developing a simple Go server program which receives client's request and process it. And the code is simplified as this:
package main
import (
"fmt"
"net"
"os"
)
const (
pduLen = 32
)
func checkError(err error) {
if err != nil {
fmt.Println(err)
os.Exit(1)
}
}
func main() {
var buffer [4096]byte
var count int
conn, err := net.Dial("tcp", fmt.Sprintf("%s:%s", os.Args[1], os.Args[2]))
checkError(err)
for count < pduLen {
n, err := conn.Read(buffer[count:])
checkError(err)
count += n
}
......
}
I assume every request's length is 32 bytes (just an example). Because the TCP is a stream protocol, I need to use a loop to check whether an integral PDU is read:
for count < pduLen {
n, err := conn.Read(buffer[count:])
checkError(err)
count += n
}
Is there any method to assure that an integral PDU is read? Personally, I think the loop code is a little ugly.
It can depend on the exact nature of the PDU you are receiving, but this example will look for the size, and then read everything (using io.ReadFul()).
func read(conn net.Conn, key string) string {
fmt.Fprintf(conn, GenerateCommand(OP_GET, key))
if verify(conn) {
var size uint16
binary.Read(conn, binary.LittleEndian, &size)
b := make([]byte, size)
// _, err := conn.Read(b)
_, err := io.ReadFull(conn, b)
if err == nil {
return string(b)
}
}
return ""
}
func verify(conn net.Conn) bool {
b := make([]byte, 1)
conn.Read(b)
return b[0] == ERR_NO_ERROR
}
Used in:
conn, err := net.Dial("tcp", ":12345")
if err != nil {
t.Error(err)
}
write(conn, "foo", "bar")
if !verify(conn) {
t.Error("Bad write!")
}
if r := read(conn, "foo"); r != "bar" {
t.Errorf("Bad read! Got %v", r)
}
After discussing this issue in golang-nuts: How to read an integral network PDU?
The code should be:
import "io"
......
pdu := make([]byte, pduLen)
io.ReadFull(conn, pdu)
Related
I receive the contents of a file from a data source in chunks. As and when I receive the chunk I want to send the chunk data to a service using http POST request. And by keeping alive the same http POST connection used for sending the first chunk I want to send the remaining chunks of data.
I came up with the following code snippet to implement something similar.
Server-Side
func handle(w http.ResponseWriter, req *http.Request) {
buf := make([]byte, 256)
var n int
for {
n, err := req.Body.Read(buf)
if n == 0 && err == io.EOF {
break
}
fmt.Printf(string(buf[:n]))
}
fmt.Printf(string(buf[:n]))
fmt.Printf("Transfer Complete")
}
Client-Side
type alphaReader struct {
reader io.Reader
}
func newAlphaReader(reader io.Reader) *alphaReader {
return &alphaReader{reader: reader}
}
func (a *alphaReader) Read(p []byte) (int, error) {
n, err := a.reader.Read(p)
return n, err
}
func (a *alphaReader) Reset(str string) {
a.reader = strings.NewReader(str)
}
func (a *alphaReader) Close() error {
return nil
}
func main() {
tr := http.DefaultTransport
alphareader := newAlphaReader(strings.NewReader("First Chunk"))
client := &http.Client{
Transport: tr,
Timeout: 0,
}
req := &http.Request{
Method: "POST",
URL: &url.URL{
Scheme: "http",
Host: "localhost:8080",
Path: "/upload",
},
ProtoMajor: 1,
ProtoMinor: 1,
ContentLength: -1,
Body: alphareader,
}
fmt.Printf("Doing request\n")
_, err := client.Do(req)
alphareader.Reset("Second Chunk")
fmt.Printf("Done request. Err: %v\n", err)
}
Here I want that when I do alphareader.Reset("Second Chunk"), the string "Second Chunk" should be sent using the POST connection made earlier. But that is not happening. The connection gets closed after sending the First Chunk of data. Also I have not written the Close() method properly which I'm not sure how to implement.
I'm newbie to golang and any suggestions would be greatly helpful regarding the same.
A *strings.Reader returns io.EOF after the initial string has been read and your wrapper does nothing to change that, so it cannot be reused. You're looking for io.Pipe to turn the request body into an io.Writer.
package main
import (
"io"
"net/http"
)
func main() {
pr, pw := io.Pipe()
req, err := http.NewRequest("POST", "http://localhost:8080/upload", pr)
if err != nil {
// TODO: handle error
}
go func() {
defer pw.Close()
if _, err := io.WriteString(pw, "first chunk"); err != nil {
_ = err // TODO: handle error
}
if _, err := io.WriteString(pw, "second chunk"); err != nil {
_ = err // TODO: handle error
}
}()
res, err := http.DefaultClient.Do(req)
if err != nil {
// TODO: handle error
}
res.Body.Close()
}
Also, don't initialize the request using a struct literal. Use one of the constructors instead. In your code you're not setting the Host and Header fields, for instance.
I am trying to find the list of all directories using a recursive function. The code to the function is
func FindDirs(dir string, nativePartitions []int64, wg *sync.WaitGroup, dirlistchan chan string) {
// defer wg.Done here will give negative waitgroup panic, commenting it will give negative waitgroup counter panic
fd, err := os.Open(dir)
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir(){
dirlistchan <- buff.String()
FindDirs(buff.String(), nativePartitions, wg, dirlistchan) //recursion happens here
} else {
//fmt.Println(i.Name(), "is not native")
}
}
}
and in the main function, I am calling it as
wg := new(sync.WaitGroup)
dirlistchan := make(chan string, 1000)
wg.Add(1)
go func() {
filtermounts.FindDirs(parsedConfig.ScanFrom, []int64{filtermounts.EXT4_SUPER_MAGIC}, wg, dirlistchan)
}()
go func() {
wg.Wait()
close(dirlistchan)
}()
for i := range dirlistchan {
fmt.Println(i)
}
wg.Wait()
and I am getting a
fatal error: all goroutines are asleep - deadlock!
I was able to get this working if I am printing the result instead of using channels, or append to a slice using mutex. (verified with the linux find command to see if the results are same.) Please find the function after omitting channels and using sync.Mutex and append.
func FindDirs(dir string, nativePartitions []int64, dirlist *[]string, mutex *sync.Mutex) []string{
fd, err := os.Open(dir)
defer fd.Close()
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir(){
//dirlistchan <- buff.String()
mutex.Lock()
*dirlist = append(*dirlist, buff.String())
mutex.Unlock()
//fmt.Println(buff.String())
FindDirs(buff.String(), nativePartitions, dirlist, mutex)
} else {
//fmt.Println(i.Name(), "is not native")
}
}
return *dirlist
}
But I cannot think of a way to make this work with channels and goroutines. Any help is greatly appreciated.
Note: Here is a link to the golang playground with the code. I couldn't find a workaround to get the syscall thing to work on the playground either. It works on my system though.
Thanks.
Short answer : You are not closing the channel.
Fix : add defer wg.Done() at beginning of the go routine that calls FindDirs
go func() {
defer wg.Done()
filtermounts.FindDirs(parsedConfig.ScanFrom, []int64{filtermounts.EXT4_SUPER_MAGIC}, wg, dirlistchan)
}()
Why did it happen
The go routine that is responsponsible for closing the channel waits for wg there is no wg.Done in the code above. So close never happens
Now the for loop blocks on the channel for close or a value for ever, this cause the error
fatal error: all goroutines are asleep - deadlock!
So here is your code ,this may be run as
go run filename.go /path/to/folder
Code
package main
import (
"bytes"
"fmt"
"os"
"sync"
"syscall"
)
func main() {
wg := new(sync.WaitGroup)
dirlistchan := make(chan string, 1000)
wg.Add(1)
go func() {
defer wg.Done()
FindDirs(os.Args[1], []int64{61267}, wg, dirlistchan)
}()
go func() {
wg.Wait()
close(dirlistchan)
}()
for i := range dirlistchan {
fmt.Println(i)
}
wg.Wait()
}
func FindDirs(dir string, nativePartitions []int64, wg *sync.WaitGroup, dirlistchan chan string) {
fd, err := os.Open(dir)
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir() {
dirlistchan <- buff.String()
FindDirs(buff.String(), nativePartitions, wg, dirlistchan) //recursion happens here
} else {
//fmt.Println(i.Name(), "is not native")
}
}
}
func checkDirIsNative(dirtype int64, nativetypes []int64) bool {
for _, i := range nativetypes {
if dirtype == i {
return true
}
}
return false
}
Find the go.play link here
As has been stated already you should close the channel if you want the main goroutine to exit.
Example of implementation :
In function func FindDirs you could make an additional channel for every recursive func FindDirs call that this function is going to make and pass that new channel in the argument. Then simultaneously listen to all those new channels and forward the strings back to the channel that function got in the argument.
After all new channels has been closed close the channel given in the argument.
In other words every func call should have its own channel that it sends to. The string is then forwarded all the way to main function.
Dynamic select described here : how to listen to N channels? (dynamic select statement)
I have a go test program to read encrypted content from file and decrypt it, but it get output like below:
illegal base64 data at input byte 0
if I hard code the encrypted content in a golang string variable, it can decrypt it fine. what I am missing here? I searched similar error in stackoverflow, there is similar report, but not exact the same problem I have. the test code like below:
package main
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/base64"
"errors"
"fmt"
"io"
"bufio"
"os"
"log"
)
func check(e error) {
if e != nil {
panic(e)
}
}
func main() {
plaintext := []byte("textstring")
key := []byte("a very very very very very secre")
fmt.Printf("%s\n", plaintext)
fh, err := os.Open("./test.txt")
check(err)
scanner := bufio.NewScanner(fh)
var encrypted_text string
if scanner.Scan() { //<==========READ FROM FILE
encrypted_text = scanner.Text()
fmt.Println("encrypted_text from file: ", encrypted_text)
} else { //<===========HARD CODE HERE
encrypted_text = "\xf2F\xbc\x15\x9d\xaf\xceϘ\xa3L(>%\xa2\x94\x03_\x99\u007fG\xd8\v\xbf\t#u\xf8:\xc0D\u007f"
fmt.Println("encrypted_text hard coded: ", encrypted_text)
}
encrypted_byte := []byte(encrypted_text)
fmt.Printf("encrypted_byte: %s\n", encrypted_byte)
result, err := decrypt(key, encrypted_byte)
if err != nil {
log.Fatal(err)
}
fmt.Printf("result %s\n", string(result))
}
func encrypt(key, text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
b := base64.StdEncoding.EncodeToString(text)
ciphertext := make([]byte, aes.BlockSize+len(b))
iv := ciphertext[:aes.BlockSize]
if _, err := io.ReadFull(rand.Reader, iv); err != nil {
return nil, err
}
cfb := cipher.NewCFBEncrypter(block, iv)
cfb.XORKeyStream(ciphertext[aes.BlockSize:], []byte(b))
return ciphertext, nil
}
func decrypt(key, text []byte) ([]byte, error) {
block, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
if len(text) < aes.BlockSize {
return nil, errors.New("ciphertext too short")
}
iv := text[:aes.BlockSize]
text = text[aes.BlockSize:]
cfb := cipher.NewCFBDecrypter(block, iv)
cfb.XORKeyStream(text, text)
data, err := base64.StdEncoding.DecodeString(string(text))
if err != nil {
return nil, err
}
return data, nil
}
You need to unquote the encrypted_text returned from the scanner.
Here's a minimal example
Modify your scanner.Scan() if block to look like this
if scanner.Scan() { //<==========READ FROM FILE
encrypted_text = scanner.Text()
fmt.Println("encrypted_text from file: ", encrypted_text)
// Unquoting, don't forget to import strconv !
encrypted_text, err := strconv.Unquote(`"` + encrypted_text + `"`)
check(err)
}
why you need to unquote
I'm guessing your file test.txt contains the raw string
\xf2F\xbc\x15\x9d\xaf\xceQ\xa3L(>%\xa2\x94\x03_\x99\u007fG\xd8\v\xbf\t#u\xf8:\xc0D\u007f
When scanner reads this from the file, it is reading a \ as a \.
However, when you hardcode it in your code like this
encrypted_text = "\xf2F\xbc\x15\x9d\xaf\xceϘ\xa3L(>%\xa2\x94\x03_\x99\u007fG\xd8\v\xbf\t#u\xf8:\xc0D\u007f"
You are using double quotes ", so a \ isn't a \. It interprets the escape sequences. If you were to use a backquote as follows
encrypted_text = `\xf2F\xbc\x15\x9d\xaf\xceϘ\xa3L(>%\xa2\x94\x03_\x99\u007fG\xd8\v\xbf\t#u\xf8:\xc0D\u007f`
you would face the same issue.
The solution is to unquote this string using strconv.Unquote
Also, take a look at This SO question
What I'm doing is fairly straight-forward. I need to create a "proxy" server that is very minimal and fast. Currently I have a baseline server that is proxied to (nodejs) and a proxy-service (go). Please excuse the lack of actual "proxy'ing" - just testing for now.
Baseline Service
var http = require('http');
http.createServer(function (req, res) {
// console.log("received request");
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(8080, '127.0.0.1');
console.log('Server running at http://127.0.0.1:8080/');
Proxy Service
package main
import (
"flag"
"log"
"net/http"
"net/url"
)
var (
listen = flag.String("listen", "0.0.0.0:9000", "listen on address")
logp = flag.Bool("log", false, "enable logging")
)
func main() {
flag.Parse()
proxyHandler := http.HandlerFunc(proxyHandlerFunc)
log.Fatal(http.ListenAndServe(*listen, proxyHandler))
log.Println("Started router-server on 0.0.0.0:9000")
}
func proxyHandlerFunc(w http.ResponseWriter, r *http.Request) {
// Log if requested
if *logp {
log.Println(r.URL)
}
/*
* Tweak the request as appropriate:
* - RequestURI may not be sent to client
* - Set new URL
*/
r.RequestURI = ""
u, err := url.Parse("http://localhost:8080/")
if err != nil {
log.Fatal(err)
}
r.URL = u
// And proxy
// resp, err := client.Do(r)
c := make(chan *http.Response)
go doRequest(c)
resp := <-c
if resp != nil {
err := resp.Write(w)
if err != nil {
log.Println("Error writing response")
} else {
resp.Body.Close()
}
}
}
func doRequest(c chan *http.Response) {
// new client for every request.
client := &http.Client{}
resp, err := client.Get("http://127.0.0.1:8080/test")
if err != nil {
log.Println(err)
c <- nil
} else {
c <- resp
}
}
My issue, as mentioned within the title, is that I am getting errors stating 2013/10/28 21:22:30 Get http://127.0.0.1:8080/test: dial tcp 127.0.0.1:8080: can't assign requested address from the doRequest function, and I have no clue why. Googling this particular error yields seemingly irrelevant results.
There are 2 major problems with this code.
You are not handling the client stalling or using keep alives (handled below by getTimeoutServer)
You are not handling the server (what your http.Client is talking to) timing out (handled below by TimeoutConn).
This is probably why you are exhausting your local ports. I know from past experience node.js will keep-alive you very aggressively.
There are lots of little issues, creating objects every-time when you don't need to. Creating unneeded goroutines (each incoming request is in its own goroutine before you handle it).
Here is a quick stab (that I don't have time to test well). Hopefully it will put you on the right track: (You will want to upgrade this to not buffer the responses locally)
package main
import (
"bytes"
"errors"
"flag"
"fmt"
"log"
"net"
"net/http"
"net/url"
"runtime"
"strconv"
"time"
)
const DEFAULT_IDLE_TIMEOUT = 5 * time.Second
var (
listen string
logOn bool
localhost, _ = url.Parse("http://localhost:8080/")
client = &http.Client{
Transport: &http.Transport{
Proxy: NoProxyAllowed,
Dial: func(network, addr string) (net.Conn, error) {
return NewTimeoutConnDial(network, addr, DEFAULT_IDLE_TIMEOUT)
},
},
}
)
func main() {
runtime.GOMAXPROCS(runtime.NumCPU())
flag.StringVar(&listen, "listen", "0.0.0.0:9000", "listen on address")
flag.BoolVar(&logOn, "log", true, "enable logging")
flag.Parse()
server := getTimeoutServer(listen, http.HandlerFunc(proxyHandlerFunc))
log.Printf("Starting router-server on %s\n", listen)
log.Fatal(server.ListenAndServe())
}
func proxyHandlerFunc(w http.ResponseWriter, req *http.Request) {
if logOn {
log.Printf("%+v\n", req)
}
// Setup request URL
origURL := req.URL
req.URL = new(url.URL)
*req.URL = *localhost
req.URL.Path, req.URL.RawQuery, req.URL.Fragment = origURL.Path, origURL.RawQuery, origURL.Fragment
req.RequestURI, req.Host = "", req.URL.Host
// Perform request
resp, err := client.Do(req)
if err != nil {
w.WriteHeader(http.StatusBadGateway)
w.Write([]byte(fmt.Sprintf("%d - StatusBadGateway: %s", http.StatusBadGateway, err)))
return
}
defer resp.Body.Close()
var respBuffer *bytes.Buffer
if resp.ContentLength != -1 {
respBuffer = bytes.NewBuffer(make([]byte, 0, resp.ContentLength))
} else {
respBuffer = new(bytes.Buffer)
}
if _, err = respBuffer.ReadFrom(resp.Body); err != nil {
w.WriteHeader(http.StatusBadGateway)
w.Write([]byte(fmt.Sprintf("%d - StatusBadGateway: %s", http.StatusBadGateway, err)))
return
}
// Write result of request
headers := w.Header()
var key string
var val []string
for key, val = range resp.Header {
headers[key] = val
}
headers.Set("Content-Length", strconv.Itoa(respBuffer.Len()))
w.WriteHeader(resp.StatusCode)
w.Write(respBuffer.Bytes())
}
func getTimeoutServer(addr string, handler http.Handler) *http.Server {
//keeps people who are slow or are sending keep-alives from eating all our sockets
const (
HTTP_READ_TO = DEFAULT_IDLE_TIMEOUT
HTTP_WRITE_TO = DEFAULT_IDLE_TIMEOUT
)
return &http.Server{
Addr: addr,
Handler: handler,
ReadTimeout: HTTP_READ_TO,
WriteTimeout: HTTP_WRITE_TO,
}
}
func NoProxyAllowed(request *http.Request) (*url.URL, error) {
return nil, nil
}
//TimeoutConn-------------------------
//Put me in my own TimeoutConn.go ?
type TimeoutConn struct {
net.Conn
readTimeout, writeTimeout time.Duration
}
var invalidOperationError = errors.New("TimeoutConn does not support or allow .SetDeadline operations")
func NewTimeoutConn(conn net.Conn, ioTimeout time.Duration) (*TimeoutConn, error) {
return NewTimeoutConnReadWriteTO(conn, ioTimeout, ioTimeout)
}
func NewTimeoutConnReadWriteTO(conn net.Conn, readTimeout, writeTimeout time.Duration) (*TimeoutConn, error) {
this := &TimeoutConn{
Conn: conn,
readTimeout: readTimeout,
writeTimeout: writeTimeout,
}
now := time.Now()
err := this.Conn.SetReadDeadline(now.Add(this.readTimeout))
if err != nil {
return nil, err
}
err = this.Conn.SetWriteDeadline(now.Add(this.writeTimeout))
if err != nil {
return nil, err
}
return this, nil
}
func NewTimeoutConnDial(network, addr string, ioTimeout time.Duration) (net.Conn, error) {
conn, err := net.DialTimeout(network, addr, ioTimeout)
if err != nil {
return nil, err
}
if conn, err = NewTimeoutConn(conn, ioTimeout); err != nil {
return nil, err
}
return conn, nil
}
func (this *TimeoutConn) Read(data []byte) (int, error) {
this.Conn.SetReadDeadline(time.Now().Add(this.readTimeout))
return this.Conn.Read(data)
}
func (this *TimeoutConn) Write(data []byte) (int, error) {
this.Conn.SetWriteDeadline(time.Now().Add(this.writeTimeout))
return this.Conn.Write(data)
}
func (this *TimeoutConn) SetDeadline(time time.Time) error {
return invalidOperationError
}
func (this *TimeoutConn) SetReadDeadline(time time.Time) error {
return invalidOperationError
}
func (this *TimeoutConn) SetWriteDeadline(time time.Time) error {
return invalidOperationError
}
We ran into this and after a lot of time trying to debug, I came across this: https://code.google.com/p/go/source/detail?r=d4e1ec84876c
This shifts the burden onto clients to read their whole response
bodies if they want the advantage of reusing TCP connections.
So be sure you read the entire body before closing, there are a couple of ways to do it. This function can come in handy to close to let you see whether you have this issue by logging the extra bytes that haven't been read and cleaning the stream out for you so it can reuse the connection:
func closeResponse(response *http.Response) error {
// ensure we read the entire body
bs, err2 := ioutil.ReadAll(response.Body)
if err2 != nil {
log.Println("Error during ReadAll!!", err2)
}
if len(bs) > 0 {
log.Println("Had to read some bytes, not good!", bs, string(bs))
}
return response.Body.Close()
}
Or if you really don't care about the body, you can just discard it with this:
io.Copy(ioutil.Discard, response.Body)
I have encountered this problem too, and i add an option {DisableKeepAlives: true} to http.Transport fixed this issue, you can have a try.
I came here when running a massive amount of SQL queries per second on a system without limiting the number of idle connections over a long period of time. As pointed out in this issue comment on github explicitly setting db.SetMaxIdleConns(5) completely solved my problem.
I try to connect a http server as long connection, like below:
func main() {
request, err := http.NewRequest("GET", "http://long.connection.org:8080/", nil)
request.SetBasicAuth("xxx", "oooo")
http_client := &http.Client{}
response, _ := http_client.Do(request)
var buf []byte
for {
_, err := response.Body.Read(buf)
if err == io.EOF { break }
fmt.Printf("%s", string(buf))
}
}
But read from response.Body always empty. And seems I can't use response.Body to send data to server.
Any one can help?
This seems to work:
package main
import (
"fmt"
"io"
"log"
"net/http"
)
func main() {
request, err := http.NewRequest("GET", "http://www.example.com/", nil)
if err != nil {
log.Fatal(err)
}
http_client := &http.Client{}
response, err := http_client.Do(request)
if err != nil {
log.Fatal(err)
}
buf := make([]byte, 4096) // any non zero value will do, try '1'.
for {
n, err := response.Body.Read(buf)
if n == 0 && err != nil { // simplified
break
}
fmt.Printf("%s", buf[:n]) // no need to convert to string here
}
fmt.Println()
}
Edit: Added forgotten error handling of NewRequest.