I am trying to convert my node.js HTTP server to Go. Here's what I want to happen:
I have a resource that gets generated intermittently (say every second or so), and I want all requests for this resource to wait until the next time it is generated. This way clients can poll and be guaranteed to get only the up-to-date resource. I am using web.go to remove a lot of the complexity of running an HTTP server.
Here is a brief version of my code:
package main
import (
"time"
"web"
"fmt"
vector "container/vector"
)
var listeners vector.Vector;
func resource (ctx *web.Context) {
c := make(chan int)
listeners.Push(c)
go func() {
<-c
go func() {
ctx.WriteString("Cool")
fmt.Println("Wrote something")
}()
}()
}
func resourceLoop() {
time.Sleep(5 * 1000 * 1000000) // sleep for 5 seconds
for ; listeners.Len() > 0 ; {
c := listeners.Pop().(chan int)
c <- 1
}
go resourceLoop()
}
func main() {
web.Get("/resource", resource)
go resourceLoop()
web.Run("localhost:4000")
}
I would expect there to be a Context.End() or similar function, but it doesn't seem to have one. I read the source for web.go, but I couldn't figure out where it was ending the response (web.go:268 is where my resource() is called). In node.js this is trivial, you can call a ServerResponse.end().
When I kill the server while running the script in Chrome, I get this output (seems to be correct, except that the response isn't ending):
4
Cool
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: Sat, 16 Apr 2011 00:37:58 GMT
Server: web.go
Transfer-Encoding: chunked
0
Is this a problem with the web.go framework or am I doing something wrong? If it's a problem with the framework, I'll file an issue with him.
I'm pretty new to Go, so I could be going about this completely wrong.
I've never used web.go, but it seems that your example is entirely too complicated. Why do you need a goroutine to spawn a goroutine? I would assume the framework itself would take care of concurrency and just write this:
func resource (ctx *web.Context, val string) string {
c := make(chan int)
listeners.Push(c)
<-c
return "Cool"
}
Otherwise, it looks like it is doing exactly what you want and you just need to close the connection if you're truly done with it:
ctx.Close()
Related
I recently run into a problem when I develope a high concurrency http client via valyala/fasthttp: The client works fine in the first 15K~ requests but after that more and more dial tcp4 127.0.0.1:80: i/o timeout and dialing to the given TCP address timed out error occours.
Sample Code
var Finished = 0
var Failed = 0
var Success = 0
func main() {
for i := 0; i < 1000; i++ {
go get()
}
start := time.Now().Unix()
for {
fmt.Printf("Rate: %.2f/s Success: %d, Failed: %d\n", float64(Success)/float64(time.Now().Unix()-start), Success, Failed)
time.Sleep(100 * time.Millisecond)
}
}
func get() {
ticker := time.NewTicker(time.Duration(100+rand.Intn(2900)) * time.Millisecond)
defer ticker.Stop()
client := &fasthttp.Client{
MaxConnsPerHost: 10000,
}
for {
req := &fasthttp.Request{}
req.SetRequestURI("http://127.0.0.1:80/require?number=10")
req.Header.SetMethod(fasthttp.MethodGet)
req.Header.SetConnectionClose()
res := &fasthttp.Response{}
err := client.DoTimeout(req, res, 5*time.Second)
if err != nil {
fmt.Println(err.Error())
Failed++
} else {
Success++
}
Finished++
client.CloseIdleConnections()
<-ticker.C
}
}
Detail
The server is built on labstack/echo/v4 and when client got timeout error, the server didn't have any error, and manually perform the request via Postman or Browser like Chrome are works fine.
The client runs pretty well in the first 15K~ request, but after that, more and more timeout error occours and the output Rate is decreasing. I seached for google and github and I found this issue may be the most suitable one, but didn't found a solution.
Another tiny problem...
As you can notice, when the client start, it will first generate some the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection error, and then works fine till around 15K issues, and then start generating more and more timeout error.Why it would generate the Connection closed error in the begining?
Machine Info
Macbook Pro 14 2021 (Apple M1 Pro) with 16GB Ram and running macOS Monterey 12.4
So basically, If you trying to open a connection and then close it as soon as possibile, it's not going to be like "connection#1 use a port then immediately return it back", there gonna be lots of processing needs to be done, so If you want to send many request at the same time, I think it's better to reuse the connection as possible as you can.
For example, in fasthttp:
req := fasthttp.AcquireRequest()
res := fasthttp.AcquireResponse()
defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(res)
// Then do the request below
I created a simple http2 server,
If I send a request to it with curl, it responds with some headers, although I did not set them explicity. How can I acces them inside the requesthandling function ( sayhello )? My code ( I've never used golang before)
server.go
package main
import (
"net/http"
"strings"
"fmt"
"github.com/gorilla/mux"
"golang.org/x/net/http2"
)
func sayHello(w http.ResponseWriter, r *http.Request) {
message := r.URL.Path
message = strings.TrimPrefix(message, "/")
message = "Hello " + message
w.Header().Set("myFirst", "golangQuestion")
w.Write([]byte(message))
for k, v := range w.Header() {
fmt.Println("[RESPONSE][Header]", k,":", v)
}
}
func main() {
router := mux.NewRouter()
router.PathPrefix("/").HandlerFunc(sayHello) // catch everything else rule
var srv = &http.Server{
Addr: "127.0.0.1:8081",
}
http2.ConfigureServer(srv, nil)
srv.Handler = router
sslCert := "./ssl.cert"
sslKey := "./ssl.key"
if err := srv.ListenAndServeTLS(sslCert, sslKey); err != nil {
panic(err)
}
}
Sending request:
curl --head --insecure https://127.0.0.1:8081
HTTP/1.1 200 OK
Myfirst: golangQuestion
Date: Tue, 18 Jun 2019 09:18:29 GMT
Content-Length: 6
Content-Type: text/plain; charset=utf-8
I can see that some headers are sent back, the one which I set explicitly is also recieved, but the output of
go run server.go
[RESPONSE][Header] Myfirst : [golangQuestion]
How can I acces the other headers, which were not explicitly set, but recieved by curl as well? I loopd through w.Headers, but it did not contain the implicitly set headers
for k, v := range w.Header() {
fmt.Println("[RESPONSE][Header]", k,":", v)
}
My expectation that the output of go run server.go shall be something like this:
[RESPONSE][Header] Myfirst : [golangQuestion]
[RESPONSE][Header] Date: [2019.02.12 ]
[RESPONSE][Header] Content-Length: [6]
Those headers are sent automatically when you call ResponseWriter.Write(). Quoting from its doc:
// Write writes the data to the connection as part of an HTTP reply.
//
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// DetectContentType. Additionally, if the total size of all written
// data is under a few KB and there are no Flush calls, the
// Content-Length header is added automatically.
//
// Depending on the HTTP protocol version and the client, calling
// Write or WriteHeader may prevent future reads on the
// Request.Body. For HTTP/1.x requests, handlers should read any
// needed request body data before writing the response. Once the
// headers have been flushed (due to either an explicit Flusher.Flush
// call or writing enough data to trigger a flush), the request body
// may be unavailable. For HTTP/2 requests, the Go HTTP server permits
// handlers to continue to read the request body while concurrently
// writing the response. However, such behavior may not be supported
// by all HTTP/2 clients. Handlers should read before writing if
// possible to maximize compatibility.
Write([]byte) (int, error)
ResponseWriter.Header() contains only the headers set explicitly. The Content-Type and Content-Length were sent by w.Write().
Note: if you want to suppress such automatic headers, you have to set their values to nil, e.g.:
w.Header()["Date"] = nil
Also note that if you set the values of such headers manually, those values will be sent without being changed.
I'm trying to learn Go web programming, and here is a simple web server: it prints out the times being called.
package main
import (
"fmt"
"net/http"
)
var calls int
// HelloWorld print the times being called.
func HelloWorld(w http.ResponseWriter, r *http.Request){
calls++
fmt.Fprintf(w, "You've called me %d times", calls)
}
func main() {
fmt.Printf("Started server at http://localhost%v.\n", 5000)
http.HandleFunc("/", HelloWorld)
http.ListenAndServe(":5000", nil)
}
When I refresh the page, I got:
You've called me 1 times
You've called me 3 times
You've called me 5 times
....
Question: Why it is 1, 3, 5 times, rather than 1,2,3...? What's the order of the function HelloWorld being called?
It is because every incoming request is routed to your HelloWorld() handler function, and the browser makes multiple calls under the hood, specifically to /favicon.ico.
And since your web server does not send back a valid favicon, it will request it again when you refresh the page in the browser.
Try it with Chrome: open the Developer tools (CTRL+SHIFT+I), and choose the "Network" tab. Hit refresh, and you will see 2 new entries:
Name Status Type
--------------------------------------------------------
localhost 200 document
favicon.ico 200 text/plain
Since your counter starts with 0 (default value for type int), you increment it once and you send back 1. Then the request for favicon.ico increments it again (2), but the result is not displayed. Then if you refresh, it gets incremented again to 3 and you send that back, etc.
Also note that multiple goroutines can serve requests concurrently, so your solution has a race. You should synchronize access to the calls variable, or use the sync/atomic package to increment the counter safely, for example:
var calls int64
func HelloWorld(w http.ResponseWriter, r *http.Request) {
count := atomic.AddInt64(&calls, 1)
fmt.Fprintf(w, "You've called me %d times", count)
}
A simple "fix" to achieve what you want would be to check the request path, and if it is not the root "/", don't increment, e.g.:
func HelloWorld(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
return
}
count := atomic.AddInt64(&calls, 1)
fmt.Fprintf(w, "You've called me %d times", count)
}
You may also choose to only exclude requests for favicon.ico, e.g.:
if r.URL.Path == "/favicon.ico" {
return
}
Reading TCP connection between Redis-server by using bufio.Scanner
fmt.Fprintf(conn, "*3\r\n$3\r\nSET\r\n$5\r\nmykey\r\n$7\r\nHello!!\r\n")
scanner := bufio.NewScanner(conn)
for {
// fmt.Println("marker00")
if ok := scanner.Scan(); !ok {
// fmt.Println("marker01")
break
}
// fmt.Println("marker02")
fmt.Println(scanner.Text())
}
"+OK" comes as the result for first scanning, but the second scanning stops just in invoking Scan method. (marker00 -> marker02 -> marker00 and no output any more)
Why does Scan stop and how can I know the end of TCP response (without using bufio.Reader)?
Redis does not close the connection for you after sending a command. Scan() ends after io.EOF which is not sent.
Check out this:
package main
import (
"bufio"
"fmt"
"net"
)
// before go run, you must hit `redis-server` to wake redis up
func main() {
conn, _ := net.Dial("tcp", "localhost:6379")
message := "*3\r\n$3\r\nSET\r\n$1\r\na\r\n$1\r\nb\r\n"
go func(conn net.Conn) {
for i := 0; i < 10; i++ {
fmt.Fprintf(conn, message)
}
}(conn)
scanner := bufio.NewScanner(conn)
for {
if ok := scanner.Scan(); !ok {
break
}
fmt.Println(scanner.Text())
}
fmt.Println("Scanning ended")
}
Old question, but I had the same issue. Two solutions:
1) Add a "QUIT\r\n" command to your Redis message. This will cause Redis to close the connection which will terminate the scan. You'll have to deal with the extra "+OK" that the quit outputs.
2) Add
conn.SetReadDeadline(time.Now().Add(time.Second*5))
just before you start scanning. This will cause the scan to stop trying after 5 seconds. Unfortunately, it will always take 5 seconds to complete the scan so choose this time wisely.
I've been using the gocraft-web package so far to do some development on an HTTP service. It's really great because you can stick middleware in it to check for stuff like the presence of a Cookie in the header.
At the moment I am wanting to implement request signing. Getting the client to sign the request is easy enough, but I am wanting to check it for all endpoints with a common piece of middleware. Basically the middleware needs to find the key to check against, compute the request HMAC, and check it against the supplied HMAC (presumably in the Authorization Header).
Computing the actual HMAC is really easy in go.
The problem is: reading the message in middleware makes it unavailable to the final endpoint.
The best solution I have come up with (example shown below) is to read everything from the Request in the middleware and stuffing it back into a bytes.Buffer for later reading. Is there a better way to do this? The current implementation seems a bit hackish.
Reading everything into memory sucks, but I can probably just put my service behind a proxy and limit the size of requests anyways. The actual content will always be pretty small(under 5 kilobytes). The extra copy introduced by this approach is likely to be quite slow, but computing the HMAC of a message is not exactly cheap to begin with.
The advantage to this is that it is transparent: it will work with any other go http code that just expects to read from Request.Body without any magic.
I suppose I could be a bit slicker and use a io.TeeReader.
This is my solution so far. If you post to localhost:3300 some JSON it prints the sha512 to the terminal in the server process, but also the response is able to contain a listing of the keys & values in it.
package main
import "fmt"
import "github.com/gocraft/web"
import "net/http"
import "bytes"
import "crypto/sha512"
import "io"
import "encoding/hex"
import "encoding/json"
type Context struct{}
type echoer struct {
*bytes.Buffer
}
func (e echoer) Close() error {
//Just do nothing to make the interface happy
return nil
}
func middlewareThatLooksAtBody(rw web.ResponseWriter, req *web.Request, next web.NextMiddlewareFunc) {
var replacement echoer
replacement.Buffer = &bytes.Buffer{}
hash := sha512.New()
hash.Write([]byte(req.Method))
reader := req.Body
var bytes []byte = make([]byte, 64)
for {
amount, err := reader.Read(bytes)
fmt.Printf("Read %d bytes\n", amount)
if err != nil {
if err == io.EOF {
break
}
panic(err)
}
if amount == 0 {
break
}
hash.Write(bytes)
replacement.Write(bytes)
}
//Is this needed?
reader.Close()
//replacement.Seek(0, 0)
req.Body = replacement
fmt.Printf("%v\n", hex.EncodeToString(hash.Sum(nil)))
next(rw, req)
}
func echoJson(rw web.ResponseWriter, req *web.Request) {
dec := json.NewDecoder(req.Body)
var obj map[string]interface{}
err := dec.Decode(&obj)
if err != nil {
rw.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(rw, "%v\n", err)
return
}
for k, v := range obj {
fmt.Fprintf(rw, "%v = %v\n", k, v)
}
}
func main() {
router := web.New(Context{})
router.Middleware(middlewareThatLooksAtBody)
router.Post("/", echoJson)
http.ListenAndServe("localhost:3300", router)
}
From your description, it looks like you need to read all the bytes from the request body, regardless of what your handlers will do.
If so, then you have at least a couple of options that would avoid the extra copy:
1) Store the read contents inside your gocraft context.
2) Do all body data processing and validation in the middleware and store the results of the processing in the context.
Granted, this means that your handlers now must know that they should look for the contents in the context instead of the req.Body.
I think it's a decent trade-off though, given your requirements.