goroutine soon blocked the http server when it was requested - http

goroutine sooblocked the http server when it was reqn uested
The following code will soon be blocked
In a device management function, by visiting the http REST ful interface to determine whether the device is online, 30s access to 1000 devices, the current program is roughly as follows to see the number of goroutine is not very high, but soon the program will not Move, cpu, memory is not occupied too high
package main
import (
"fmt"
"net/http"
"runtime"
"time"
)
func a() {
b()
//.....
}
var bb = 0
func b() {
fmt.Printf("b:%d\n", bb)
bb++
resp, err := http.Get("http://www.baidu.com")
if err == nil {
resp.Body.Close()
}
//...
}
func c() {
t := time.NewTicker(time.Second * 30)
for {
fmt.Printf("start time:%s\n", time.Now().Format("15:04:05"))
bb = 0
for i := 0; i < 1000; i++ {
go a()
if i%11 == 0 {
time.Sleep(time.Millisecond * 300)
fmt.Printf("i:%d go:%d\n", i, runtime.NumGoroutine())
}
}
<-t.C
fmt.Printf("over time:%s\n", time.Now().Format("15:04:05"))
}
}
func main() {
go c()
for {
}
}
block
The following code will not blockļ¼ŒThis is why, hope to give me some advice, thank you
package main
import (
"fmt"
"net/http"
"runtime"
"time"
)
func a() {
b()
}
var bb = 0
func b() {
fmt.Printf("b:%d\n", bb)
bb++
resp, err := http.Get("http://www.baidu.com")
if err == nil {
resp.Body.Close()
}
}
func main() {
for {
for {
go b()
time.Sleep(time.Millisecond * 10)
fmt.Printf("go:%d\n", runtime.NumGoroutine())
}
}
no-block

I think there is no switching point.
the Go scheduler is non preemptive. (cooperative)
all goroutines must be cooperative of scheduling
func main() {
go c()
for {
// it is not cooperative
}
}
the Go scheduler can switch only at specific points.
specific points is I/O, chan, Sleep, Gosched
try below code on block example
func main() {
go c()
for {
runtime.Gosched() // or time.Sleep(any)
}
}
I hope this would help you

Related

Must I synchronise read and write operations when doing it from different goroutines?

As I know, every requests to the server creates new goroutine. For ex (probably incorrect code, but this topic is not about it):
package main
import "net/http"
var exampleMap map[string]string
func handlerPost(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case "POST":
{
exampleMap["test"] = test // Must I syncrhonise this writing?
}
case "GET":
{
if v, ok := exampleMap["test"] { // And what about reading?
fmt.Println(v)
}
}
}
}
func main() {
http.HandleFunc("/", handlerPost)
http.ListenAndServe(":8080", nil)
}
Does it mean that its unsafe to do it like this and I have to use sync.Map (for example), and what about if instead map here was a database queries? What can I do in this case. Thank you!
exampleMap is shared among goroutines, so you have to synchronize access to it. A mutex would do, a RWMutex would perform better:
var exampleMap map[string]string
var exampleMutex sync.RWMutex
...
exampleMutex.Lock()
exampleMap["test"] = test
exampleMutex.Unlock()
...
exampleMutex.RLock()
v, ok := exampleMap["test"]
exampleMutex.RUnlock()
if ok {
...
}

how to pass reference of a pointer with parallelism?

I'm writing a bot to run some commands in parallel and at the same time run the bots in parallel, but I'm having trouble starting and pausing functions.
Below I'll leave an example I set up. It was expected that one of the bots would continue to run and others would stop, but all would end up running.
Could someone explain to me why, when using the startbot() command, it does not get bool?
package main
import (
"log"
"time"
)
type botBase struct {
isEnabled bool
}
func (b *botBase) startFunctionX() {
b.isEnabled = true
}
func (b *botBase) pauseFunctionX() {
b.isEnabled = false
}
func (b botBase) runCommandX() {
for {
if b.isEnabled {
log.Print("running...")
} else {
log.Print("paused...")
}
time.Sleep(1 * time.Second)
}
}
type bot struct {
botBase
//other stuffs
}
func (b bot) runAllCommands() {
go b.runCommandX()
//wait parallels commands
for{
time.Sleep(10 * time.Hour)
}
}
type bots struct {
List []bot
}
func (b *bots) loadListDB() {
b1 := bot{}
b1.isEnabled = false
b2 := bot{}
b2.isEnabled = false
b.List = []bot{b1, b2}
}
var myBots bots
func main() {
myBots.loadListDB()
for _, b := range myBots.List {
b.startFunctionX()
go b.runAllCommands()
}
//control stop and start bots
log.Print("expected true = ", myBots.List[0].isEnabled)
myBots.List[0].pauseFunctionX()
log.Print("expected false = ", myBots.List[0].isEnabled)
//wait bots parallels
for {
time.Sleep(10 * time.Hour)
}
}
the range statement returns the value of a bot which is then changed so you're actually checking a different bot.. work with references -
for i := range myBots.List {
b := &myBots.List[i]
b.startFunctionX()
go b.runAllCommands()
}
https://play.golang.org/p/1V8tKx431QJ

go - software panics for concurrent access despite being handled by mutexes

i have a map that is being read and written by 3 goroutines constantly, the program always ends up with a "fatal error: concurrent map iteration and map write" despite me setting up the mutex to protect it, I know I could use sync.Map or I could sync with a channel but I'd really like to understand what I am doing wrong. this is the code:
//book.go
type OrderbookMap map[float64]float64
type Orderbook struct {
Bids OrderbookMap
Asks OrderbookMap
Symbol string
IsInit bool
UpdateId int
mu sync.Mutex
}
func (book *Orderbook) Init() {
book.mu.Lock()
defer book.mu.Unlock()
if book.IsInit {
return
}
book.Asks = make(OrderbookMap)
book.Bids = make(OrderbookMap)
book.IsInit = true
}
//functions with mutexes
func DelBid2(b *Orderbook, price float64) {
b.mu.Lock()
defer b.mu.Unlock()
if _, ok := b.Bids[price]; ok {
delete(b.Bids, price)
} else {
fmt.Printf("VALUE NOT FOUND %v\n", price)
}
}
func AddBid2(b *Orderbook, price float64, qty float64) {
b.mu.Lock()
defer b.mu.Unlock()
b.Bids[price] = qty
}
func GetBids2(b *Orderbook) OrderbookMap {
b.mu.Lock()
defer b.mu.Unlock()
return b.Bids
}
//TesterFile.go
func TestBookRace(t *testing.T) {
var B Orderbook
B.Init()
//add
go func() {
for {
b, q := rFloat(), rFloat()
AddBid2(&B, b, q)
fmt.Printf("ADD %v NEW: %v\n", b, GetBids2(&B))
}
}()
//del
go func() {
for {
b := rFloat()
DelBid2(&B, b)
fmt.Printf("DEL %v NEW: %v\n", b, GetBids2(&B))
}
}()
//read
go func() {
for {
fmt.Printf("READ %v\n", GetBids2(&B))
}
}()
for { time.Sleep(10 * time.Second)}
}

Golang serve static files from memory

I have a quick question about serving files in Go. There is the great timesaving FileServer handler, but for my use case, I only have 2 or 3 files (js and css) that go with my app and I dont want to complicate the deployment to have to think about those.
Do you think there is an easy way to build those couple of files into the binary and serve them from there. For example base64 encode the data of the files as constants and server the files from the constants. This would work in its most simple form, but I dont want to go through the pain of doing everything that a file server does (headers, expiries, mime-types, etc) on my own. So would there be an easy way to bake those static files into the binary in some form and serve them that way?
The FileServer requires a FileSystem object in its constructor. Usually, you would provide something based on http.Dir to make that FileSystem for you from the actual file system, but nothing prevents you from implementing your own:
package main
import "os"
import "time"
import "net/http"
type InMemoryFS map[string]http.File
// Implements FileSystem interface
func (fs InMemoryFS) Open(name string) (http.File, error) {
if f, ok := fs[name]; ok {
return f, nil
}
panic("No file")
}
type InMemoryFile struct {
at int64
Name string
data []byte
fs InMemoryFS
}
func LoadFile(name string, val string, fs InMemoryFS) *InMemoryFile {
return &InMemoryFile{at: 0,
Name: name,
data: []byte(val),
fs: fs}
}
// Implements the http.File interface
func (f *InMemoryFile) Close() error {
return nil
}
func (f *InMemoryFile) Stat() (os.FileInfo, error) {
return &InMemoryFileInfo{f}, nil
}
func (f *InMemoryFile) Readdir(count int) ([]os.FileInfo, error) {
res := make([]os.FileInfo, len(f.fs))
i := 0
for _, file := range f.fs {
res[i], _ = file.Stat()
i++
}
return res, nil
}
func (f *InMemoryFile) Read(b []byte) (int, error) {
i := 0
for f.at < int64(len(f.data)) && i < len(b) {
b[i] = f.data[f.at]
i++
f.at++
}
return i, nil
}
func (f *InMemoryFile) Seek(offset int64, whence int) (int64, error) {
switch whence {
case 0:
f.at = offset
case 1:
f.at += offset
case 2:
f.at = int64(len(f.data)) + offset
}
return f.at, nil
}
type InMemoryFileInfo struct {
file *InMemoryFile
}
// Implements os.FileInfo
func (s *InMemoryFileInfo) Name() string { return s.file.Name }
func (s *InMemoryFileInfo) Size() int64 { return int64(len(s.file.data)) }
func (s *InMemoryFileInfo) Mode() os.FileMode { return os.ModeTemporary }
func (s *InMemoryFileInfo) ModTime() time.Time { return time.Time{} }
func (s *InMemoryFileInfo) IsDir() bool { return false }
func (s *InMemoryFileInfo) Sys() interface{} { return nil }
const HTML = `<html>
Hello world !
</html>
`
const CSS = `
p {
color:red;
text-align:center;
}
`
func main() {
FS := make(InMemoryFS)
FS["foo.html"] = LoadFile("foo.html", HTML, FS)
FS["bar.css"] = LoadFile("bar.css", CSS, FS)
http.Handle("/", http.FileServer(FS))
http.ListenAndServe(":8080", nil)
}
This implementation is very buggy at best, and you should probably never ever use it, but it should show you how the FileSystem interface can be implemented for arbitrary 'files'.
A more credible (and certainly less dangerous) implementation of something similar is available here. This is the one used to fake the filesystem on Go playground, so it should be a good reference (much better than mine anyway).
Whether it is simpler to reimplement this FileSystem interface or a custom FileServer as other suggested, is entirely up to you and your project ! I suspect however that for serving a couple of predefined files, rewriting the serving part might be easier than emulating a full file-system.
The "go.rice" package takes care of this for you - embedding resources in your binaries, and providing an http.FileSystem implementation.
It is not very difficult to do what you request. You don't have to base64 encode it or anything (it will just make it harder for you to edit.).
Below is an example of how to output a javascript file with correct mime type:
package main
import (
"fmt"
"log"
"net/http"
)
const jsFile = `alert('Hello World!');`
func main() {
http.HandleFunc("/file.js", JsHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func JsHandler(w http.ResponseWriter, r *http.Request) {
// Getting the headers so we can set the correct mime type
headers := w.Header()
headers["Content-Type"] = []string{"application/javascript"}
fmt.Fprint(w, jsFile)
}
I would store the files in variable as plain text.
Something like this:
package main
import (
"fmt"
"log"
"net/http"
)
var files = map[string]string{}
func init() {
files["style.css"] = `
/* css file content */
body { background-color: pink; }
`
}
func init() {
files["index.html"] = `
<!-- Html content -->
<html><head>
<link rel="stylesheet" type="text/css" href="style.css">
</head><body>Hello world!</body></html>
`
}
func main() {
for fileName, content := range files {
contentCpy := content
http.HandleFunc("/"+fileName, func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%s\n", contentCpy)
})
}
log.Fatal(http.ListenAndServe(":8080", nil))
}
That way, it is pretty easy to have your makefile or build script so something like:
for file in index.html style.css; do echo "package main\nfunc init() { files[\"$file\"] = \`$(cat $file)\` }" | gofmt -s > $file.go; done; go build && ./httptest

Unable to send gob data over TCP in Go Programming

I have a client server application, using TCP connection
Client:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
//read M and N
...
tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
...
var p P
p.M = M
p.N = N
err = enc.Encode(p)
}
Server:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
tcpAddr, err := net.ResolveTCPAddr("ip4", service)
listener, err := net.ListenTCP("tcp", tcpAddr)
...
var connB bytes.Buffer
dec := gob.NewDecoder(&connB)
var p P
err = dec.Decode(p)
fmt.Printf("{%d, %d}\n", p.M, p.N)
}
The result on serve is {0, 0} because I don't know how to obtain a bytes.Buffer variable from net.Conn.
Is there any way for sending gob variables over TCP ?
If true, how can this be done ? Or there are any alternative in sending numbers over TCP ?
Any help or sample code would really be appreciated.
Here's a complete example.
Server:
package main
import (
"fmt"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func handleConnection(conn net.Conn) {
dec := gob.NewDecoder(conn)
p := &P{}
dec.Decode(p)
fmt.Printf("Received : %+v", p);
conn.Close()
}
func main() {
fmt.Println("start");
ln, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
for {
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
// handle error
continue
}
go handleConnection(conn) // a goroutine handles conn so that the loop can accept other connections
}
}
Client :
package main
import (
"fmt"
"log"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func main() {
fmt.Println("start client");
conn, err := net.Dial("tcp", "localhost:8080")
if err != nil {
log.Fatal("Connection error", err)
}
encoder := gob.NewEncoder(conn)
p := &P{1, 2}
encoder.Encode(p)
conn.Close()
fmt.Println("done");
}
Launch the server, then the client, and you see the server displaying the received P value.
A few observations to make it clear :
When you listen on a socket, you should pass the open socket to a goroutine that will handle it.
Conn implements the Reader and Writer interfaces, which makes it easy to use : you can give it to a Decoder or Encoder
In a real application you would probably have the P struct definition in a package imported by both programs

Resources