i have a map that is being read and written by 3 goroutines constantly, the program always ends up with a "fatal error: concurrent map iteration and map write" despite me setting up the mutex to protect it, I know I could use sync.Map or I could sync with a channel but I'd really like to understand what I am doing wrong. this is the code:
//book.go
type OrderbookMap map[float64]float64
type Orderbook struct {
Bids OrderbookMap
Asks OrderbookMap
Symbol string
IsInit bool
UpdateId int
mu sync.Mutex
}
func (book *Orderbook) Init() {
book.mu.Lock()
defer book.mu.Unlock()
if book.IsInit {
return
}
book.Asks = make(OrderbookMap)
book.Bids = make(OrderbookMap)
book.IsInit = true
}
//functions with mutexes
func DelBid2(b *Orderbook, price float64) {
b.mu.Lock()
defer b.mu.Unlock()
if _, ok := b.Bids[price]; ok {
delete(b.Bids, price)
} else {
fmt.Printf("VALUE NOT FOUND %v\n", price)
}
}
func AddBid2(b *Orderbook, price float64, qty float64) {
b.mu.Lock()
defer b.mu.Unlock()
b.Bids[price] = qty
}
func GetBids2(b *Orderbook) OrderbookMap {
b.mu.Lock()
defer b.mu.Unlock()
return b.Bids
}
//TesterFile.go
func TestBookRace(t *testing.T) {
var B Orderbook
B.Init()
//add
go func() {
for {
b, q := rFloat(), rFloat()
AddBid2(&B, b, q)
fmt.Printf("ADD %v NEW: %v\n", b, GetBids2(&B))
}
}()
//del
go func() {
for {
b := rFloat()
DelBid2(&B, b)
fmt.Printf("DEL %v NEW: %v\n", b, GetBids2(&B))
}
}()
//read
go func() {
for {
fmt.Printf("READ %v\n", GetBids2(&B))
}
}()
for { time.Sleep(10 * time.Second)}
}
Related
I'm writing a bot to run some commands in parallel and at the same time run the bots in parallel, but I'm having trouble starting and pausing functions.
Below I'll leave an example I set up. It was expected that one of the bots would continue to run and others would stop, but all would end up running.
Could someone explain to me why, when using the startbot() command, it does not get bool?
package main
import (
"log"
"time"
)
type botBase struct {
isEnabled bool
}
func (b *botBase) startFunctionX() {
b.isEnabled = true
}
func (b *botBase) pauseFunctionX() {
b.isEnabled = false
}
func (b botBase) runCommandX() {
for {
if b.isEnabled {
log.Print("running...")
} else {
log.Print("paused...")
}
time.Sleep(1 * time.Second)
}
}
type bot struct {
botBase
//other stuffs
}
func (b bot) runAllCommands() {
go b.runCommandX()
//wait parallels commands
for{
time.Sleep(10 * time.Hour)
}
}
type bots struct {
List []bot
}
func (b *bots) loadListDB() {
b1 := bot{}
b1.isEnabled = false
b2 := bot{}
b2.isEnabled = false
b.List = []bot{b1, b2}
}
var myBots bots
func main() {
myBots.loadListDB()
for _, b := range myBots.List {
b.startFunctionX()
go b.runAllCommands()
}
//control stop and start bots
log.Print("expected true = ", myBots.List[0].isEnabled)
myBots.List[0].pauseFunctionX()
log.Print("expected false = ", myBots.List[0].isEnabled)
//wait bots parallels
for {
time.Sleep(10 * time.Hour)
}
}
the range statement returns the value of a bot which is then changed so you're actually checking a different bot.. work with references -
for i := range myBots.List {
b := &myBots.List[i]
b.startFunctionX()
go b.runAllCommands()
}
https://play.golang.org/p/1V8tKx431QJ
goroutine sooblocked the http server when it was reqn uested
The following code will soon be blocked
In a device management function, by visiting the http REST ful interface to determine whether the device is online, 30s access to 1000 devices, the current program is roughly as follows to see the number of goroutine is not very high, but soon the program will not Move, cpu, memory is not occupied too high
package main
import (
"fmt"
"net/http"
"runtime"
"time"
)
func a() {
b()
//.....
}
var bb = 0
func b() {
fmt.Printf("b:%d\n", bb)
bb++
resp, err := http.Get("http://www.baidu.com")
if err == nil {
resp.Body.Close()
}
//...
}
func c() {
t := time.NewTicker(time.Second * 30)
for {
fmt.Printf("start time:%s\n", time.Now().Format("15:04:05"))
bb = 0
for i := 0; i < 1000; i++ {
go a()
if i%11 == 0 {
time.Sleep(time.Millisecond * 300)
fmt.Printf("i:%d go:%d\n", i, runtime.NumGoroutine())
}
}
<-t.C
fmt.Printf("over time:%s\n", time.Now().Format("15:04:05"))
}
}
func main() {
go c()
for {
}
}
block
The following code will not blockļ¼This is why, hope to give me some advice, thank you
package main
import (
"fmt"
"net/http"
"runtime"
"time"
)
func a() {
b()
}
var bb = 0
func b() {
fmt.Printf("b:%d\n", bb)
bb++
resp, err := http.Get("http://www.baidu.com")
if err == nil {
resp.Body.Close()
}
}
func main() {
for {
for {
go b()
time.Sleep(time.Millisecond * 10)
fmt.Printf("go:%d\n", runtime.NumGoroutine())
}
}
no-block
I think there is no switching point.
the Go scheduler is non preemptive. (cooperative)
all goroutines must be cooperative of scheduling
func main() {
go c()
for {
// it is not cooperative
}
}
the Go scheduler can switch only at specific points.
specific points is I/O, chan, Sleep, Gosched
try below code on block example
func main() {
go c()
for {
runtime.Gosched() // or time.Sleep(any)
}
}
I hope this would help you
Is it safe to range map without locking if multiple goroutines will run notifyAll func? Actually in a range I need to sometimes remove entries from a map.
var mu sync.RWMutex
func (self *Server) notifyAll(event *Event)
ch := make(chan int, 64)
num := 0
for k, v := range self.connections {
num++
ch <- num
go func(int k, conn *Conn) {
err := conn.sendMessage(event)
<-ch
if err != nil {
self.removeConn(k)
}
}(k, v)
}
}
func (self *Server) removeConn(int k) {
mu.Lock()
defer mu.Unlock()
delete(self.connections, k)
}
// Somewhere in another goroutine
func (self *Server) addConn(conn *Conn, int k) {
mu.Lock()
defer mu.Unlock()
self.connections[k] = conn
}
Or I must RLock map before range?
func (self *Server) notifyAll(event *Event)
mu.RLock()
defer mu.RUnlock()
// Skipped previous body...
}
Short answer: maps are not concurrent-safe (one can still say thread-safe) in Go.
So, if you need to access a map from different go-routines, you must employ some form of access orchestration, otherwise "uncontrolled map access can crash the program" (see this).
Edit:
This is another implementation (without considering housekeeping concerns - timeouts, quit, log, etc) which ignores the mutex all-together and uses a more Goish approach (this is just for demonstrating this approach which helps us to clear access orchestration concerns - might be right or not for your case):
type Server struct {
connections map[*Conn]struct{}
_removeConn, _addConn chan *Conn
_notifyAll chan *Event
}
func NewServer() *Server {
s := new(Server)
s.connections = make(map[*Conn]struct{})
s._addConn = make(chan *Conn)
s._removeConn = make(chan *Conn, 1)
s._notifyAll = make(chan *Event)
go s.agent()
return s
}
func (s *Server) agent() {
for {
select {
case c := <-s._addConn:
s.connections[c] = struct{}{}
case c := <-s._removeConn:
delete(s.connections, c)
case e := <-s._notifyAll:
for c := range s.connections {
closure := c
go func() {
err := closure.sendMessage(e)
if err != nil {
s._removeConn <- closure
}
}()
}
}
}
}
func (s *Server) removeConn(c *Conn) {
s._removeConn <- c
}
func (s *Server) addConn(c *Conn) {
s._addConn <- c
}
Edit:
I stand corrected; according to Damian Gryski maps are safe for concurrent reads. The reason that the map order changes on each iteration is "the random seed chosen for map iteration order, which is local to the goroutine iterating" (another tweet of him). This fact does not affect the first edit and suggested solution.
I am working with golang's pointers the way I did with c++, but it seems not to work, which would be the right way to do it? or what am I doing wrong?, Thanks.
ftw I'm doing AsyncBinaryTrees.
type Obj interface {
Compare(node Obj) int
}
type Tree struct {
Item Obj
Rigth, Left *Tree
height int16
}
func Insert(t *Tree, item Obj) chan struct{} {
done := make(chan struct{}, 1)
go insert(t, item, done)
return done
}
func insert(t *Tree, item Obj, done chan struct{}) {
if t == nil {
t = &Tree{Item: nil, Rigth: nil, Left: nil, height: 0}
var signal struct{}
done <- signal
close(done)
} else {
if t.Item.Compare(item) == 1 { //Left
insert(t.Left, item, done)
} else if t.Item.Compare(item) == -1 { //Rigth
insert(t.Right, item, done)
} else {
close(done)
panic
}
}
}
//=== testing
func assertSignal(ch_signal chan struct{}, t *testing.T) {
_, done := <-ch_signal
if !done {
t.Error("Error: it should send a signal of empty struct")
}
}
func TestInsertion(t *testing.T) {
var tree *Tree
ch_signal := Insert(tree, newObjInt())
fmt.PrintLn(t) //=> <nil>
assertSignal(ch_signal, t) //=>PASS
ch_signal = Insert(tree, newObjInt())
fmt.PrintLn(t) //=> <nil>
assertSignal(ch_signal, t) //=>PASS
ch_signal = Insert(tree, newObjInt())
fmt.PrintLn(t) //=> <nil>
assertSignal(ch_signal, t) //=>PASS
ch_signal = Insert(tree, newObjInt())
assertSignal(ch_signal, t) //=>PASS
}
nil
nil
nil
TEST PASS
In your insert function you have:
func insert(t *Tree, item Obj, done chan struct{}) {
if t == nil {
t = &Tree{Item: nil, Rigth: nil, Left: nil, height: 0}
...
}
This updates the local variable t, but will not change the variable passed in the calling scope since Go passes function parameters by value. So when you make the following call:
insert(t.Left, item, done)
if t.Left is nil, its value will not be changed by the function call. If you do want it to update the variable, you'll need to define the function argument as t **Tree, change references to set *t instead, and change the call to:
insert(&t.Left, item, done)
There is no equivalent to C++'s syntax for passing function arguments by reference: instead you need to be explicit when passing pointers.
I have a client server application, using TCP connection
Client:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
//read M and N
...
tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
...
var p P
p.M = M
p.N = N
err = enc.Encode(p)
}
Server:
type Q struct {
sum int64
}
type P struct {
M, N int64
}
func main() {
...
tcpAddr, err := net.ResolveTCPAddr("ip4", service)
listener, err := net.ListenTCP("tcp", tcpAddr)
...
var connB bytes.Buffer
dec := gob.NewDecoder(&connB)
var p P
err = dec.Decode(p)
fmt.Printf("{%d, %d}\n", p.M, p.N)
}
The result on serve is {0, 0} because I don't know how to obtain a bytes.Buffer variable from net.Conn.
Is there any way for sending gob variables over TCP ?
If true, how can this be done ? Or there are any alternative in sending numbers over TCP ?
Any help or sample code would really be appreciated.
Here's a complete example.
Server:
package main
import (
"fmt"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func handleConnection(conn net.Conn) {
dec := gob.NewDecoder(conn)
p := &P{}
dec.Decode(p)
fmt.Printf("Received : %+v", p);
conn.Close()
}
func main() {
fmt.Println("start");
ln, err := net.Listen("tcp", ":8080")
if err != nil {
// handle error
}
for {
conn, err := ln.Accept() // this blocks until connection or error
if err != nil {
// handle error
continue
}
go handleConnection(conn) // a goroutine handles conn so that the loop can accept other connections
}
}
Client :
package main
import (
"fmt"
"log"
"net"
"encoding/gob"
)
type P struct {
M, N int64
}
func main() {
fmt.Println("start client");
conn, err := net.Dial("tcp", "localhost:8080")
if err != nil {
log.Fatal("Connection error", err)
}
encoder := gob.NewEncoder(conn)
p := &P{1, 2}
encoder.Encode(p)
conn.Close()
fmt.Println("done");
}
Launch the server, then the client, and you see the server displaying the received P value.
A few observations to make it clear :
When you listen on a socket, you should pass the open socket to a goroutine that will handle it.
Conn implements the Reader and Writer interfaces, which makes it easy to use : you can give it to a Decoder or Encoder
In a real application you would probably have the P struct definition in a package imported by both programs