I was making a WaitForResponse function for my Discord bot, and it works, but the user can still use commands even when the bot is expecting a response. I combated this by using a map with the user and channel IDs, but I was then hit with the dreaded fatal error: concurrent map read and write. So I tried using a sync.Map, however it wouldn't always work when I spammed the command. I could sometimes still run commands when the bot was expecting a response. Is there any way I can ensure that the values are getting added and removed from the map when and as they should?
For these scenarios, sync.Mutex can be used to ensure that only one modification is allowed by acquiring a lock around the code that you want to be thread-safe.
var mu sync.Mutex
func readMap(key string) {
mu.Lock()
defer mu.Unlock()
return yourMap[key]
}
func updateMap(key, value string) {
mu.Lock()
defer mu.Unlock()
yourMap[key] = value
}
Mutex ensures that ONLY ONE goroutine can is allowed access to the locked code, which means for your case, only one operation, either read or write can be performed.
For better efficiency, you should consider using sync.RWMutex since you might not want to lock the map when it's being read. From GoDoc:
A RWMutex is a reader/writer mutual exclusion lock. The lock can be held by an arbitrary number of readers or a single writer. The zero value for a RWMutex is an unlocked mutex.
var mu sync.RWMutex
func readMap(key string) {
mu.RLock()
defer mu.RUnlock()
return yourMap[key]
}
func updateMap(key, value string) {
mu.Lock()
defer mu.Unlock()
yourMap[key] = value
}
Related
Why is the res variable always false even though the parameters are correct?
fun isValidUser(_studentEmail: String, _password: String): Boolean {
var res = false
userList.get().addOnSuccessListener { result ->
for (document in result) {
val studentEmail = document.data["studentEmail"] as String
val password = document.data["password"] as String
if (_studentEmail == studentEmail && _password == password) {
res = true
}
}
}
return res
}
This doesn't behave like you think it does.
The userList.get() launches an asynchronous task, which means the program will continue its execution without waiting for the task to be done.
addOnSuccessListener only registers a listener - a piece of code to run when the query successfully completes, but that piece of code will only be run in the future, not right away, because the query takes some time.
When the program reaches the return res, the query still hasn't completed and the listener hasn't been executed yet.
If you want to reason more sequentially about this stuff, you can use Kotlin coroutine extensions to await() the result instead of registering a listener. Check out kotlinx-coroutines-play-services for instance. However, if you have trouble understanding asynchronous behaviour, you should probably read up more about this before diving into coroutines.
I have the following struct that contains channels and a map for storage of data. I want to be able to pass that struct into functions in order to make use of those channels so that once they are triggered/have incoming messages, to use them in order to update the map that is associated with it.
I understand that maps by default are passed by reference when sent to various functions. Would this be the same case even when they are contained within a custom struct? How do i make sure that my entire struct is passed around to functions by reference in order to update Storage and also make use of its channels?
type CustomStrct struct {
Storage map[string]string
RetrieveChannel chan string
InsertChannel chan string
}
This is a constructor I have created for initialising a new instance of the struct:
func InitializeNewStore() CustomStrct {
newCustomStruct := CustomStrct {
Storage: make(map[string]string),
RetrieveChannel: make(chan Request),
InsertChannel: make(chan Request),
}
return newCustomStruct
}
Slices, maps and channels are pointer-like values in Go: copying a struct containing a channel copies a reference to the channel, not the channel itself:
a := CustomStrct{
RetrieveChannel: make(chan Request),
}
b := a
log.Println(a.RetrieveChannel == b.RetrieveChannel) // logs true
So it's quite fine to pass your struct either by value or by reference.
If you need to ensure that go vet will flag attempts to pass your struct by value, the simplest solution is to embed a sync.Mutex inside the struct:
type CustomStrct struct {
mu sync.Mutex
...
}
You don't need to actually use the mutex: just having it embedded in the struct will cause go vet to complain whenever you attempt to pass it by value.
I have struct that uses sync.Pool.
Is it safe to use this reference as context value?
type User struct {
ID string
}
var userPool = sync.Pool{
New: func() interface{} {
return &User{}
},
}
func getUser() *User {
return userPool.Get().(*User)
}
func recycleUser(user *User) {
userPool.Put(user)
}
The user struct is retrieved from pool in middleware.
func middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// get user from pool
user := getUser()
// user must be recycled
ctx := context.WithValue(r.Context(), "user", user)
}
}
And recycled in handler.
func getUser(w http.ResponseWriter, r *http.Request) {
user := r.Context().Value("user").(*User)
// TODO: do something with user
// put user struct back into pool
recycleUser(user)
}
Edit:
My question is more about how context deals with pointer to my object. Does it make a copy? Is it safe to use non-primitive object with context?
There are three points to make:
Usage of sync.Pool
Your usage of pool is as intended. From golang Documentation
A Pool is safe for use by multiple goroutines simultaneously.
Retrieving users
getUser uses func (p *Pool) Get() interface{} which removes the returned item from the pool, afterwards you can do whatever you please
with that value. In your case, thats a *User. Whether the associated
ressource is safe to use, well, depends on your usage of those values in the rest of the program.
Recycling users
Calling recycleUser within the hander is potentially dangerous, depending on your environment.
What could possibly go wrong?
After recycleUser returned your *User back to the pool, it could be retrieved and used by a different goroutine immediately. But at the same time, *User is still stored in the context assiociated with the request. So it depends if any of the functions of your middleware use the *User from the context, too and wether they store the *User value. Or if you later add some code after the recycleUser, which uses the *User value. After the recycleUser call, all those usages probably operate on the wrong user, which is already used by some different request.
How to solve those issues
coding convention
if you retrieve the user in middleware functions, don't store it
if you retrieve the user in middleware functions, don't use it after the call to the next handler in the chain.
don't call recycleUser in middleware functions
retrieve the user in the leaf handler (last handler in the chain, doesn't calls any further handlers propagating the context and thus the *User) and use defer recycleUser(user) to put back the *User in the pool and make it impossible that later added code uses *User after the call to recycleUser(user).
in the leaf handler, don't use *User in code which is called by some defer usage in the leaf handler, other than recycleUser.
ensuring the coding convention by technical means. My only idea here would be to place the *User inside some struct and only use that struct for one request and mark it as empty, once *User is put back into the pool. Make all access methods of that struct check for emptyness and panic/log/whatever, if an empty struct is accessed. Place a pointer to that struct in the context. But that is probably pointless, as you now allocate that struct for each request, instead of a User, which probably was, what you tried to avoid.
A minor remark
You probably meant your middleware function to read like:
func middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// get user from pool
user := getUser()
// user must be recycled
ctx := context.WithValue(r.Context(), "user", user)
// probably do something more
next(w, r.WithContext(ctx))
}
}
Say I have this http handler:
func SomeHandler(w http.ResponseWriter, r *http.Request) {
data := GetSomeData()
_, err := w.Write(data)
}
Should I check the error returned by w.Write? Examples I've seen just ignore it and do nothing. Also, functions like http.Error() do not return an error to be handled.
It's up to you. My advice is that unless the documentation of some method / function explicitly states that it never returns a non-nil error (such as bytes.Buffer.Write()), always check the error and the least you can do is log it, so if an error occurs, it will leave some mark which you can investigate should it become a problem later.
This is also true for writing to http.ResponseWriter.
You might think ResponseWriter.Write() may only return errors if sending the data fails (e.g. connection closed), but that is not true. The concrete type that implements http.ResponseWriter is the unexported http.response type, and if you check the unexported response.write() method, you'll see it might return a non-nil error for a bunch of other reasons.
Reasons why ResponseWriter.Write() may return a non-nil error:
If the connection was hijacked (see http.Hijacker): http.ErrHijacked
If content length was specified, and you attempt to write more than that: http.ErrContentLength
If the HTTP method and / or HTTP status does not allow a response body at all, and you attempt to write more than 0 bytes: http.ErrBodyNotAllowed
If writing data to the actual connection fails.
Even if you can't do anything with the error, logging it may be of great help debugging the error later on. E.g. you (or someone else in the handler chain) hijacked the connection, and you attempt to write to it later; you get an error (http.ErrHijacked), logging it will reveal the cause immediately.
Tip for "easy" logging errors
If you can't do anything with the occasional error and it's not a "showstopper", you may create and use a simple function that does the check and logging, something like this:
func logerr(n int, err error) {
if err != nil {
log.Printf("Write failed: %v", err)
}
}
Using it:
logerr(w.Write(data))
Tip for "auto-logging" errors
If you don't even want to use the logerr() function all the time, you may create a wrapper for http.ResponseWriter which does this "automatically":
type LogWriter struct {
http.ResponseWriter
}
func (w LogWriter) Write(p []byte) (n int, err error) {
n, err = w.ResponseWriter.Write(p)
if err != nil {
log.Printf("Write failed: %v", err)
}
return
}
Using it:
func SomeHandler(w http.ResponseWriter, r *http.Request) {
w = LogWriter{w}
w.Write([]byte("hi"))
}
Using LogWriter as a wrapper around http.ResponseWriter, should writes to the original http.ResponseWriter fail, it will be logged automatically.
This also has the great benefit of not expecting a logger function to be called, so you can pass a value of your LogWriter "down" the chain, and everyone who attempts to write to it will be monitored and logged, they don't have to worry or even know about this.
But care must be taken when passing LogWriter down the chain, as there's also a downside to this: a value of LogWriter will not implement other interfaces the original http.ResponseWriter might also do, e.g. http.Hijacker or http.Pusher.
Here's an example on the Go Playground that shows this in action, and also shows that LogWriter will not implement other interfaces; and also shows a way (using 2 "nested" type assertions) how to still get out what we want from LogWriter (an http.Pusher in the example).
I want to add to #icza solution. You don't need to create logging structure, you can use simple function:
func logWrite(write func([]byte) (int, error), body []byte) {
_, err := write(body)
if err != nil {
log.Printf("Write failed: %v", err)
}
}
Take a look on that approach based on the #icza code: https://play.golang.org/p/PAetVixCgv4
How do you recover from a runtime panic on a "concurrent map read and map write"? The usual defer with recover doesn't seem to work. Why is that?
I know that you are not supposed to use maps in concurrent contexts, but still: how to recover here?
Example:
package main
import "time"
var m = make(map[string]string)
func main() {
go func() {
for {
m["x"] = "foo"
}
}()
go func() {
for {
m["x"] = "foo"
}
}()
time.Sleep(1 * time.Second)
}
Please add recovery code. :)
Recovering doesn't work here because what you experience is not a panicing state.
Go 1.6 added a lightweight concurrent misuse of maps detection to the runtime:
The runtime has added lightweight, best-effort detection of concurrent misuse of maps. As always, if one goroutine is writing to a map, no other goroutine should be reading or writing the map concurrently. If the runtime detects this condition, it prints a diagnosis and crashes the program. The best way to find out more about the problem is to run the program under the race detector, which will more reliably identify the race and give more detail.
What you experience is an intentional crash by the runtime, it's not the result of a panic() call that a recover() call in a deferred function could stop.
There's nothing you can do to stop that except prevent the concurrent misuse of maps. If you would leave your app like that and it wouldn't crash, you could experience mysterious, undefined behavior at runtime.
Do not recover, guard your code with mutexes form package sync.
package main
import (
"sync"
"time"
)
var m = make(map[string]string)
var l = sync.Mutex{}
func main() {
go func() {
for {
l.Lock()
m["x"] = "foo"
l.Unlock()
}
}()
go func() {
for {
l.Lock()
m["x"] = "foo"
l.Unlock()
}
}()
time.Sleep(1 * time.Second)
}