When ranging over a map m that has concurrent writers, including ones that could delete from the map, is it not thread-safe to do this?:
for k, v := range m { ... }
I'm thinking to be thread-safe I need to prevent other possible writers from changing the value v while I'm reading it, and (when using a mutex and because locking is a separate step) verify that the key k is still in the map. For example:
for k := range m {
m.mutex.RLock()
v, found := m[k]
m.mutex.RUnlock()
if found {
... // process v
}
}
(Assume that other writers are write-locking m before changing v.) Is there a better way?
Edit to add: I'm aware that maps aren't thread-safe. However, they are thread-safe in one way, according to the Go spec at http://golang.org/ref/spec#For_statements (search for "If map entries that have not yet been reached are deleted during iteration"). This page indicates that code using range needn't be concerned about other goroutines inserting into or deleting from the map. My question is, does this thread-safe-ness extend to v, such that I can get v for reading only using only for k, v := range m and no other thread-safe mechanism? I created some test code to try to force an app crash to prove that it doesn't work, but even running blatantly thread-unsafe code (lots of goroutines furiously modifying the same map value with no locking mechanism in place) I couldn't get Go to crash!
No, map operations are not atomic/thread-safe, as the commenter to your question pointed to the golang FAQ “Why are map operations not defined to be atomic?”.
To secure your accessing it, you are encouraged to use Go's channels as a means of resource access token. The channel is used to simply pass around a token. Anyone wanting to modify it will request so from the channel - blocking or non-blocking. When done with working with the map it passes the token back to the channel.
Iterating over and working with the map should be sufficiently simple and short, so you should be ok using just one token for full access.
If that is not the case, and you use the map for more complex stuff/a resource consumer needs more time with it, you may implement a reader- vs writer-access-token. So at any given time, only one writer can access the map, but when no writer is active the token is passed to any number of readers, who will not modify the map (thus they can read simultaneously).
For an introduction to channels, see the Effective Go docs on channels.
You could use concurrent-map to handle the concurrency pains for you.
// Create a new map.
map := cmap.NewConcurretMap()
// Add item to map, adds "bar" under key "foo"
map.Add("foo", "bar")
// Retrieve item from map.
tmp, ok := map.Get("foo")
// Checks if item exists
if ok == true {
// Map stores items as interface{}, hence we'll have to cast.
bar := tmp.(string)
}
// Removes item under key "foo"
map.Remove("foo")
Related
We want to remove elements from Map bin based on size. There will be multiple threads which will try to do above operation. So writing an UDF to do this operation will make it synchronized between threads. But remove_by_rank_range is not working inside lua. Below is the error iwe are getting:
attempt to call field 'remove_by_rank_range' (a nil value)
sample lua code:
function delete(rec)
local testBinMap = rec.testBin
map.remove_by_rank_range(testBinMap, 0, 5)
end
The Lua map API does not include most of the operations of the Map data type, as implemented in the clients (for example, the Java client's MapOperation class).
The performance of the native map operations is significantly higher, so why would you use a UDF here, instead of calling remove_by_rank_range from the client?
The next thing to be aware of is that any write operation, whether it's a UDF or a client calling the map remove_by_rank_range method, first grabs a lock on the record. I answered another stackoverflow question about this request flow. Your UDF doesn't give any advantage to the problem you described over the client map operation.
If you want to cap the size of your map you should be doing it at the very same time you're adding new elements to the map. The two operations would be wrapped together with operate() - an insert, followed by the remove. I have an example of how to do this in rbotzer/aerospike-cdt-examples.
I have a node on firebase that lists all the players in the game. This list will update as and when new players join. And when the current user ( me ) disconnects, I would like to remove myself from the list.
As the list will change over time, at the moment I disconnect, I would like to update this list and update firebase.
This is the way I am thinking of doing it, but it doesn't work as .update doesnt accept a function. Only the object. But if I create the object beforehand, when .onDisconnect calls, it will not be the latest object... How should I go about doing this?
payload.onDisconnect().update( () => {
const withoutMe = state.roomObj
const index = withoutMe.players.indexOf( state.userObj.name )
if ( index > -1 ) {
withoutMe.players.splice( index, 1 )
}
return withoutMe
})
The onDisconnect handler was made for this use-case. But it requires that the data of the write operation is known at the time that you set the onDisconnect. If you think about it, this should make sense: since the onDisconnect happens after your client is disconnected, the data of the data of that write operation must be known before the disconnect.
It sounds like you're building a so-called presence system: a list that contains a node for each user that is currently online. The Firebase documentation has an example of such a presence system. The key difference from your approach is that it in the documentation each user only modifies their own node.
So: when the user comes online, they write a node for themselves. And then when they get disconnected, that node gets removed. Since all users write their node under the same parent, that parent will reflect the users that are online.
The actual implementation is a bit more involved since it deals with some edge cases too. So I recommend you check out the code in the documentation I linked, and use that as the basis for your own similar system.
I am trying to implement a set of functions in go. The context is an event server; I would like to prevent (or at least warn) adding the same handler more than once for an event.
I have read that maps are idiomatic to use as sets because of the ease of checking for membership:
if _, ok := set[item]; ok {
// don't add item
} else {
// do add item
}
I'm having some trouble with using this paradigm for functions though. Here is my first attempt:
// this is not the actual signature
type EventResponse func(args interface{})
type EventResponseSet map[*EventResponse]struct{}
func (ers EventResponseSet) Add(r EventResponse) {
if _, ok := ers[&r]; ok {
// warn here
return
}
ers[&r] = struct{}{}
}
func (ers EventResponseSet) Remove(r EventResponse) {
// if key is not there, doesn't matter
delete(ers, &r)
}
It is clear why this doesn't work: functions are not reference types in Go, though some people will tell you they are. I have proof, though we shouldn't need it since the language specification says that everything other than maps, slices, and pointers are passed by value.
Attempt 2:
func (ers EventResponseSet) Add(r *EventResponse) {
// ...
}
This has a couple of problems:
Any EventResponse has to be declared like fn := func(args interface{}){} because you can't address functions declared in the usual manner.
You can't pass a closure at all.
Using a wrapper is not an option because any function passed to the wrapper will get a new address from the wrapper - no function will be uniquely identifiable by address, and all this careful planning is for nought.
Is it silly of me to not accept defining functions as variables as a solution? Is there another (good) solution?
To be clear, I accept that there are cases that I can't catch (closures), and that's fine. The use case that I envision is defining a bunch of handlers and being relatively safe that I won't accidentally add one to the same event twice, if that makes sense.
You could use reflect.Value presented by Uvelichitel, or the function address as a string acquired by fmt.Sprint() or the address as uintptr acquired by reflect.Value.Pointer() (more in the answer How to compare 2 functions in Go?), but I recommend against it.
Since the language spec does not allow to compare function values, nor does it allow to take their addresses, you have no guarantee that something that works at a time in your program will work always, including a specific run, and including different (future) Go compilers. I would not use it.
Since the spec is strict about this, this means compilers are allowed to generate code that would for example change the address of a function at runtime (e.g. unload an unused function, then load it again later if needed again). I don't know about such behavior currently, but this doesn't mean that a future Go compiler will not take advantage of such thing.
If you store a function address (in whatever format), that value does not count as keeping the function value anymore. And if no one else would "own" the function value anymore, the generated code (and the Go runtime) would be "free" to modify / relocate the function (and thus changing its address) – without violating the spec and Go's type safety. So you could not be rightfully angry at and blame the compiler, but only yourself.
If you want to check against reusing, you could work with interface values.
Let's say you need functions with signature:
func(p ParamType) RetType
Create an interface:
type EventResponse interface {
Do(p ParamType) RetType
}
For example, you could have an unexported struct type, and a pointer to it could implement your EventResponse interface. Make an exported function to return the single value, so no new values may be created.
E.g.:
type myEvtResp struct{}
func (m *myEvtResp) Do(p ParamType) RetType {
// Your logic comes here
}
var single = &myEvtResp{}
func Get() EventResponse { return single }
Is it really needed to hide the implementation in a package, and only create and "publish" a single instance? Unfortunately yes, because else you could create other value like &myEvtResp{} which may be different pointers still having the same Do() method, but the interface wrapper values might not be equal:
Interface values are comparable. Two interface values are equal if they have identical dynamic types and equal dynamic values or if both have value nil.
[...and...]
Pointer values are comparable. Two pointer values are equal if they point to the same variable or if both have value nil. Pointers to distinct zero-size variables may or may not be equal.
The type *myEvtResp implements EventResponse and so you can register a value of it (the only value, accessible via Get()). You can have a map of type map[EventResponse]bool in which you may store your registered handlers, the interface values as keys, and true as values. Indexing a map with a key that is not in the map yields the zero value of the value type of the map. So if the value type of the map is bool, indexing it with a non-existing key will result in false – telling it's not in the map. Indexing with an already registered EventResponse (an existing key) will result in the stored value – true – telling it's in the map, it's already registered.
You can simply check if one already been registered:
type EventResponseSet map[*EventResponse]bool
func (ers EventResponseSet) Add(r EventResponse) {
if ers[r] {
// warn here
return
}
ers[r] = true
}
Closing: This may seem a little too much hassle just to avoid duplicated use. I agree, and I wouldn't go for it. But if you want to...
Which functions you mean to be equal? Comparability is not defined for functions types in language specification. reflect.Value gives you the desired behaviour more or less
type EventResponseSet map[reflect.Value]struct{}
set := make(EventResponseSet)
if _, ok := set[reflect.ValueOf(item)]; ok {
// don't add item
} else {
// do add item
set[reflect.ValueOf(item)] = struct{}{}
}
this assertion will treat as equal items produced by assignments only
//for example
item1 := fmt.Println
item2 := fmt.Println
item3 := item1
//would have all same reflect.Value
but I don't think this behaviour guaranteed by any documentation.
How does map in Golang compare keys? For some reason, I need to have a struct as a key, which has 2 values inside. I want map to compare by only first value, not second. Second is for my usage. Like in java, I can customise equals method, so map will take only logically equal keys within. Is there any way to do that?
Edit: Looks like there is no way to do that. So I am now putting down my problem here. Please help me to think in 'Go-way'.
So, I want to implement a 'timed map', which tracks the key insertion time. In other words, there is a map which accepts and processes the values. Now, if the data in map is older than some specific time-interval, then I should clear it out.
So, I thought of having a key struct which has id and timestamp. When a new key comes, map takes it with id and currentTimeInMillis. After sometime, if a key comes which already exists, then map should preserve the first insertion time and only updates the value array.
To process, I will have a looping over map and check if any particular key is inside for more than threshold limit, then I clear it out. I can have this timestamp in value array, but that also has a timestamp of its own, so putting one more might confuse someone else.
Please suggest something.
Put the time on your value. Here's some example of how to structure your data.
type DataObj struct {
Id int
Updated time.Date
// other fields
}
m := map[int]DataObj{}
m[d.Id] = d // assign using the id as your key
for k, v := range m {
if time.Since(v.Updated) > duration {
delete(m, k) // remove the stale item
}
}
// some logic like this for adding/overwriting
v, ok := m[newObj.Id]
if ok { // an element with this id existed
if time.Since(v.Updated) > duration {
m[v.Id] = newObj // assign new value over old one
}
}
I can't provide anything much more specific because you don't have any code with which to work. It seems like you'd probably like some of this (like the remove bits) to run on a timer. To do that, invoke the function as a goroutine and use a timer so every X seconds it unblocks and removes items from the map. If you're doing this you also need to use a mutex so the calling scope doesn't access the map while the remove function running the background is filtering out old items.
The overwrite bit is really straight forward, just test if the item is in the map, check it's time stamp, if it's beyond the threshold assign the new value, if not do nothing.
The main thing to take away here is to not use a struct for your key... There is no reason to do object equality, your object has an id, us it as your key. Everything else you care about can be held on the value (even the key itself is). As someone pointed out this isn't Java and even if it were, equality overrides in C# and Java are literally a fucking nightmare.
I made bindings to a C api (bullet physics engine) using cgo, some functions make use of data pointers. The idea is that I can attach a pointer to an object and retrieve it later when the physics engine invokes a callback. My problem is that when i get the value back, it change and I didn't do it. It seems that no source code is explicitelly changing the value.
CollisionObject: source, header,
The go codes that interracts with that class
heres how i send the values, the reconversion to *int and int is fine, the correct numbers are printed:
num := x*amounty*amountz + y*amountz + z + 1
ptr := unsafe.Pointer(&num)
fmt.Printf("created %v %v\n", ptr, *(*int)(ptr))
rb := sphere.RigidBody(ptr, 1)
But when I get it back from a raytest the value changed:
ptr := hit.GetUserPointer()
log.Printf("we got back: %v %v", ptr, *(*int)(ptr))
the pointer value itself didnt change, i can look up and see that there was a pointer pointing to this location, but the value its pointing at is different.
Now i'm wondering if maybe go didn't clean the value (garbage collected) since it wouldn't be used anymore and replaced this memory location with something else.
example output (with junk values removed):
created: 0xc2080006e0 40
2014/11/07 17:10:01 we got back: 0xc2080006e0 4921947622888946315
ANY pointer (hehe) is appreciated :)
Go's garbage collector doesn't know about the pointers held by C or C++ code, so there is nothing to keep the num variable alive.
You can work around this by storing a second copy of the pointer in a Go variable. One way is to use a global variable with a type like map[*C.some_c_type]*int or similar, and store &num there too. Remember to protect the map with a mutex so things behave correctly when you have concurrent access.
In order not to leak, you will need to manually delete &num from the map when the underlying C code is no longer holding a reference to it. If the C library provides the ability to set a destroy notify function when storing the user pointer, this will be easy: just export a Go function to C and use it as the notify function. If it doesn't, but the Go binding knows when the the pointer will be finished with (e.g. if the RigidBody variable is always freed via the Go API, you can do the clean up there.