How to solve concurrency access of Golang map? - dictionary

Now i have a map with only one write/delete goroutine and many read goroutines, there are some solutions upon Map with concurrent access, such as RWMutex, sync.map, concurrent-map, sync.atomic, sync.Value, what's the best choice for me?
RWMutex's read lock is a little redundant
sync.map and concurrent-map focus on many write goroutines

Your question is a little vague - so I'll break it down.
What form of concurrent access should I use for a map?
The choice depends on the performance you require from the map. I would opt for a simple mutex (or a RWMutex) based approach.
Granted, you can get better performance from a concurrent map. sync.Mutex locks all of a maps buckets, whereas in a concurrent map, each bucket has it's own sync.Mutex.
Again - it all depends on the scale of your program and the performance you require.
How would I use a mutex for concurrent access?
To ensure the map is being used correctly, you can wrap this in a struct.
type Store struct {
Data map[T]T
}
This a more object-oriented solution, but it allows us to make sure any read/writes are performed concurrently. As well as this, we can easily store other information that may be useful for debugging or security, such as author.
Now, we would implement this with a set of methods like so:
mux sync.Mutex
// New initialises a Store type with an empty map
func New(t, h uint) *Store {
return &Store{
Data: map[T]T{},
}
}
// Insert adds a new key i to the store and places the value of x at this location
// If there is an error, this is returned - if not, this is nil
func (s *Store) Insert(i, x T) error {
mux.Lock()
defer mux.Unlock()
_, ok := s.Data[i]
if ok {
return fmt.Errorf("index %s already exists; use update", i)
}
s.Data[i] = x
return nil
}
// Update changes the value found at key i to x
// If there is an error, this is returned - if not, this is nil
func (s *Store) Update(i, x T) error {
mux.Lock()
defer mux.Unlock()
_, ok := s.Data[i]
if !ok {
return fmt.Errorf("value at index %s does not exist; use insert", i)
}
s.Data[i] = x
return nil
}
// Fetch returns the value found at index i in the store
// If there is an error, this is returned - if not, this is nil
func (s *Store) Fetch(i T) (T, error) {
mux.Lock()
defer mux.Unlock()
v, ok := s.Data[i]
if !ok {
return "", fmt.Errorf("no value for key %s exists", i)
}
return v, nil
}
// Delete removes the index i from store
// If there is an error, this is returned - if not, this is nil
func (s *Store) Delete(i T) (T, error) {
mux.Lock()
defer mux.Unlock()
v, ok := s.Data[i]
if !ok {
return "", fmt.Errorf("index %s already empty", i)
}
delete(s.Data, i)
return v, nil
}
In my solution, I've used a simple sync.Mutex - but you can simply change this code to accommodate RWMutex.
I recommend you take a look at How to use RWMutex in Golang?.

Related

How can I create a first-class map iterator in Go?

I am writing a function that iterates over the entries in a map. I want to be able to deal cleanly with items which are added or deleted from the map while iterating, like for k, v := range myMap { //... does, but I am processing just one key/value pair per iteration so I can't use range. I want something like:
func processItem(i iterator) bool {
k, v, ok := i.next()
if(!ok) {
return false
}
process(v)
return true
}
var m = make(map[string]widget)
// ...
i := makeIterator(m)
for processItem(i) {
// code which might add/remove item from m here
}
I know that range is using a 'hiter' struct and associated functions, as defined in src/runtime/hashmap.go, to perform iteration. Is there some way to gain access to this iterator as a reified (first-class) Go object?
Is there an alternative strategy for iterating over a map which would deal well with insertions/deletions but give a first-class iterator object?
Bonus question: is there an alternative strategy for iterating over a map which could also deal with the map and iterator being serialised to disk and then restored, with iteration continuing from where it left off? (Obviously the built-in range iterator does not have this capability!)
You can't :(
The only way to iterate over a map is by using for range and you can't get an iterator object out of that.
You can use channels as iterators.
Your iterator would be a function returning a channel that communicates the current iteration value to whoever receives it:
func iterator(m map[string]widget) chan iteration {
c := make(chan iteration)
go func() {
for k,v := range m {
c <- iteration{k,v}
}
close(c)
}()
return c
}
This is of course not generic, you could make it generic using interface{} and/or reflection but that shouldn't be too hard if you actually need it.
Closing the channel at the end of iteration will notify the end of iteration, demonstrated later.
The iteration type is just there so you can send key and value at the same time, it would look something like this:
type iteration struct {
key string
value widget
}
With this you can then do this (on play):
m := map[string]widget{"foo": widget{3}, "bar": widget{4}}
i := iterator(m)
iter, ok := <- i
fmt.Println(iter, ok)
iter, ok = <- i
fmt.Println(iter, ok)
iter, ok = <- i
fmt.Println(iter, ok)
which yields
{foo {3}} true
{bar {4}} true
{ {0}} false
A very simple approach is to obtain a list of all the keys in the map, and package the list and the map up in an iterator struct. When we want the next key, we take the next one from the list that hasn't been deleted from the map:
type iterator struct {
m map[string]widget
keys []string
}
func newIterator(m map[string]widget) *iterator {
it := iterator{m, make([]string, len(m))}
i := 0
for k, _ := range m {
it.keys[i] = k
i++
}
return &it
}
func (it *iterator) next() (string, widget, bool) {
for len(it.keys) > 0 {
k := it.keys[0]
it.keys = it.keys[1:]
if _, exists := it.m[k]; exists {
return k, it.m[k], true
}
}
return "", widget{0}, false
}
See running example on play.
You can define your own map type. Also it will be good to solve concurrency problem:
type ConcurrentMap struct {
sync.RWMutex
items map[string]interface{}
}
type ConcurrentMapItem struct {
Key string
Value interface{}
}
func (cm *ConcurrentMap) Iter() <-chan ConcurrentMapItem {
c := make(chan ConcurrentMapItem)
f := func() {
cm.Lock()
defer cm.Unlock()
for k, v := range cm.items {
c <- ConcurrentMapItem{k, v}
}
close(c)
}
go f()
return c
}

Concise nil checks for struct field pointers?

Say I have this struct:
type Foo struct {
Bar *string `json:"bar"`
Baz *int64 `json:"baz,omitempty"`
Qux *string `json:"qux"`
Quux string `json:"quux"`
}
After unmarshalling the json, I check for nil like so:
switch {
case f.Bar == nil:
return errors.New("Missing 'bar'")
case f.Baz == nil:
f.Baz = 42
case f.Qux == nil:
return errors.New("Missing 'qux'")
}
(or through a series of if statements, etc...)
I understand that I can put all the nil comparisons in one comma separated case, but each nil check will have differing returns.
My question: is there a less verbose way of doing the nil checks?
A question to you: how less verbose you want to get? Because you want to do different things on different conditions (different fields being nil). Your code contains these different things and the different conditions. Beyond that what's "redundant" in your code are just the switch and case keywords. You want to leave them out? Because the rest is not "redundant", they are required.
Also note that in Go cases do not fall through even without a break (unlike in other languages), so in your above example if f.Baz is nil, you will set it to 42 and f.Qux will not be checked (so no error will be returned), but if f.Baz is non-nil and f.Qux is nil, an error will be returned. I know it's just an example, but this is something to keep in mind. You should handle errors first if you use a switch! Or use if statements and then the error will be detected and returned regardless of the order of field checks.
Your code with switch is clean and efficient. If you want to make it less verbose, readability (and performance) will suffer.
You may use a helper function which checks if a pointer value is nil:
func n(i interface{}) bool {
v := reflect.ValueOf(i)
return v.Kind() == reflect.Ptr && v.IsNil()
}
And using it:
func check(f *Foo) error {
switch {
case n(f.Bar):
return errors.New("Missing 'bar'")
case n(f.Qux):
return errors.New("Missing 'qux'")
case n(f.Baz):
x := int64(42)
f.Baz = &x
}
return nil
}
Or using if statements:
func check2(f *Foo) error {
if n(f.Bar) {
return errors.New("Missing 'bar'")
}
if n(f.Qux) {
return errors.New("Missing 'qux'")
}
if n(f.Baz) {
x := int64(42)
f.Baz = &x
}
return nil
}
Try these on the Go Playground.

assigning nil to pointer

I'm trying to implement a delete() method to a list (no HEAD ref)
I find out that I can modify the parameter to a struct.
func (l *LinkedList) Delete(n *Node) {
if n.next == nil {
n = nil
} else {
current := &n
*n = *n.next
*current = nil
}
}
The "else" part works fine, but deleting the last node does not modify the list
Tried using
*n = nil
But then I have the compile error.
cannot use nil as type Node in assignment
Code complete in this playground:
http://play.golang.org/p/Dhzyd7QHEw
You're just doing it wrong. I mean classic element removal from single linked list. Right way:
func (l *LinkedList) Delete(n *Node) {
// if we want to delete the head element - just move the head pointer
if l.head == n {
l.head = n.next
return
}
// otherwise find previous element to the one we are deleting
current := l.head
for current != nil && current.next != n {
current = current.next
}
// and move that previous element next pointer to the next element
if current != nil {
current.next = n.next
}
}
https://play.golang.org/p/_NlJw_fPWQD
So what was wrong in your example? In your Delete function you are receiving a pointer to some node. This pointer is local to your function, it's like a local variable. It doesn't matter if you assign nil to a local variable inside your function. Outside - no one will see such assignments. What you want to do - is to change the next pointer of the previous list item. This way the item will no longer be in the list. GC will remove the actual allocated memory.
UPDATE:
Since go pointers are "real" pointers, this can be implemented without special case for the head removal, by using an additional level of indirection, as suggested by Linus in his famous TED talk (and earlier in slashdot Q&A - see "favorite hack" question):
func (l *LinkedList) Delete(n *Node) {
// initialize indirect with the address of a head pointer
indirect := &(l.head)
// until indirect has address of a pointer to the node we're deleting
for *indirect != n {
// check that it's not the end of the list
if (*indirect).next == nil {
// the node we're tryign to delete is not in the list
return
}
// set indirect to the address of the next pointer
indirect = &(*indirect).next
}
// indirect has address of a pointer we need to modify to delete the node
*indirect = n.next
}
https://play.golang.org/p/hDy3hB5LUME
IMO two levels of inderection is harder to understand than a simple special case for deleting the head element, but Linus is not exactly an ordinary developer like myself :)

How can I block (and join) on a channel fed by an unknown number of goroutines?

I have a recursive function. The function will call itself with various different values depending on the data it gets, so the arity and depth of recursion is not known: each call may call itself zero or more times. The function may return any number of values.
I want to parallelise it by getting goroutines and channels involved. Each recursion of inner runs in its own goroutine, and sends back a value on the channel. The outer function deals with those values.
func outer(response []int) {
results := make([]int)
resultsChannel := make(chan int)
inner := func(...) {
resultsChannel <- «some result»;
// Recurse in a new goroutine.
for _, recursionArgument in «some calculated data» {
go inner(recursionArgument)
}
}
go inner(«initial values»);
for {
result := <- resultsChannel
results = append(results, result)
// HELP! How do I decide when to break?
}
return results
}
The problem comes with escaping the results channel loop. Because of the 'shape' of the recursion (unknown arity and depth) I can't say "finish after n events" and I can't send a sentinel value.
How do I detect when all my recursions have happened and return from outer? Is there a better way to approach this?
You can use a sync.WaitGroup to manage the collection of goroutines you spawn: call Add(1) before spawning each new goroutine, and Done when each goroutine completes. So something like this:
var wg sync.WaitGroup
inner := func(...) {
...
// Recurse in a new goroutine.
for _, recursionArgument := range «some calculated data» {
wg.Add(1)
go inner(recursionArgument)
}
...
wg.Done()
}
wg.Add(1)
go inner(«initial values»)
Now waiting on wg will tell you when all the goroutines have completed.
If you are reading the results from a channel, the obvious way to tell when there are no more results is by closing the channel. You can achieve this through another goroutine to do this for us:
go func() {
wg.Wait()
close(resultsChannel)
}()
You should now be able to simply range over resultsChannel to read all the results.

Using reflection with structs to build generic handler function

I have some trouble building a function that can dynamically use parametrized structs. For that reason my code has 20+ functions that are similar except basically for one type that gets used. Most of my experience is with Java, and I'd just develop basic generic functions, or use plain Object as parameter to function (and reflection from that point on). I would need something similar, using Go.
I have several types like:
// The List structs are mostly needed for json marshalling
type OrangeList struct {
Oranges []Orange
}
type BananaList struct {
Bananas []Banana
}
type Orange struct {
Orange_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
type Banana struct {
Banana_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
Then I have function, basically for each list type:
// In the end there are 20+ of these, the only difference is basically in two types!
// This is very un-DRY!
func buildOranges(rows *sqlx.Rows) ([]byte, error) {
oranges := OrangeList{} // This type changes
for rows.Next() {
orange := Orange{} // This type changes
err := rows.StructScan(&orange) // This can handle each case already, could also use reflect myself too
checkError(err, "rows.Scan")
oranges.Oranges = append(oranges.Oranges,orange)
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(oranges)
return jsontext, err
}
Yes, I could change the sql library to use more intelligent ORM or framework, but that's besides the point. I want to learn on how to build generic function that can handle similar function for all my different types.
I got this far, but it still doesn't work properly (target isn't expected struct I think):
func buildWhatever(rows *sqlx.Rows, tgt interface{}) ([]byte, error) {
tgtValueOf := reflect.ValueOf(tgt)
tgtType := tgtValueOf.Type()
targets := reflect.SliceOf(tgtValueOf.Type())
for rows.Next() {
target := reflect.New(tgtType)
err := rows.StructScan(&target) // At this stage target still isn't 1:1 smilar struct so the StructScan fails... It's some perverted "Value" object instead. Meh.
// Removed appending to the list because the solutions for that would be similar
checkError(err, "rows.Scan")
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(targets)
return jsontext, err
}
So umm, I would need to give the list type, and the vanilla type as parameters, then build one of each, and the rest of my logic would be probably fixable quite easily.
Turns out there's an sqlx.StructScan(rows, &destSlice) function that will do your inner loop, given a slice of the appropriate type. The sqlx docs refer to caching results of reflection operations, so it may have some additional optimizations compared to writing one.
Sounds like the immediate question you're actually asking is "how do I get something out of my reflect.Value that rows.StructScan will accept?" And the direct answer is reflect.Interface(target); it should return an interface{} representing an *Orange you can pass directly to StructScan (no additional & operation needed). Then, I think targets = reflect.Append(targets, target.Indirect()) will turn your target into a reflect.Value representing an Orange and append it to the slice. targets.Interface() should get you an interface{} representing an []Orange that json.Marshal understands. I say all these 'should's and 'I think's because I haven't tried that route.
Reflection, in general, is verbose and slow. Sometimes it's the best or only way to get something done, but it's often worth looking for a way to get your task done without it when you can.
So, if it works in your app, you can also convert Rows straight to JSON, without going through intermediate structs. Here's a sample program (requires sqlite3 of course) that turns sql.Rows into map[string]string and then into JSON. (Note it doesn't try to handle NULL, represent numbers as JSON numbers, or generally handle anything that won't fit in a map[string]string.)
package main
import (
_ "code.google.com/p/go-sqlite/go1/sqlite3"
"database/sql"
"encoding/json"
"os"
)
func main() {
db, err := sql.Open("sqlite3", "foo")
if err != nil {
panic(err)
}
tryQuery := func(query string, args ...interface{}) *sql.Rows {
rows, err := db.Query(query, args...)
if err != nil {
panic(err)
}
return rows
}
tryQuery("drop table if exists t")
tryQuery("create table t(i integer, j integer)")
tryQuery("insert into t values(?, ?)", 1, 2)
tryQuery("insert into t values(?, ?)", 3, 1)
// now query and serialize
rows := tryQuery("select * from t")
names, err := rows.Columns()
if err != nil {
panic(err)
}
// vals stores the values from one row
vals := make([]interface{}, 0, len(names))
for _, _ = range names {
vals = append(vals, new(string))
}
// rowMaps stores all rows
rowMaps := make([]map[string]string, 0)
for rows.Next() {
rows.Scan(vals...)
// now make value list into name=>value map
currRow := make(map[string]string)
for i, name := range names {
currRow[name] = *(vals[i].(*string))
}
// accumulating rowMaps is the easy way out
rowMaps = append(rowMaps, currRow)
}
json, err := json.Marshal(rowMaps)
if err != nil {
panic(err)
}
os.Stdout.Write(json)
}
In theory, you could build this to do fewer allocations by not reusing the same rowMap each time and using a json.Encoder to append each row's JSON to a buffer. You could go a step further and not use a rowMap at all, just the lists of names and values. I should say I haven't compared the speed against a reflect-based approach, though I know reflect is slow enough it might be worth comparing them if you can put up with either strategy.

Resources