I am implementing some DB logic without using an ORM.
Can I somehow create a mapping between struct keys and db enums?
type Message struct {
SomeKey string
SomeOtherKey string
}
MessageToDBEnum: = map[ ? MessageKey] string {
SomeKey: "some_key",
SomeOtherKey: "some_other_key"
}
can I later dynamically use the map key (eg when iterating throgh the map) to set/get struct values?
also can I somehow assure that MessageToDBEnum is exhaustive (all public keys of Message are included)?
You essentially want to convert a struct to map and vice versa. There are 3rd party libs to do that, e.g. github.com/mitchellh/mapstructure.
But we can do it ourselves too, it's not that hard. We may use reflection to do it. Without checking errors, here's the essence of the struct to map conversion:
func struct2Map(x interface{}) map[string]interface{} {
m := map[string]interface{}{}
v := reflect.ValueOf(x)
t := reflect.TypeOf(x)
for i := 0; i < v.NumField(); i++ {
m[t.Field(i).Name] = v.Field(i).Interface()
}
return m
}
You may use it like this:
msg := Message{
SomeKey: "v1",
SomeOtherKey: "v2",
}
m := struct2Map(msg)
fmt.Println(m)
Which outputs:
map[SomeKey:v1 SomeOtherKey:v2]
The backward conversion is even simpler, but know that for a function to modify a struct, you must pass a pointer to it. Again, without checking for possible errors, the essence of it is:
func map2Struct(m map[string]interface{}, d interface{}) {
s := reflect.ValueOf(d).Elem()
for k, v := range m {
s.FieldByName(k).Set(reflect.ValueOf(v))
}
}
Using it:
var msg2 Message
map2Struct(m, &msg2)
fmt.Printf("%+v\n", msg2)
Which outputs:
{SomeKey:v1 SomeOtherKey:v2}
Try the examples on the Go Playground.
You may build the validation into these conversion functions, and return an error or use default / zero values when an invalid value is found.
Related
Here is the simplified code of a cache. Suppose Container placed in a package, so it don't know about Member.
While I wanna store instances of Member in Container, So I store an empty instance of Member in Container as outerType.
In the Container->GetMysql, I fill a new variable by test values (but, in real world, It fill by data of database, dynamically).
then in the function Put, I store data in items as Cache for next uses. In the Get I get the data stored in the items.
Before this every thing is fine. My problem is where i want to convert result of Get to type of Member m = res.(Member) . How Can I convert it to an instance of Member
I found many question about this subject, but none of them solved my problem
For more detail: I want the Get return data with its pointer of where it stored in items. So if I get some variable of same member, an change in one are shown in others
package main
import (
"fmt"
"reflect"
)
type Member struct {
Id int
Name string
Credit int
Age int
}
type Container struct {
outerType interface{}
items map[string]*interface{}
}
func (cls *Container)GetMysql(s string, a int64) interface{}{
obj := reflect.New(reflect.TypeOf(cls.outerType))
elem := obj.Elem()
//elem := reflect.ValueOf(o).Elem()
if elem.Kind() == reflect.Struct {
f := elem.FieldByName("Name")
f.SetString(s)
f = elem.FieldByName("Credit")
f.SetInt(a)
}
return obj.Interface()
}
func (cls *Container)Get(value string) *interface{}{
return cls.items[value]
}
func (cls *Container)Put(value string, a int64) {
res := cls.GetMysql(value, a)
cls.items[value] = &res
}
func main() {
c := Container{outerType:Member{}}
c.items = make(map[string]*interface{})
c.Put("Jack", 500)
res := c.Get("Jack")
fmt.Println(*res)
m := &Member{}
m = res.(Member) // Here is the problem. How to convert ?
fmt.Println(m)
}
You should hardly ever use pointer to interface. My advice is to never use it, when you'll need it, you'll know.
Instead if you need a pointer to something (so you can have the same pointer at multiple places, and so modifying the pointed value somewhere, it will have effect on the others), "wrap the pointer" in the interface value.
So first modify the items field so that it stores interface{} values instead of pointers:
items map[string]interface{}
This means no restriction: you can pass and store pointers, that's not a problem.
Next modify Get() to return interface{}:
func (cls *Container) Get(value string) interface{}{
return cls.items[value]
}
And also in Put(), don't take the address of an interface{}:
func (cls *Container) Put(value string, a int64) {
res := cls.GetMysql(value, a)
cls.items[value] = res
}
And you have to type-assert *Member from the values returned by Get().
And now testing it:
c := Container{outerType: Member{}}
c.items = make(map[string]interface{})
c.Put("Jack", 500)
res := c.Get("Jack")
fmt.Println(res)
m := res.(*Member) // Here is the problem. How to convert ?
fmt.Println(m)
Output (try it on the Go Playground):
&{0 Jack 500 0}
&{0 Jack 500 0}
Now if you would modify a field of m:
m.Credit = 11
And then get the value form the cache:
fmt.Println(c.Get("Jack"))
We'll see the modified value, even though we did not call Put() (try it on the Go Playground):
&{0 Jack 11 0}
I am writing a function that iterates over the entries in a map. I want to be able to deal cleanly with items which are added or deleted from the map while iterating, like for k, v := range myMap { //... does, but I am processing just one key/value pair per iteration so I can't use range. I want something like:
func processItem(i iterator) bool {
k, v, ok := i.next()
if(!ok) {
return false
}
process(v)
return true
}
var m = make(map[string]widget)
// ...
i := makeIterator(m)
for processItem(i) {
// code which might add/remove item from m here
}
I know that range is using a 'hiter' struct and associated functions, as defined in src/runtime/hashmap.go, to perform iteration. Is there some way to gain access to this iterator as a reified (first-class) Go object?
Is there an alternative strategy for iterating over a map which would deal well with insertions/deletions but give a first-class iterator object?
Bonus question: is there an alternative strategy for iterating over a map which could also deal with the map and iterator being serialised to disk and then restored, with iteration continuing from where it left off? (Obviously the built-in range iterator does not have this capability!)
You can't :(
The only way to iterate over a map is by using for range and you can't get an iterator object out of that.
You can use channels as iterators.
Your iterator would be a function returning a channel that communicates the current iteration value to whoever receives it:
func iterator(m map[string]widget) chan iteration {
c := make(chan iteration)
go func() {
for k,v := range m {
c <- iteration{k,v}
}
close(c)
}()
return c
}
This is of course not generic, you could make it generic using interface{} and/or reflection but that shouldn't be too hard if you actually need it.
Closing the channel at the end of iteration will notify the end of iteration, demonstrated later.
The iteration type is just there so you can send key and value at the same time, it would look something like this:
type iteration struct {
key string
value widget
}
With this you can then do this (on play):
m := map[string]widget{"foo": widget{3}, "bar": widget{4}}
i := iterator(m)
iter, ok := <- i
fmt.Println(iter, ok)
iter, ok = <- i
fmt.Println(iter, ok)
iter, ok = <- i
fmt.Println(iter, ok)
which yields
{foo {3}} true
{bar {4}} true
{ {0}} false
A very simple approach is to obtain a list of all the keys in the map, and package the list and the map up in an iterator struct. When we want the next key, we take the next one from the list that hasn't been deleted from the map:
type iterator struct {
m map[string]widget
keys []string
}
func newIterator(m map[string]widget) *iterator {
it := iterator{m, make([]string, len(m))}
i := 0
for k, _ := range m {
it.keys[i] = k
i++
}
return &it
}
func (it *iterator) next() (string, widget, bool) {
for len(it.keys) > 0 {
k := it.keys[0]
it.keys = it.keys[1:]
if _, exists := it.m[k]; exists {
return k, it.m[k], true
}
}
return "", widget{0}, false
}
See running example on play.
You can define your own map type. Also it will be good to solve concurrency problem:
type ConcurrentMap struct {
sync.RWMutex
items map[string]interface{}
}
type ConcurrentMapItem struct {
Key string
Value interface{}
}
func (cm *ConcurrentMap) Iter() <-chan ConcurrentMapItem {
c := make(chan ConcurrentMapItem)
f := func() {
cm.Lock()
defer cm.Unlock()
for k, v := range cm.items {
c <- ConcurrentMapItem{k, v}
}
close(c)
}
go f()
return c
}
TL;DR Somehow, I am appending a pointer to a list instead of the object within a for loop of objects so at the end the entire slice is composed of the same object multiple times. I just don't know how to fix that.
The Long Way
I am still having a super hard time trying to figure out pointers in go. I posted a question yesterday and got some help but now I am stuck on a slightly different issue in the same piece of code.
I am working with gocql and cqlr go packages to try and bit a small object mapper for my Cassandra queries. Essentially the problem I am having is I am appending what appears to be a pointer to an object, not a new instance of the obj to the array. How do I fix that? I have tried adding & and * in front of value but that doesn't seem to work. How do I fix these? The bind function needs an & according to their docs.
Code
type Query struct {
query string
values interface{}
attempts int
maxAttempts int
structType reflect.Type
}
func (query Query) RetryingQuery() (results []interface{}) {
var q *gocql.Query
if query.values != nil {
q = c.Session.Query(query.query, query.values)
} else {
q = c.Session.Query(query.query)
}
bindQuery := cqlr.BindQuery(q)
value := reflect.New(query.structType).Interface()
for bindQuery.Scan(value) {
fmt.Println(value)
results = append(results, value)
}
return
}
The docs ask for var value type then in bind you would pass &value. I quoted the docs below.
var t Tweet
var s []Tweet
for b.Scan(&t) {
// Application specific code goes here
append(s, t)
}
The issue is I cannot directly go var value query.structType to define its type then pass the reference of that to bindQuery.Scan().
What is printed
&{result1 x86_64 24 3.2.0-74-generic Linux}
&{result2 x86_64 24 3.19.0-25-generic Linux}
&{result3 x86_64 4 3.13.0-48-generic Linux}
&{result4 x86_64 2 3.13.0-62-generic Linux}
&{result5 x86_64 4 3.13.0-48-generic Linux}
What is in the slice
Spoiler, it is result5 repeated over and over. I understand that I am just appending the pointer to same object to the list and that every loop iteration the object is changed and that changes all the results in the slice to that new object. I just don't know how to fix it.
[{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"}]
Well I can at least tell you what you're doing. bindQuery takes a pointer. It changes the value stored at the address.
What you're essentially doing is this:
package main
import "fmt"
func main() {
var q int
myInts := make([]*int, 0, 5)
for i := 0; i < 5; i++ {
q = i
fmt.Printf("%d ", q)
myInts = append(myInts, &q)
}
fmt.Printf("\n")
for _, value := range myInts {
fmt.Printf("%d ", *value)
}
fmt.Printf("\n")
fmt.Println(myInts)
}
Which, as you can probably guess, gives you this:
0 1 2 3 4
4 4 4 4 4
[0x104382e0 0x104382e0 0x104382e0 0x104382e0 0x104382e0]
Things get a little more confusing with reflect. You can get your type as an interface, but that is it (unless you want to play with unsafe). An interface, in simple terms, contains a pointer to the original type underneath (and some other stuff). So in your function you are passing a pointer (and some other stuff). Then you're appending the pointer. It might be nice just to get concrete and type switch your interface. I assume you know what types it could be. In which case you'd have to have something along these lines:
package main
import (
"fmt"
"reflect"
)
type foo struct {
fooval string
}
type bar struct {
barval string
}
func main() {
f1 := foo{"hi"}
f2 := &foo{"hi"}
b1 := bar{"bye"}
b2 := &bar{"bye"}
doSomething(f1)
doSomething(f2)
doSomething(b1)
doSomething(b2)
}
func doSomething(i interface{}) {
n := reflect.TypeOf(i)
// get a new one
newn := reflect.New(n).Interface()
// find out what we got and handle each case
switch t := newn.(type) {
case **foo:
*t = &foo{"hi!"}
fmt.Printf("It was a **foo, here is the address %p and here is the value %v\n", *t, **t)
case **bar:
*t = &bar{"bye :("}
fmt.Printf("It was a **bar, here is the address %p and here is the value %v\n", *t, **t)
case *foo:
t = &foo{"hey!"}
fmt.Printf("It was a *foo, here is the address %p and here is the value %v\n", t, *t)
case *bar:
t = &bar{"ahh!"}
fmt.Printf("It was a *bar, here is the address %p and here is the value %v\n", t, *t)
default:
panic("AHHHH")
}
}
You could also just keep calling value = reflect.New(query.structType).Interface() inside of the loop which will give you new interfaces every time. Reassigning value after every append. Last time through the loop would make one extra though..
I am trying to pass to a third-party package a variadic list of pointers to fields in a struct. The package accepts a variadic interface{} list ( func Persist(...interface) error ), where each of the interface values is a pointer to a variable. I created a function that mocks how the third-party library and prints out the Type and Kind of the pointers (called mockFunction below).
When I pass it the address of the struct variables in a non-variadic way, they have their primitive Types and Values within the mocked function using the reflect calls. However, when I pass them in a variadic way using expansion, they have Type: Type: reflect.Value and Kind: struct. The third-party package does not know how to handle them in this form.
I would like to figure out a way to call the third-party package with a slice of interface{} (e.g. inv := make([]interface{}, 3) and use variadic expansion on the call Persist(inv...) if at all possible.
Here is the code with a link to Go Playground below:
package main
import (
"fmt"
"reflect"
)
type Investment struct {
Price float64
Symbol string
Rating int64
}
func main() {
inv := Investment{Price: 534.432, Symbol: "GBG", Rating: 4}
s := reflect.ValueOf(&inv).Elem()
variableParms := make([]interface{}, s.NumField())
for i := 0; i < s.NumField(); i++ {
variableParms[i] = s.Field(i).Addr()
}
// non-variadic call
mockFunction(&inv.Price, &inv.Symbol, &inv.Rating)
//variadic call
mockFunction(variableParms...)
}
func mockFunction(values ...interface{}) {
for i, value := range values {
rv := reflect.ValueOf(value)
fmt.Printf("value %d has Type: %s and Kind %s\n", i, rv.Type(), rv.Kind())
}
}
Go Playground Link
When I run it with the non-variadic parameters, the call to mockFunction returns the native Types and Kinds and the third-party package processes them fine:
value 0 has Type: *float64 and Kind ptr
value 1 has Type: *string and Kind ptr
value 2 has Type: *int64 and Kind ptr
When I run it with the variadic parameters, the values are different and the third-party package does not know how to handle these types:
value 0 has Type: reflect.Value and Kind struct
value 1 has Type: reflect.Value and Kind struct
value 2 has Type: reflect.Value and Kind struct
Is there any way to structure the slice definition and the call to what is placed in to the slice so that it can be variadic expanded and look like passing the pointers to the struct fields in the non-variadic way?
Addr() returns the reflect Value for the field pointer. Call Ptr() on the value to get the actual pointer as an interface{}.
variableParms[i] = s.Field(i).Addr().Ptr()
playground
I think that perhaps Go's handling for this case has changed since 2014 - certainly the code above no longer works for me with Go 1.10...
However the following code works for me to create an appropriate []interface{} to use in the described way...
func settableSliceFromStruct(inStruct interface{}) ([]interface{}, error) {
t := reflect.TypeOf(inStruct)
if t.Kind() != reflect.Ptr {
return nil, errors.New("can only assign values with pointer to struct")
}
v := reflect.ValueOf(inStruct).Elem()
t = t.Elem()
dataColumns := make([]interface{}, 0, t.NumField())
for i := 0; i < t.NumField(); i++ {
if weWantToIncludeThis(t.Field(i)) {
dataColumns = append(dataColumns, v.Field(i).Addr().Interface())
}
}
return dataColumns, nil
}
The critical part here would be for your code to use:
variableParms[i] = s.Field(i).Addr().Interface()
I have some trouble building a function that can dynamically use parametrized structs. For that reason my code has 20+ functions that are similar except basically for one type that gets used. Most of my experience is with Java, and I'd just develop basic generic functions, or use plain Object as parameter to function (and reflection from that point on). I would need something similar, using Go.
I have several types like:
// The List structs are mostly needed for json marshalling
type OrangeList struct {
Oranges []Orange
}
type BananaList struct {
Bananas []Banana
}
type Orange struct {
Orange_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
type Banana struct {
Banana_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
Then I have function, basically for each list type:
// In the end there are 20+ of these, the only difference is basically in two types!
// This is very un-DRY!
func buildOranges(rows *sqlx.Rows) ([]byte, error) {
oranges := OrangeList{} // This type changes
for rows.Next() {
orange := Orange{} // This type changes
err := rows.StructScan(&orange) // This can handle each case already, could also use reflect myself too
checkError(err, "rows.Scan")
oranges.Oranges = append(oranges.Oranges,orange)
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(oranges)
return jsontext, err
}
Yes, I could change the sql library to use more intelligent ORM or framework, but that's besides the point. I want to learn on how to build generic function that can handle similar function for all my different types.
I got this far, but it still doesn't work properly (target isn't expected struct I think):
func buildWhatever(rows *sqlx.Rows, tgt interface{}) ([]byte, error) {
tgtValueOf := reflect.ValueOf(tgt)
tgtType := tgtValueOf.Type()
targets := reflect.SliceOf(tgtValueOf.Type())
for rows.Next() {
target := reflect.New(tgtType)
err := rows.StructScan(&target) // At this stage target still isn't 1:1 smilar struct so the StructScan fails... It's some perverted "Value" object instead. Meh.
// Removed appending to the list because the solutions for that would be similar
checkError(err, "rows.Scan")
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(targets)
return jsontext, err
}
So umm, I would need to give the list type, and the vanilla type as parameters, then build one of each, and the rest of my logic would be probably fixable quite easily.
Turns out there's an sqlx.StructScan(rows, &destSlice) function that will do your inner loop, given a slice of the appropriate type. The sqlx docs refer to caching results of reflection operations, so it may have some additional optimizations compared to writing one.
Sounds like the immediate question you're actually asking is "how do I get something out of my reflect.Value that rows.StructScan will accept?" And the direct answer is reflect.Interface(target); it should return an interface{} representing an *Orange you can pass directly to StructScan (no additional & operation needed). Then, I think targets = reflect.Append(targets, target.Indirect()) will turn your target into a reflect.Value representing an Orange and append it to the slice. targets.Interface() should get you an interface{} representing an []Orange that json.Marshal understands. I say all these 'should's and 'I think's because I haven't tried that route.
Reflection, in general, is verbose and slow. Sometimes it's the best or only way to get something done, but it's often worth looking for a way to get your task done without it when you can.
So, if it works in your app, you can also convert Rows straight to JSON, without going through intermediate structs. Here's a sample program (requires sqlite3 of course) that turns sql.Rows into map[string]string and then into JSON. (Note it doesn't try to handle NULL, represent numbers as JSON numbers, or generally handle anything that won't fit in a map[string]string.)
package main
import (
_ "code.google.com/p/go-sqlite/go1/sqlite3"
"database/sql"
"encoding/json"
"os"
)
func main() {
db, err := sql.Open("sqlite3", "foo")
if err != nil {
panic(err)
}
tryQuery := func(query string, args ...interface{}) *sql.Rows {
rows, err := db.Query(query, args...)
if err != nil {
panic(err)
}
return rows
}
tryQuery("drop table if exists t")
tryQuery("create table t(i integer, j integer)")
tryQuery("insert into t values(?, ?)", 1, 2)
tryQuery("insert into t values(?, ?)", 3, 1)
// now query and serialize
rows := tryQuery("select * from t")
names, err := rows.Columns()
if err != nil {
panic(err)
}
// vals stores the values from one row
vals := make([]interface{}, 0, len(names))
for _, _ = range names {
vals = append(vals, new(string))
}
// rowMaps stores all rows
rowMaps := make([]map[string]string, 0)
for rows.Next() {
rows.Scan(vals...)
// now make value list into name=>value map
currRow := make(map[string]string)
for i, name := range names {
currRow[name] = *(vals[i].(*string))
}
// accumulating rowMaps is the easy way out
rowMaps = append(rowMaps, currRow)
}
json, err := json.Marshal(rowMaps)
if err != nil {
panic(err)
}
os.Stdout.Write(json)
}
In theory, you could build this to do fewer allocations by not reusing the same rowMap each time and using a json.Encoder to append each row's JSON to a buffer. You could go a step further and not use a rowMap at all, just the lists of names and values. I should say I haven't compared the speed against a reflect-based approach, though I know reflect is slow enough it might be worth comparing them if you can put up with either strategy.