If sqlite3_open_v2 is commentted, it gives an error: unknown function C.sqlite3
which is strange, why should the prototype affect db :=C.sqlite3(0)?
But if it is commented then I get a bad parameter or other API misuse.
What am I doing wrong?
import sqlite
//fn C.sqlite3_open_v2(charptr, &&C.sqlite3, int, charptr) int
struct C.sqlite3_stmt { }
fn main(){
db := &C.sqlite3(0)
stm := &C.sqlite3_stmt(0)
db_path := ':memory:'
query := 'select 1'
C.sqlite3_open(db_path.str, &db)
err := C.sqlite3_prepare_v2(&db, query.str, -1, &stm, C.NULL)
if err != C.SQLITE_OK {
C.puts(C.sqlite3_errstr(err))
}
C.sqlite3_close(db)
}
You're importing sqlite without really using it. In the sqlite module, there are wrapper functions to do what you want. This is how you would do it:
import sqlite
fn main() {
db_path := ':memory:'
db := sqlite.connect(db_path) or {
panic(err)
}
query := 'select 1'
val, code := db.exec(query) // declare code as 'mut' so it can be reused later
assert code == 101 // 101 is the same as C.SQLITE_DONE
println('$val') // prints the following:
// [sqlite.Row{
// vals: ['1']
// }]
// No closing right now, not sure if it's in the works or needed
}
Related
Following up on old post here.
I am iterating over flatProduct.Catalogs slice and populating my productCatalog concurrent map in golang. I am using upsert method so that I can add only unique productID's into my productCatalog map.
Below code is called by multiple go routines in parallel that is why I am using concurrent map here to populate data into it. This code runs in background to populate data in the concurrent map every 30 seconds.
var productRows []ClientProduct
err = json.Unmarshal(byteSlice, &productRows)
if err != nil {
return err
}
for i := range productRows {
flatProduct, err := r.Convert(spn, productRows[i])
if err != nil {
return err
}
if flatProduct.StatusCode == definitions.DONE {
continue
}
r.products.Set(strconv.Itoa(flatProduct.ProductId, 10), flatProduct)
for _, catalogId := range flatProduct.Catalogs {
catalogValue := strconv.FormatInt(int64(catalogId), 10)
r.productCatalog.Upsert(catalogValue, flatProduct.ProductId, func(exists bool, valueInMap interface{}, newValue interface{}) interface{} {
productID := newValue.(int64)
if valueInMap == nil {
return map[int64]struct{}{productID: {}}
}
oldIDs := valueInMap.(map[int64]struct{})
// value is irrelevant, no need to check if key exists
// I think problem is here
oldIDs[productID] = struct{}{}
return oldIDs
})
}
}
And below are my getters in the same class where above code is there. These getters are used by main application threads to get data from the map or get the whole map.
func (r *clientRepository) GetProductMap() *cmap.ConcurrentMap {
return r.products
}
func (r *clientRepository) GetProductCatalogMap() *cmap.ConcurrentMap {
return r.productCatalog
}
func (r *clientRepository) GetProductData(pid string) *definitions.FlatProduct {
pd, ok := r.products.Get(pid)
if ok {
return pd.(*definitions.FlatProduct)
}
return nil
}
This is how I am reading data from this productCatalog cmap but my system is crashing on the below range statement -
// get productCatalog map which was populated above
catalogProductMap := clientRepo.GetProductCatalogMap()
productIds, ok := catalogProductMap.Get("211")
data, _ := productIds.(map[int64]struct{})
// I get panic here after sometime
for _, pid := range data {
...
}
Error I am getting as - fatal error: concurrent map iteration and map write.
I think issue is r.productCatalog is a concurrentmap, but oldIDs[productID] is a normal map which is causing issues while I am iterating in the for loop above.
How can I fix this race issue I am seeing? One way I can think of is making oldIDs[productID] as concurrent map but if I do that approach then my memory increase by a lot and eventually goes OOM. Below is what I have tried which works and it solves the race condition but it increases the memory by a lot which is not what I want -
r.productCatalog.Upsert(catalogValue, flatProduct.ProductId, func(exists bool, valueInMap interface{}, newValue interface{}) interface{} {
productID := newValue.(int64)
if valueInMap == nil {
// return map[int64]struct{}{productID: {}}
return cmap.New()
}
// oldIDs := valueInMap.(map[int64]struct{})
oldIDs := valueInMap.(cmap.ConcurrentMap)
// value is irrelevant, no need to check if key exists
// oldIDs[productID] = struct{}{}
oldIDs.Set(strconv.FormatInt(productID, 10), struct{}{})
return oldIDs
})
Any other approach I can do which doesn't increase memory and also fixes the race condition I am seeing?
Note
I am still using v1 version of cmap without generics and it deals with strings as keys.
Rather than a plain map[int64]struct{} type, you could define a struct which holds the map and a mutex to control the access to the map:
type myMap struct{
m sync.Mutex
data map[int64]struct{}
}
func (m *myMap) Add(productID int64) {
m.m.Lock()
defer m.m.Unlock()
m.data[productID] = struct{}{}
}
func (m *myMap) List() []int64 {
m.m.Lock()
defer m.m.Unlock()
var res []int64
for id := range m.data {
res = append(res, id)
}
// sort slice if you need
return res
}
With the sample implementation above, you would have to be careful to store *myMap pointers (as opposed to plain myMap structs) in your cmap.ConcurrentMap structure.
In Go, I'm trying to pass an interface{} to the statement.Exec() function from go-sqlite3. I'm sure this is a solved problem, but I cannot figure it out.
Basically I have a struct with the row data which I want to pass to a function that will insert it to a sqlite db. The thing is I want to be able to programmatically control what goes into the statement.Exec() function
Here is an excerpt:
type hostRows struct {
domain string
}
type clientRows struct {
name string
}
func main() {
...
data := hostRows{domain: "dom.com"}
insertRow(sqliteDatabase, data)
data2 := clientRows{name: "bob"}
insertRow(sqliteDatabase, data2)
...
}
func insertRow(db *sql.DB, row interface{}) {
insertSQL := "INSERT INTO table(col) VALUES (?)"
statement, _ := db.Prepare(insertSQL)
statement.Exec(row) // here's the issue, how can I extract the element in the interface to pass it to the function for Exec to understand
}
I know that in this example, I could hard code the row type to the struct and type statement.Exec(row.(hostRows).domain), but now the code will break to when the client struct is passed.
here is the deceleration for the Exec function
func (s *Stmt) Exec(args ...interface{}) (Result, error)
I've tried playing with reflect but it hasn't worked for me so far.
My only solution for the moment is using a switch condition that could check and prepare the right command for Exec, but this is less than dodgy.
type hostRows struct {
domain string
}
type clientRows struct {
name string
}
func main() {
...
data := hostRows{domain: "dom.com"}
insertRow(sqliteDatabase, 1, data)
data2 := clientRows{name: "bob"}
insertRow(sqliteDatabase, 2, data2)
...
}
func insertRow(db *sql.DB, i int, row interface{}) {
insertSQL := "INSERT INTO table(col) VALUES (?)"
statement, _ := db.Prepare(insertSQL)
// This basically could be a working solution, but I'm sure there is a better one
switch i {
case 1:
data := row.(hostRows)
statement.Exec(data.domain)
case 2:
data := row.(clientRows)
statement.Exec(data.name)
}
}
edit: corrected the INSERT statement ; forget the columns. corrected statement.Exec(row.domain) to statement.Exec(row.(hostRows).domain)
edit2: added second example
Remember that in order for reflect.Interface() to work, you must export the fields. To achieve what you want using reflection, you could try something like this:
type hostRows struct {
//Should export field to read it using reflect.Value.Interface()
Domain string
}
type clientRows struct {
//Should export field to read it using reflect.Value.Interface()
Name string
}
func insertRow(db *sql.DB, i int, row interface{}) {
rv := reflect.ValueOf(row)
var args []interface{}
for i := 0; i < rv.NumField(); i++ {
args = append(args, rv.Field(i).Interface())
}
db.Exec("Insert Satement...", args...)
}
I was able to partially solve this with this case
Unfortunately, the Preload() function doesn't seem to work with delving further down in the related object set.
To clarify, I have the following models:
type Room struct {
gorm.Model
Name string
Games []Game `gorm:"ForeignKey:RoomID"`
}
type Game struct {
gorm.Model
RoomID int `gorm:"index"`
Players []Player `gorm:"ForeignKey:GameID"`
}
type Player struct {
gorm.Model
Name string
GameID int `gorm:"index"`
}
When I create a new object with a new Room, Game, and Player object created I get the following data returned (json encoded):
{"Value":{"ID":26,"CreatedAt":"2016-05-15T01:21:22.426234189-07:00","UpdatedAt":"2016-05-15T01:21:22.426234189-07:00","DeletedAt":null,"Name":"foo","Games":[{"ID":17,"CreatedAt":"2016-05-15T01:21:22.427026134-07:00","UpdatedAt":"2016-05-15T01:21:22.427026134-07:00","DeletedAt":null,"RoomID":26,"Turns":null,"Players":[{"ID":4,"CreatedAt":"2016-05-15T01:21:22.427560561-07:00","UpdatedAt":"2016-05-15T01:21:22.427560561-07:00","DeletedAt":null,"Name":"TestPlayer","GameID":17}],"Counter":1,"Assigned":false}],"Testing":false},"Error":null,"RowsAffected":1}
But if I try to query the structure with a preload() function, I get the following:
{"Value":{"ID":26,"CreatedAt":"2016-05-15T01:21:22.426234189-07:00","UpdatedAt":"2016-05-15T01:21:22.426234189-07:00","DeletedAt":null,"Name":"foo","Games":[{"ID":17,"CreatedAt":"2016-05-15T01:21:22.427026134-07:00","UpdatedAt":"2016-05-15T01:21:22.427026134-07:00","DeletedAt":null,"RoomID":26,"Turns":null,"Players":null,"Counter":1,"Assigned":false}],"Testing":false},"Error":null,"RowsAffected":1}
Note that the Players section is now null. Here's my room registration and room query functions:
func RegisterRoom(w http.ResponseWriter, r *http.Request) {
err := r.ParseForm()
if err != nil {
// Handle error
}
// r.PostForm is a map of our POST form values
room := Room{
Name: r.PostFormValue("label"),
Games: []Game{{
Counter: 1,
Players: []Player{{Name: r.PostFormValue("username")}},
}},
}
t := db.Create(&room)
if err := json.NewEncoder(w).Encode(t); err != nil {
panic(err)
}
}
func RoomShow(w http.ResponseWriter, r *http.Request) {
var rm Room
vars := mux.Vars(r)
roomId := vars["roomId"]
id, _ := strconv.Atoi(roomId)
room := db.Preload("Games").First(&rm, id)
result := db.Find(&room)
json.NewEncoder(w).Encode(result)
}
Solved it! Answer was under Nested Preloading smacks forehead
I have the following file structure:
models/db.go
type DB struct {
*sql.DB
}
var db *DB
func init() {
dbinfo := fmt.Sprintf("user=%s password=%s dbname=%s sslmode=disable",
DB_USER, DB_PASSWORD, DB_NAME)
db, err := NewDB(dbinfo)
checkErr(err)
rows, err := db.Query("SELECT * FROM profile")
checkErr(err)
fmt.Println(rows)
}
func NewDB(dataSourceName string) (*DB, error) {
db, err := sql.Open("postgres", dataSourceName)
if err != nil {
return nil, err
}
if err = db.Ping(); err != nil {
return nil, err
}
return &DB{db}, nil
}
models/db_util.go
func (p *Profile) InsertProfile() {
if db != nil {
_, err := db.Exec(...)
checkErr(err)
} else {
fmt.Println("DB object is NULL")
}
}
When I try to access db in InsertProfile function, it says NULL ptr exception. How do I access the db in db_utils.go?
I would not like to capitalize db (as it would give access to all the packages).
I am getting the QUERY returned from the db in init() correctly.
Edit: The problem is that you used Short variable declaration := and you just stored the created *DB value in a local variable and not in the global one.
This line:
db, err := NewDB(dbinfo)
Creates 2 local variables: db and err, and this local db has nothing to do with your global db variable. Your global variable will remain nil. You have to assign the created *DB to the global variable. Do not use short variable declaration but simple assignment, e.g:
var err error
db, err = NewDB(dbinfo)
if err != nil {
log.Fatal(err)
}
Original answer follows.
It's a pointer type, you have to initialize it before you use it. The zero value for pointer types is nil.
You don't have to export it (that's what starting it with a capital letter does). Note that it doesn't matter that you have multiple files as long as they are part of the same package, they can access identifiers defined in one another.
A good solution would be to do it in the package init() function which is called automatically.
Note that sql.Open() may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call DB.Ping().
For example:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdrivername", "somesource")
if err != nil {
log.Fatal(err)
}
if err = db.Ping(); err != nil {
log.Fatal(err)
}
}
icza has already correctly answered your specific problem but it's worth adding some additional explanation on what you're doing wrong so you understand how not to make the mistake in the future. In Go, the syntax := for assignment creates new variables with the names to the left of the :=, possibly shadowing package, or even parent scope function/method variables. As an example:
package main
import "fmt"
var foo string = "global"
func main() {
fmt.Println(foo) // prints "global"
// using := creates a new function scope variable
// named foo that shadows the package scope foo
foo := "function scope"
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo := "nested scope"
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
}
// the foo created inside the if goes out of scope when
// the code block is exited
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo = "nested scope" // note just = not :=
}
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
setGlobalFoo()
printGlobalFoo() // prints "new value"
}
func printGlobalFoo() {
fmt.Println(foo)
}
func setGlobalFoo() {
foo = "new value" // note just = not :=
}
Note Go has no way to delete or unset a variable, so once you have shadowed a higher scope variables (such as by creating a function scope variable of the same name as a package scope variable), there is no way to access the higher scope variable within that code block.
Also be aware that := is a shorthand for var foo =. Both act in exactly the same way, however := is only valid syntax within a function or method, while the var syntax is valid everywhere.
For who came here and wants a fast answer.
in db.go file:
package db
var db *DB
type DB struct {
*gorm.DB // or what database you want like *mongo.Client
}
func GetDB() *DB {
if db == nil{
db = ConnectToYourDbFunc("connection_string")
}
return db
}
then in your other packages you can get it just with this:
db := db.GetDB()
thats all.
I want to convert a struct to map in Golang. It would also be nice if I could use the JSON tags as keys in the created map (otherwise defaulting to field name).
Edit Dec 14, 2020
Since structs repo was archived, you can use mapstructure instead.
Edit TL;DR version, Jun 15, 2015
If you want the fast solution for converting a structure to map, see the accepted answer, upvote it and use that package.
Happy coding! :)
Original Post
So far I have this function, I am using the reflect package but I don't understand well how to use the package, please bear with me.
func ConvertToMap(model interface{}) bson.M {
ret := bson.M{}
modelReflect := reflect.ValueOf(model)
if modelReflect.Kind() == reflect.Ptr {
modelReflect = modelReflect.Elem()
}
modelRefType := modelReflect.Type()
fieldsCount := modelReflect.NumField()
var fieldData interface{}
for i := 0; i < fieldsCount; i++ {
field := modelReflect.Field(i)
switch field.Kind() {
case reflect.Struct:
fallthrough
case reflect.Ptr:
fieldData = ConvertToMap(field.Interface())
default:
fieldData = field.Interface()
}
ret[modelRefType.Field(i).Name] = fieldData
}
return ret
}
Also I looked at JSON package source code, because it should contain my needed implementation (or parts of it) but don't understand too much.
I also had need for something like this. I was using an internal package which was converting a struct to a map. I decided to open source it with other struct based high level functions. Have a look:
https://github.com/fatih/structs
It has support for:
Convert struct to a map
Extract the fields of a struct to a []string
Extract the values of a struct to a []values
Check if a struct is initialized or not
Check if a passed interface is a struct or a pointer to struct
You can see some examples here: http://godoc.org/github.com/fatih/structs#pkg-examples
For example converting a struct to a map is a simple:
type Server struct {
Name string
ID int32
Enabled bool
}
s := &Server{
Name: "gopher",
ID: 123456,
Enabled: true,
}
// => {"Name":"gopher", "ID":123456, "Enabled":true}
m := structs.Map(s)
The structs package has support for anonymous (embedded) fields and nested structs. The package provides to filter certain fields via field tags.
From struct to map[string]interface{}
package main
import (
"fmt"
"encoding/json"
)
type MyData struct {
One int
Two string
Three int
}
func main() {
in := &MyData{One: 1, Two: "second"}
var inInterface map[string]interface{}
inrec, _ := json.Marshal(in)
json.Unmarshal(inrec, &inInterface)
// iterate through inrecs
for field, val := range inInterface {
fmt.Println("KV Pair: ", field, val)
}
}
go playground here
Here is a function I've written in the past to convert a struct to a map, using tags as keys
// ToMap converts a struct to a map using the struct's tags.
//
// ToMap uses tags on struct fields to decide which fields to add to the
// returned map.
func ToMap(in interface{}, tag string) (map[string]interface{}, error){
out := make(map[string]interface{})
v := reflect.ValueOf(in)
if v.Kind() == reflect.Ptr {
v = v.Elem()
}
// we only accept structs
if v.Kind() != reflect.Struct {
return nil, fmt.Errorf("ToMap only accepts structs; got %T", v)
}
typ := v.Type()
for i := 0; i < v.NumField(); i++ {
// gets us a StructField
fi := typ.Field(i)
if tagv := fi.Tag.Get(tag); tagv != "" {
// set key of map to value in struct field
out[tagv] = v.Field(i).Interface()
}
}
return out, nil
}
Runnable example here.
Note, if you have multiple fields with the same tag value, then you will obviously not be able to store them all within a map. It might be prudent to return an error if that happens.
I like the importable package for the accepted answer, but it does not translate my json aliases. Most of my projects have a helper function/class that I import.
Here is a function that solves my specific problem.
// Converts a struct to a map while maintaining the json alias as keys
func StructToMap(obj interface{}) (newMap map[string]interface{}, err error) {
data, err := json.Marshal(obj) // Convert to a json string
if err != nil {
return
}
err = json.Unmarshal(data, &newMap) // Convert to a map
return
}
And in the main, this is how it would be called...
package main
import (
"fmt"
"encoding/json"
"github.com/fatih/structs"
)
type MyStructObject struct {
Email string `json:"email_address"`
}
func main() {
obj := &MyStructObject{Email: "test#test.com"}
// My solution
fmt.Println(StructToMap(obj)) // prints {"email_address": "test#test.com"}
// The currently accepted solution
fmt.Println(structs.Map(obj)) // prints {"Email": "test#test.com"}
}
package main
import (
"fmt"
"reflect"
)
type bill struct {
N1 int
N2 string
n3 string
}
func main() {
a := bill{4, "dhfthf", "fdgdf"}
v := reflect.ValueOf(a)
values := make(map[string]interface{}, v.NumField())
for i := 0; i < v.NumField(); i++ {
if v.Field(i).CanInterface() {
values[v.Type().Field(i).Name] = v.Field(i).Interface()
} else {
fmt.Printf("sorry you have a unexported field (lower case) value you are trying to sneak past. I will not allow it: %v\n", v.Type().Field(i).Name)
}
}
fmt.Println(values)
passObject(&values)
}
func passObject(v1 *map[string]interface{}) {
fmt.Println("yoyo")
}
I'm a bit late but I needed this kind of feature so I wrote this. Can resolve nested structs. By default, uses field names but can also use custom tags. A side effect is that if you set the tagTitle const to json, you could use the json tags you already have.
package main
import (
"fmt"
"reflect"
)
func StructToMap(val interface{}) map[string]interface{} {
//The name of the tag you will use for fields of struct
const tagTitle = "kelvin"
var data map[string]interface{} = make(map[string]interface{})
varType := reflect.TypeOf(val)
if varType.Kind() != reflect.Struct {
// Provided value is not an interface, do what you will with that here
fmt.Println("Not a struct")
return nil
}
value := reflect.ValueOf(val)
for i := 0; i < varType.NumField(); i++ {
if !value.Field(i).CanInterface() {
//Skip unexported fields
continue
}
tag, ok := varType.Field(i).Tag.Lookup(tagTitle)
var fieldName string
if ok && len(tag) > 0 {
fieldName = tag
} else {
fieldName = varType.Field(i).Name
}
if varType.Field(i).Type.Kind() != reflect.Struct {
data[fieldName] = value.Field(i).Interface()
} else {
data[fieldName] = StructToMap(value.Field(i).Interface())
}
}
return data
}
map := Structpb.AsMap()
// map is the map[string]interface{}