How to use global var across files in a package? - pointers

I have the following file structure:
models/db.go
type DB struct {
*sql.DB
}
var db *DB
func init() {
dbinfo := fmt.Sprintf("user=%s password=%s dbname=%s sslmode=disable",
DB_USER, DB_PASSWORD, DB_NAME)
db, err := NewDB(dbinfo)
checkErr(err)
rows, err := db.Query("SELECT * FROM profile")
checkErr(err)
fmt.Println(rows)
}
func NewDB(dataSourceName string) (*DB, error) {
db, err := sql.Open("postgres", dataSourceName)
if err != nil {
return nil, err
}
if err = db.Ping(); err != nil {
return nil, err
}
return &DB{db}, nil
}
models/db_util.go
func (p *Profile) InsertProfile() {
if db != nil {
_, err := db.Exec(...)
checkErr(err)
} else {
fmt.Println("DB object is NULL")
}
}
When I try to access db in InsertProfile function, it says NULL ptr exception. How do I access the db in db_utils.go?
I would not like to capitalize db (as it would give access to all the packages).
I am getting the QUERY returned from the db in init() correctly.

Edit: The problem is that you used Short variable declaration := and you just stored the created *DB value in a local variable and not in the global one.
This line:
db, err := NewDB(dbinfo)
Creates 2 local variables: db and err, and this local db has nothing to do with your global db variable. Your global variable will remain nil. You have to assign the created *DB to the global variable. Do not use short variable declaration but simple assignment, e.g:
var err error
db, err = NewDB(dbinfo)
if err != nil {
log.Fatal(err)
}
Original answer follows.
It's a pointer type, you have to initialize it before you use it. The zero value for pointer types is nil.
You don't have to export it (that's what starting it with a capital letter does). Note that it doesn't matter that you have multiple files as long as they are part of the same package, they can access identifiers defined in one another.
A good solution would be to do it in the package init() function which is called automatically.
Note that sql.Open() may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call DB.Ping().
For example:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdrivername", "somesource")
if err != nil {
log.Fatal(err)
}
if err = db.Ping(); err != nil {
log.Fatal(err)
}
}

icza has already correctly answered your specific problem but it's worth adding some additional explanation on what you're doing wrong so you understand how not to make the mistake in the future. In Go, the syntax := for assignment creates new variables with the names to the left of the :=, possibly shadowing package, or even parent scope function/method variables. As an example:
package main
import "fmt"
var foo string = "global"
func main() {
fmt.Println(foo) // prints "global"
// using := creates a new function scope variable
// named foo that shadows the package scope foo
foo := "function scope"
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo := "nested scope"
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
}
// the foo created inside the if goes out of scope when
// the code block is exited
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo = "nested scope" // note just = not :=
}
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
setGlobalFoo()
printGlobalFoo() // prints "new value"
}
func printGlobalFoo() {
fmt.Println(foo)
}
func setGlobalFoo() {
foo = "new value" // note just = not :=
}
Note Go has no way to delete or unset a variable, so once you have shadowed a higher scope variables (such as by creating a function scope variable of the same name as a package scope variable), there is no way to access the higher scope variable within that code block.
Also be aware that := is a shorthand for var foo =. Both act in exactly the same way, however := is only valid syntax within a function or method, while the var syntax is valid everywhere.

For who came here and wants a fast answer.
in db.go file:
package db
var db *DB
type DB struct {
*gorm.DB // or what database you want like *mongo.Client
}
func GetDB() *DB {
if db == nil{
db = ConnectToYourDbFunc("connection_string")
}
return db
}
then in your other packages you can get it just with this:
db := db.GetDB()
thats all.

Related

Race condition while writing and reading from the map

Following up on old post here.
I am iterating over flatProduct.Catalogs slice and populating my productCatalog concurrent map in golang. I am using upsert method so that I can add only unique productID's into my productCatalog map.
Below code is called by multiple go routines in parallel that is why I am using concurrent map here to populate data into it. This code runs in background to populate data in the concurrent map every 30 seconds.
var productRows []ClientProduct
err = json.Unmarshal(byteSlice, &productRows)
if err != nil {
return err
}
for i := range productRows {
flatProduct, err := r.Convert(spn, productRows[i])
if err != nil {
return err
}
if flatProduct.StatusCode == definitions.DONE {
continue
}
r.products.Set(strconv.Itoa(flatProduct.ProductId, 10), flatProduct)
for _, catalogId := range flatProduct.Catalogs {
catalogValue := strconv.FormatInt(int64(catalogId), 10)
r.productCatalog.Upsert(catalogValue, flatProduct.ProductId, func(exists bool, valueInMap interface{}, newValue interface{}) interface{} {
productID := newValue.(int64)
if valueInMap == nil {
return map[int64]struct{}{productID: {}}
}
oldIDs := valueInMap.(map[int64]struct{})
// value is irrelevant, no need to check if key exists
// I think problem is here
oldIDs[productID] = struct{}{}
return oldIDs
})
}
}
And below are my getters in the same class where above code is there. These getters are used by main application threads to get data from the map or get the whole map.
func (r *clientRepository) GetProductMap() *cmap.ConcurrentMap {
return r.products
}
func (r *clientRepository) GetProductCatalogMap() *cmap.ConcurrentMap {
return r.productCatalog
}
func (r *clientRepository) GetProductData(pid string) *definitions.FlatProduct {
pd, ok := r.products.Get(pid)
if ok {
return pd.(*definitions.FlatProduct)
}
return nil
}
This is how I am reading data from this productCatalog cmap but my system is crashing on the below range statement -
// get productCatalog map which was populated above
catalogProductMap := clientRepo.GetProductCatalogMap()
productIds, ok := catalogProductMap.Get("211")
data, _ := productIds.(map[int64]struct{})
// I get panic here after sometime
for _, pid := range data {
...
}
Error I am getting as - fatal error: concurrent map iteration and map write.
I think issue is r.productCatalog is a concurrentmap, but oldIDs[productID] is a normal map which is causing issues while I am iterating in the for loop above.
How can I fix this race issue I am seeing? One way I can think of is making oldIDs[productID] as concurrent map but if I do that approach then my memory increase by a lot and eventually goes OOM. Below is what I have tried which works and it solves the race condition but it increases the memory by a lot which is not what I want -
r.productCatalog.Upsert(catalogValue, flatProduct.ProductId, func(exists bool, valueInMap interface{}, newValue interface{}) interface{} {
productID := newValue.(int64)
if valueInMap == nil {
// return map[int64]struct{}{productID: {}}
return cmap.New()
}
// oldIDs := valueInMap.(map[int64]struct{})
oldIDs := valueInMap.(cmap.ConcurrentMap)
// value is irrelevant, no need to check if key exists
// oldIDs[productID] = struct{}{}
oldIDs.Set(strconv.FormatInt(productID, 10), struct{}{})
return oldIDs
})
Any other approach I can do which doesn't increase memory and also fixes the race condition I am seeing?
Note
I am still using v1 version of cmap without generics and it deals with strings as keys.
Rather than a plain map[int64]struct{} type, you could define a struct which holds the map and a mutex to control the access to the map:
type myMap struct{
m sync.Mutex
data map[int64]struct{}
}
func (m *myMap) Add(productID int64) {
m.m.Lock()
defer m.m.Unlock()
m.data[productID] = struct{}{}
}
func (m *myMap) List() []int64 {
m.m.Lock()
defer m.m.Unlock()
var res []int64
for id := range m.data {
res = append(res, id)
}
// sort slice if you need
return res
}
With the sample implementation above, you would have to be careful to store *myMap pointers (as opposed to plain myMap structs) in your cmap.ConcurrentMap structure.

Wrapping a pointer in Go

A library foo exposes a type A and a function Fn in that library returns a *A.
I have defined a "wrapper" for A called B:
type B foo.A
Can I convert the *A to a *B without dereferencing the A?
In other words, if I have
a := foo.Fn() // a is a *A
b := B(*a)
return &b
How can I convert the *a to a *b without using *a?
The reason that I ask is that in the library that I am using, github.com/coreos/bbolt, the *DB value returned from the Open function includes a sync.Mutex and so the compiler complains when I try to make a copy of the Mutex.
UPDATE TO EXPLAIN HOW I'LL USE THIS
I have a
type Datastore struct {
*bolt.DB
}
I also have a function (one of many) like this:
func (ds *Datastore) ReadOne(bucket, id string, data interface{}) error {
return ds.View(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists([]byte(bucket))
if err != nil {
return fmt.Errorf("opening bucket %s: %v", bucket, err)
}
bytes := b.Get([]byte(id))
if bytes == nil {
return fmt.Errorf("id %s not found", id)
}
if err := json.Unmarshal(bytes, data); err != nil {
return fmt.Errorf("unmarshalling item: %v", err)
}
return nil
})
}
I would like to mock the underlying BoltDB database using a hash map. I ran into a problem mocking this because of the View expecting a function that takes bolt.Tx. That tx is then used to create a new bucket in CreateBucketIfNotExists. I cannot replace that anonymous function argument with one that calls my hash map mock version of CreateBucketIfNotExists.
I came up with this:
package boltdb
import (
"github.com/coreos/bbolt"
)
type (
bucket bolt.Bucket
// Bucket is a wrapper for bolt.Bucket to facilitate mocking.
Bucket interface {
ForEach(fn func([]byte, []byte) error) error
Get(key []byte) []byte
NextSequence() (uint64, error)
Put(key, value []byte) error
}
db bolt.DB
// DB is a wrapper for bolt.DB to facilitate mocking.
DB interface {
Close() error
Update(fn func(*Tx) error) error
View(fn func(*Tx) error) error
}
transaction bolt.Tx
// Tx is a wrapper for bolt.Tx to facilitate mocking.
Tx interface {
CreateBucketIfNotExists(name []byte) (Bucket, error)
}
)
// ForEach executes a function for each key/value pair in a bucket.
func (b *bucket) ForEach(fn func([]byte, []byte) error) error {
return ((*bolt.Bucket)(b)).ForEach(fn)
}
// Get retrieves the value for a key in the bucket.
func (b *bucket) Get(key []byte) []byte {
return ((*bolt.Bucket)(b)).Get(key)
}
// NextSequence returns an autoincrementing integer for the bucket.
func (b *bucket) NextSequence() (uint64, error) {
return ((*bolt.Bucket)(b)).NextSequence()
}
// Put sets the value for a key in the bucket.
func (b *bucket) Put(key, value []byte) error {
return ((*bolt.Bucket)(b)).Put(key, value)
}
// Close releases all database resources.
func (db *db) Close() error {
return ((*bolt.DB)(db)).Close()
}
// Update executes a function within the context of a read-write managed transaction.
func (db *db) Update(fn func(Tx) error) error {
return ((*bolt.DB)(db)).Update(func(tx *bolt.Tx) error {
t := transaction(*tx)
return fn(&t)
})
}
// View executes a function within the context of a managed read-only transaction.
func (db *db) View(fn func(Tx) error) error {
return ((*bolt.DB)(db)).View(func(tx *bolt.Tx) error {
t := transaction(*tx)
return fn(&t)
})
}
// CreateBucketIfNotExists creates a new bucket if it doesn't already exist.
func (tx *transaction) CreateBucketIfNotExists(name []byte) (Bucket, error) {
b, err := ((*bolt.Tx)(tx)).CreateBucketIfNotExists(name)
if err != nil {
return nil, err
}
w := bucket(*b)
return &w, nil
}
So far, in my code, I am only using the functions shown above. I can add more if new code requires.
I will replace each bolt.DB with DB, bolt.Tx with Tx, and bolt.Bucket with Bucket in the real code. The mocker will use replacements for all three types that use the underlying hash map instead of storing to disk. I can then test all of my code, right down to the database calls.
You can simply / directly convert a value of type *A to a value of type *B, you just have to parenthesize *B:
a := foo.Fn() // a is a *A
b := (*B)(a)
return b
You can even convert the return value of the function call:
return (*B)(foo.Fn())
Try it on the Go Playground.
This is possible, because Spec: Conversions:
A non-constant value x can be converted to type T in any of these cases:
x is assignable to T.
...
And Spec: Assignability:
A value x is assignable to a variable of type T ("x is assignable to T") if one of the following conditions applies:
...
x's type V and T have identical underlying types and at least one of V or T is not a defined type.
Both *B and *A types are not defined, and the underlying type of *B is the same as the underlying type of *A (which is the pointer to the underlying type of whatever type there is in the type declaration of A).

Golang Invalid Receiver Type in Method Func

I'm trying to make a simple package to send SSH commands to a server.
I have the following code:
type Connection *ssh.Client
func Connect(addr, user, password string) (conn Connection, err error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
return
}
func (conn Connection) SendCommand() ([]byte, error) {
session, err := (*ssh.Client)(conn).NewSession()
// ...
}
My problem is on the two lines func (conn Connection) SendCommand() ([]byte, error) and session, err := (*ssh.Client)(conn).NewSession().
I can't figure out how to use the methods available for *ssh.Client from my overlaying Connection type.
I understand that I need to do some conversion, and using ssh.Client(*conn).NewSession() would work, but it copies the values of the *ssh.Client which doesn't seem to be the right method.
What should do to access the methods available for a *ssh.Client when working with my custom type Connection *ssh.Client type?
You can't declare a new type with a pointer TypeSpec. Also declaring a new type is used specifically to remove the entire method set, so you won't have any of the original methods from the *ssh.Client.
What you want is to use composition by embedding the *ssh.Client in your own struct type:
type Connection struct {
*ssh.Client
}
func Connect(addr, user, password string) (*Connection, error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
if err != nil {
return nil, err
}
return &Connection{conn}, nil
}
func (conn *Connection) SendCommand() ([]byte, error) {
session, err := conn.NewSession()
// ...
}
This is the best I can come up with:
type Connection ssh.Client
func (conn *Connection) SendCommand() ([]byte, error) {
(*ssh.Client)(conn).NewSession()
Note that I've changed the type to not be a pointer type (but then I've made a pointer receiver for SendCommand). I'm not sure there's any way to create a function with a pointer type as a receiver.
Another option is to use type aliasing to achieve the desired behavior. I was trying to do something "clever" for readability:
type foo struct {
i int
}
type foo_ptr = *foo
type foo_ptr_slice = []foo_ptr
type foo_ptr_map = map[string]foo_ptr
type foo_ptr_slice_map = map[string]foo_ptr_slice
func (r foo_ptr) dump() {
fmt.Printf("%d\n", r.i)
}
func main() {
// need a map of slice of pointers
var m foo_ptr_map
m = make(foo_ptr_map, 0)
m["test"] = &foo{i: 1}
var m2 foo_ptr_slice_map
m2 = make(foo_ptr_slice_map, 0)
m2["test"] = make(foo_ptr_slice, 0, 10)
m2["test"] = append(m2["test"], &foo{i: 2})
fmt.Printf("%d\n", m["test"].i)
fmt.Printf("%d\n", m2["test"][0].i)
m["test"].dump()
}
I acknowledge that type aliasing is used for large-scale refactoring but this seems like a very good use for readability sake.

Passing structs to functions across packages

In an attempt to become more familiar with go, I am trying to refactor some code which is already working fine.
The original code has three structs:
type ConfigGroup struct {
Interval int
Tprefix string
Target []string
}
type ConfigDefaults struct {
Interval int
Sprefix string
}
type Config struct {
Group map[string]*ConfigGroup
Defaults ConfigDefaults
}
These structs get passed to a function like so:
func runpinger(clientStatsd statsd.Statter, defaults *ConfigDefaults, group *ConfigGroup) {
// some stuff here
}
Now, I've reworked the config (which uses gocfg) to use hcl instead, which seems to provide a cleaner config syntax.
I've moved the config parser into a package, config, with structs that look like this:
type Config struct {
Interval int `hcl:"interval"`
Prefix string `hcl:"prefix"`
Groups []TargetGroups `hcl:"target_group"`
}
type TargetGroups struct {
Name string `hcl:",key"`
Prefix string `hcl:"prefix"`
Interval int `hcl:"interval"`
Targets []Targets `hcl:"target"`
}
type Targets struct {
Address string `hcl:"address"`
Label string `hcl:"label"`
}
and then a function in config package that looks like this:
func Parse(ConfigFile string) (*Config, error) {
result := &Config{}
var errors *multierror.Error
config, err := ioutil.ReadFile(ConfigFile)
if err != nil {
return nil, err
}
hclParseTree, err := hcl.Parse(string(config))
if err != nil {
return nil, err
}
if err := hcl.DecodeObject(&result, hclParseTree); err != nil {
return nil, err
}
return result, errors.ErrorOrNil()
}
Now, in my main package I'd like to pass these structs to the function again. How can I do this across packages?
I tried:
func(runpinger config *config.Config) {
// here
}
But that didn't seem to work. Ideally, I'd like to just pass a pointer to the "sub-struct" (ie the TargetGroups struct) as well, although I'm not sure if that's possible.
You should be able to pass the structs to the main package, just check that you put import "path/to/config" at the top of your file.
The path has to be the full path to your package from your $GOPATH/src/ directory

Returning File Pointer in Golang

I'm still struggling with the basics of Golang.
Consider the following sample code:
func OpenOutputFile(name string) (fp *os.File) {
fp, err := os.Create(name)
if err != nil {
panic(err)
}
defer func() {
if err := fp.Close(); err != nil {
panic(err)
}
}()
return fp
}
I would assume that calling:
fp := OpenOutputFile("output.txt")
would now make fp a file pointer (*os.File), so that I could call a statement like:
io.WriteString(fp, "Hello World")
In another function. But when calling this method, the error is generated:
0 write output.txt: bad file descriptor
So it appears that the pointer returned is not valid. How can I return a properly formed pointer to use with io.WriteString?
I appreciate the help!
Of note: Everything executes as intended when the creation of the file pointer and the writing to the file pointer exists in the same method. Breaking the logic into a function causes it to not behave as intended.
The Go Programming Language Specification
Defer statements
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns, either because the
surrounding function executed a return statement, reached the end of
its function body, or because the corresponding goroutine is
panicking.
Each time a "defer" statement executes, the function value and
parameters to the call are evaluated as usual and saved anew but the
actual function is not invoked. Instead, deferred functions are
invoked immediately before the surrounding function returns, in the
reverse order they were deferred. If a deferred function value
evaluates to nil, execution panics when the function is invoked, not
when the "defer" statement is executed.
For instance, if the deferred function is a function literal and the
surrounding function has named result parameters that are in scope
within the literal, the deferred function may access and modify the
result parameters before they are returned. If the deferred function
has any return values, they are discarded when the function completes.
func OpenOutputFile(name string) (fp *os.File) {
fp, err := os.Create(name)
if err != nil {
panic(err)
}
defer func() {
if err := fp.Close(); err != nil {
panic(err)
}
}()
return fp
}
You open the file
fp, err := os.Create(name)
You close the file
err := fp.Close()
After the Close, fp no longer points to a valid file descriptor.
Returning the close function and deferring it in higher scope worked for me. I don't know if this is a good practice in Go though. It relies on closing/deferring outside of original function:
func OpenFileFromArgs() (*os.File, func()) {
if len(os.Args) < 2 {
panic("input file not provided")
}
inputFilePath := os.Args[1]
stat, err := os.Stat(inputFilePath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
panic(fmt.Sprintf("file %s doens't exist", inputFilePath))
} else {
panic(fmt.Sprintf("error: %v", err))
}
}
if stat.IsDir() {
panic("provided path is a directory")
}
inputFile, err := os.Open(inputFilePath)
closeFn := func() {
err := inputFile.Close()
if err != nil {
panic("failed to close input file")
}
}
return inputFile, closeFn
and then in the higher scope:
inputFile, closeFn := library.OpenFileFromArgs()
defer closeFn()
Confirmed using step by step debugger - at program end, the deferred close function is correctly called and file descriptor closed.

Resources