Returning File Pointer in Golang - pointers

I'm still struggling with the basics of Golang.
Consider the following sample code:
func OpenOutputFile(name string) (fp *os.File) {
fp, err := os.Create(name)
if err != nil {
panic(err)
}
defer func() {
if err := fp.Close(); err != nil {
panic(err)
}
}()
return fp
}
I would assume that calling:
fp := OpenOutputFile("output.txt")
would now make fp a file pointer (*os.File), so that I could call a statement like:
io.WriteString(fp, "Hello World")
In another function. But when calling this method, the error is generated:
0 write output.txt: bad file descriptor
So it appears that the pointer returned is not valid. How can I return a properly formed pointer to use with io.WriteString?
I appreciate the help!
Of note: Everything executes as intended when the creation of the file pointer and the writing to the file pointer exists in the same method. Breaking the logic into a function causes it to not behave as intended.

The Go Programming Language Specification
Defer statements
A "defer" statement invokes a function whose execution is deferred to
the moment the surrounding function returns, either because the
surrounding function executed a return statement, reached the end of
its function body, or because the corresponding goroutine is
panicking.
Each time a "defer" statement executes, the function value and
parameters to the call are evaluated as usual and saved anew but the
actual function is not invoked. Instead, deferred functions are
invoked immediately before the surrounding function returns, in the
reverse order they were deferred. If a deferred function value
evaluates to nil, execution panics when the function is invoked, not
when the "defer" statement is executed.
For instance, if the deferred function is a function literal and the
surrounding function has named result parameters that are in scope
within the literal, the deferred function may access and modify the
result parameters before they are returned. If the deferred function
has any return values, they are discarded when the function completes.
func OpenOutputFile(name string) (fp *os.File) {
fp, err := os.Create(name)
if err != nil {
panic(err)
}
defer func() {
if err := fp.Close(); err != nil {
panic(err)
}
}()
return fp
}
You open the file
fp, err := os.Create(name)
You close the file
err := fp.Close()
After the Close, fp no longer points to a valid file descriptor.

Returning the close function and deferring it in higher scope worked for me. I don't know if this is a good practice in Go though. It relies on closing/deferring outside of original function:
func OpenFileFromArgs() (*os.File, func()) {
if len(os.Args) < 2 {
panic("input file not provided")
}
inputFilePath := os.Args[1]
stat, err := os.Stat(inputFilePath)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
panic(fmt.Sprintf("file %s doens't exist", inputFilePath))
} else {
panic(fmt.Sprintf("error: %v", err))
}
}
if stat.IsDir() {
panic("provided path is a directory")
}
inputFile, err := os.Open(inputFilePath)
closeFn := func() {
err := inputFile.Close()
if err != nil {
panic("failed to close input file")
}
}
return inputFile, closeFn
and then in the higher scope:
inputFile, closeFn := library.OpenFileFromArgs()
defer closeFn()
Confirmed using step by step debugger - at program end, the deferred close function is correctly called and file descriptor closed.

Related

Golang Invalid Receiver Type in Method Func

I'm trying to make a simple package to send SSH commands to a server.
I have the following code:
type Connection *ssh.Client
func Connect(addr, user, password string) (conn Connection, err error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
return
}
func (conn Connection) SendCommand() ([]byte, error) {
session, err := (*ssh.Client)(conn).NewSession()
// ...
}
My problem is on the two lines func (conn Connection) SendCommand() ([]byte, error) and session, err := (*ssh.Client)(conn).NewSession().
I can't figure out how to use the methods available for *ssh.Client from my overlaying Connection type.
I understand that I need to do some conversion, and using ssh.Client(*conn).NewSession() would work, but it copies the values of the *ssh.Client which doesn't seem to be the right method.
What should do to access the methods available for a *ssh.Client when working with my custom type Connection *ssh.Client type?
You can't declare a new type with a pointer TypeSpec. Also declaring a new type is used specifically to remove the entire method set, so you won't have any of the original methods from the *ssh.Client.
What you want is to use composition by embedding the *ssh.Client in your own struct type:
type Connection struct {
*ssh.Client
}
func Connect(addr, user, password string) (*Connection, error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
if err != nil {
return nil, err
}
return &Connection{conn}, nil
}
func (conn *Connection) SendCommand() ([]byte, error) {
session, err := conn.NewSession()
// ...
}
This is the best I can come up with:
type Connection ssh.Client
func (conn *Connection) SendCommand() ([]byte, error) {
(*ssh.Client)(conn).NewSession()
Note that I've changed the type to not be a pointer type (but then I've made a pointer receiver for SendCommand). I'm not sure there's any way to create a function with a pointer type as a receiver.
Another option is to use type aliasing to achieve the desired behavior. I was trying to do something "clever" for readability:
type foo struct {
i int
}
type foo_ptr = *foo
type foo_ptr_slice = []foo_ptr
type foo_ptr_map = map[string]foo_ptr
type foo_ptr_slice_map = map[string]foo_ptr_slice
func (r foo_ptr) dump() {
fmt.Printf("%d\n", r.i)
}
func main() {
// need a map of slice of pointers
var m foo_ptr_map
m = make(foo_ptr_map, 0)
m["test"] = &foo{i: 1}
var m2 foo_ptr_slice_map
m2 = make(foo_ptr_slice_map, 0)
m2["test"] = make(foo_ptr_slice, 0, 10)
m2["test"] = append(m2["test"], &foo{i: 2})
fmt.Printf("%d\n", m["test"].i)
fmt.Printf("%d\n", m2["test"][0].i)
m["test"].dump()
}
I acknowledge that type aliasing is used for large-scale refactoring but this seems like a very good use for readability sake.

Golang return pointer to interface throws error

I have a basic function in Go that opens a file and tries to decode its JSON contents.
I am trying to extract the default json.NewDecoder() function so I can easily mock this in my tests.
However, my implementation seems to return an error:
cannot use json.NewDecoder (type func(io.Reader) *json.Decoder) as type decoderFactory in argument to NewConfig
Code:
package main
import (
"encoding/json"
"fmt"
"io"
"os"
)
type openFile func(name string) (*os.File, error)
type decoderFactory func(r io.Reader) decoder
type decoder interface {
Decode(v interface{}) error
}
type Config struct {
ConsumerKey,
ConsumerSecret,
AccessToken,
AccessTokenSecret string
}
func NewConfig(open openFile, d decoderFactory) (*Config, error) {
c := new(Config)
file, err := open("some.file")
if err != nil {
return nil, fmt.Errorf("error opening config file")
}
defer file.Close()
decoder := d(file)
if err := decoder.Decode(&c); err != nil {
return nil, fmt.Errorf("error decoding config JSON")
}
return c, nil
}
func main() {
_, err := NewConfig(os.Open, json.NewDecoder)
if err != nil {
fmt.Fprintf(os.Stderr, "something bad happened: %v\n", err)
}
}
Here's a link to the Go playground
Where am I going wrong?
The json.NewDecoder() is a function with the following declaration:
func NewDecoder(r io.Reader) *Decoder
Its return type is *json.Decoder. json.Decoder is not an interface, it's a concrete type. And 2 function types are different if their return type is different: Spec: Function types:
A function type denotes the set of all functions with the same parameter and result types.
So you can't construct a new type returning an interface, and expect to be the same as json.NewDecoder, or that it'll accept the value json.NewDecoder.
But the "seemingly" easy fix is: define your decoderFactory to be a function type exactly what json.NewDecoder is:
type decoderFactory func(r io.Reader) *json.Decoder
This compiles, ok... but how to mock now?
How to mock now?
Of course in this form, you'll lose the possibility to mock json.NewDecoder() (because a "mocker" would have to return a value of type *json.Decoder and nothing else would be accepted). What to do then?
You have to use a different factory type. The factory type should be a function which returns an interface (of which you can provide different implementations), you were on the right track:
type MyDecoder interface {
Decode(v interface{}) error
// List other methods that you need from json.Decoder
}
type decoderFactory func(r io.Reader) MyDecoder
But you can't use json.NewEncoder as-is to pass as a value of decoderFactory. But fear not, it is very easy to create a function of type decoderFactory which will call json.NewEncoder() under the hood:
func jsonDecoderFact(r io.Reader) MyDecoder {
return json.NewDecoder(r)
}
We're mocking the behaviour of json.Decoder, and not the json.NewDecoder() factory function.
Using this jsonDecoderFact():
_, err := NewConfig(os.Open, jsonDecoderFact)
if err != nil {
fmt.Fprintf(os.Stderr, "something bad happened: %v\n", err)
}
This is valid and compiles, because jsonDecoderFact has exactly the same type as decoderFactory.
If you want to test / mock with a different implementation:
type TestDecoder struct {
r io.Reader
}
func (t TestDecoder) Decode(v interface{}) error {
// Test / mocking logic here
return nil
}
func testDecoderFact(r io.Reader) MyDecoder {
return TestDecoder{r}
}
Using it:
_, err2 := NewConfig(os.Open, testDecoderFact)
if err2 != nil {
fmt.Fprintf(os.Stderr, "something bad happened: %v\n", err2)
}
Try the examples on the Go Playground.

Passing values to interface{}

Short
The following code does not exactly do what expected:
https://play.golang.org/p/sO4w4I_Lle
I assume that I mess up some pointer/reference stuff as usual, however I expect my...
func unmarshalJSON(in []byte, s interface{}) error
... and encoding/jsons...
func Unmarshal(data []byte, v interface{}) error
...to behave the same way (eg. update the referenced passed as second argument).
Long
The example above is a minimal reproducer that does not make much sense. This is in order to make it work on the playground. However, an less minimal example that does make sense is this:
package main
import (
"fmt"
"gopkg.in/yaml.v2"
)
func unmarshalYAML(in []byte, s interface{}) error {
var result map[interface{}]interface{}
err := yaml.Unmarshal(in, &result)
s = cleanUpInterfaceMap(result)
// s is printed as expected
fmt.Println(s) // map[aoeu:[test aoeu] oaeu:[map[mahl:aoec tase:aoeu]]]
return err
}
func cleanUpInterfaceArray(in []interface{}) []interface{} {
out := make([]interface{}, len(in))
for i, v := range in {
out[i] = cleanUpMapValue(v)
}
return out
}
func cleanUpInterfaceMap(in map[interface{}]interface{}) map[string]interface{} {
out := make(map[string]interface{})
for k, v := range in {
out[fmt.Sprintf("%v", k)] = cleanUpMapValue(v)
}
return out
}
func cleanUpMapValue(v interface{}) interface{} {
switch v := v.(type) {
case []interface{}:
return cleanUpInterfaceArray(v)
case map[interface{}]interface{}:
return cleanUpInterfaceMap(v)
case string:
return v
default:
return fmt.Sprintf("%v", v)
}
}
func main() {
s := make(map[string]interface{})
b := []byte(`---
aoeu:
- test
- aoeu
oaeu:
- { tase: aoeu, mahl: aoec}
`)
err := unmarshalYAML(b, &s)
if err != nil {
panic(err)
}
// s is still an empty map
fmt.Println(s) // map[]
}
The idea is to unmarshal YAML to map[string]interface{} (instead of map[interface{}]interface{}) is order to allow to serialize to JSON (where identifiers need to be strings). The unmarshalYAML function should provide the same func signture as yaml.Unmarshal...
Using Type assertion
Inside your unmarshalJSON() function the parameter s behaves like a local variable. When you assign something to it:
s = result
It will only change the value of the local variable.
Since you want it to work with changing the value of a *map[string]interface{} and that is what you pass to it, you could use a simple type assertion to obtain the map pointer from it, and pass this pointer to json.Unmarshal():
func unmarshalJSON(in []byte, s interface{}) error {
if m, ok := s.(*map[string]interface{}); !ok {
return errors.New("Expecting *map[string]interface{}")
} else {
return json.Unmarshal(in, m)
}
}
Try your modified, working example on the Go Playground.
Just passing it along
Also note that however this is completely unnecessary as json.Unmarshal() is also defined to take the destination as a value of type interface{}, the same thing you have. So you don't even have to do anything just pass it along:
func unmarshalJSON(in []byte, s interface{}) error {
return json.Unmarshal(in, s)
}
Try this on the Go Playground.
With a variable of function type
As an interesting thing note that the signature of your unmarshalJSON() and the library function json.Unmarshal() is identical:
// Yours:
func unmarshalJSON(in []byte, s interface{}) error
// json package
func Unmarshal(data []byte, v interface{}) error
This means there is another option, that is you could use a variable named unmarshalJSON of a function type, and just simply assign the function value json.Unmarshal:
var unmarshalJSON func([]byte, interface{}) error = json.Unmarshal
Now you have a variable unmarshalJSON which is of function type, and you can call it as if it would be a function:
err := unmarshalJSON(b, &s)
Try this function value on the Go Playground.
Now on to your unmarshalYAML() function
In your unmarshalYAML() you do the same mistake:
s = cleanUpInterfaceMap(result)
This will only change the value of your local s variable (parameter), and it will not "populate" the map (pointer) passed to unmarshalYAML().
Use the type assertion technique detailed above to obtain the pointer from the s interface{} argument, and once you have that, you can change the pointed object (the "outside" map).
func unmarshalYAML(in []byte, s interface{}) error {
var dest *map[string]interface{}
var ok bool
if dest, ok = s.(*map[string]interface{}); !ok {
return errors.New("Expecting *map[string]interface{}")
}
var result map[interface{}]interface{}
if err := yaml.Unmarshal(in, &result); err != nil {
return err
}
m := cleanUpInterfaceMap(result)
// m holds the results, dest is the pointer that was passed to us,
// we can just set the pointed object (map):
*dest = m
return nil
}

How to use global var across files in a package?

I have the following file structure:
models/db.go
type DB struct {
*sql.DB
}
var db *DB
func init() {
dbinfo := fmt.Sprintf("user=%s password=%s dbname=%s sslmode=disable",
DB_USER, DB_PASSWORD, DB_NAME)
db, err := NewDB(dbinfo)
checkErr(err)
rows, err := db.Query("SELECT * FROM profile")
checkErr(err)
fmt.Println(rows)
}
func NewDB(dataSourceName string) (*DB, error) {
db, err := sql.Open("postgres", dataSourceName)
if err != nil {
return nil, err
}
if err = db.Ping(); err != nil {
return nil, err
}
return &DB{db}, nil
}
models/db_util.go
func (p *Profile) InsertProfile() {
if db != nil {
_, err := db.Exec(...)
checkErr(err)
} else {
fmt.Println("DB object is NULL")
}
}
When I try to access db in InsertProfile function, it says NULL ptr exception. How do I access the db in db_utils.go?
I would not like to capitalize db (as it would give access to all the packages).
I am getting the QUERY returned from the db in init() correctly.
Edit: The problem is that you used Short variable declaration := and you just stored the created *DB value in a local variable and not in the global one.
This line:
db, err := NewDB(dbinfo)
Creates 2 local variables: db and err, and this local db has nothing to do with your global db variable. Your global variable will remain nil. You have to assign the created *DB to the global variable. Do not use short variable declaration but simple assignment, e.g:
var err error
db, err = NewDB(dbinfo)
if err != nil {
log.Fatal(err)
}
Original answer follows.
It's a pointer type, you have to initialize it before you use it. The zero value for pointer types is nil.
You don't have to export it (that's what starting it with a capital letter does). Note that it doesn't matter that you have multiple files as long as they are part of the same package, they can access identifiers defined in one another.
A good solution would be to do it in the package init() function which is called automatically.
Note that sql.Open() may just validate its arguments without creating a connection to the database. To verify that the data source name is valid, call DB.Ping().
For example:
var db *sql.DB
func init() {
var err error
db, err = sql.Open("yourdrivername", "somesource")
if err != nil {
log.Fatal(err)
}
if err = db.Ping(); err != nil {
log.Fatal(err)
}
}
icza has already correctly answered your specific problem but it's worth adding some additional explanation on what you're doing wrong so you understand how not to make the mistake in the future. In Go, the syntax := for assignment creates new variables with the names to the left of the :=, possibly shadowing package, or even parent scope function/method variables. As an example:
package main
import "fmt"
var foo string = "global"
func main() {
fmt.Println(foo) // prints "global"
// using := creates a new function scope variable
// named foo that shadows the package scope foo
foo := "function scope"
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo := "nested scope"
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
}
// the foo created inside the if goes out of scope when
// the code block is exited
fmt.Println(foo) // prints "function scope"
printGlobalFoo() // prints "global"
if true {
foo = "nested scope" // note just = not :=
}
fmt.Println(foo) // prints "nested scope"
printGlobalFoo() // prints "global"
setGlobalFoo()
printGlobalFoo() // prints "new value"
}
func printGlobalFoo() {
fmt.Println(foo)
}
func setGlobalFoo() {
foo = "new value" // note just = not :=
}
Note Go has no way to delete or unset a variable, so once you have shadowed a higher scope variables (such as by creating a function scope variable of the same name as a package scope variable), there is no way to access the higher scope variable within that code block.
Also be aware that := is a shorthand for var foo =. Both act in exactly the same way, however := is only valid syntax within a function or method, while the var syntax is valid everywhere.
For who came here and wants a fast answer.
in db.go file:
package db
var db *DB
type DB struct {
*gorm.DB // or what database you want like *mongo.Client
}
func GetDB() *DB {
if db == nil{
db = ConnectToYourDbFunc("connection_string")
}
return db
}
then in your other packages you can get it just with this:
db := db.GetDB()
thats all.

Is it Necessary to Check "n" returned by Write In Golang

I have a server and start a goroutine for sending data. Code like this
func handleSend(conn *net.TcpConn, ch <-chan []byte) {
for {
select {
case msg, ok := <-ch:
if !ok {
return
}
n, err := conn.Write(msg)
if err != nil {
log.Error("conn write error", err)
return
}
//here
}
}
}
Today, I think I should check n return by conn.Write to make sure that msg is writed completely. So I add the following code in the place here
for ;n!= len(msg);{
log.Error("conn write not completely", len(msg), "actually", n)
msg = msg[n:]
n, err = sess.conn.Write(msg)
if err != nil {
log.Error("conn write error", err)
return
}
}
And I want to know is it right to do this?
PS what if the peer receive message slowly and Write return with only part of the data sended successfully ?
Docs say Write must return a non-nil error if it returns n < len(p). You may well find there's not much you can do with n, in which case you can assign it to _.

Resources