How to pretty print the contents of a sync Map - dictionary

I want to pretty print the contents of a sync map in Go.
I have a sync.Map data I want to print the contents of it.
To view the value of a specific key SiteData I can run the following code.
var data sync.Map
siteData := map[string]string{"Name": "StackOverflow"}
data.Store("SiteData", siteData)
temp, _ := data.Load("SiteData")
b, _ := json.MarshalIndent(temp, "", " ")
fmt.Println(string(b))
But I wish to print the entire map at once. This is because the data can have many keys and I want to print them at once.
Running the below code doesn't work and prints {}
var data sync.Map
siteData := map[string]string{"Name": "StackOverflow"}
data.Store("SiteData", siteData)
b, _ := json.MarshalIndent(data, "", " ")
fmt.Println(string(b))

Fields (internals) of sync.Map are not exported, so they can't be accessed, and more importantly they can't be accessed without synchronization. So you can't just print the contents of a sync.Map.
What you may do is iterate over all entries of your sync.Map, build an "ordinary" map from it, and display that. Note that the "ordinary" map must have string key type (maps with interface{} key type are not supported by the encoding/json package). We may simply convert interface{} keys to string using fmt.Sprint(). To get all entries, you may use Map.Range().
For example:
var data sync.Map
data.Store("SiteData", map[string]string{
"Name": "StackOverflow",
"Url": "https://so.com",
})
data.Store("Else", "something else")
m := map[string]interface{}{}
data.Range(func(key, value interface{}) bool {
m[fmt.Sprint(key)] = value
return true
})
b, err := json.MarshalIndent(m, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(b))
This will output (try it on the Go Playground):
{
"Else": "something else",
"SiteData": {
"Name": "StackOverflow",
"Url": "https://so.com"
}
}

Here is a simple util function (you may modify base on it as need):
// print map's key/value, with index (not stable),
func PrintSyncMap(m sync.Map) {
// print map,
fmt.Println("map content:")
i := 0
m.Range(func(key, value interface{}) bool {
fmt.Printf("\t[%d] key: %v, value: %v\n", i, key, value)
i++
return true
})
}
Example output:
map content:
[0] key: Qian, value: class 2
[1] key: Zhao, value: class 1
[2] key: Li, value: class 4
Tips:
Index is not stable. (aka. Order of printed item may change among multiple calls.)

Related

Adding nested struct to Firestore

I am trying to add a nested struct to Firestore and for some reason the contents added are all non-structs, which look something like:
The structs look something like this:
type Status struct {
Title string `json:"title,omitempty" firestore:"title,omitempty"`
Message string `json:"message,omitempty" firestore:"title,omitempty"`
}
type Config struct {
Name string `json:"name,omitempty" firestore:"name,omitempty"`
Status Status `json:"status,omitempty" firestore:"status,omitempty"`
}
And the code looks something like this:
import (
"context"
firebase "firebase.google.com/go/v4"
"google.golang.org/api/option"
)
func main() {
configuration := Config{
Name: "Test",
Status: Status{
Title: "hello",
Message: "hi",
},
}
ctx := context.Background()
config := firebase.Config{
ProjectID: "",
StorageBucket: "",
}
opt := option.WithCredentialsFile("firebase_config.json")
app, err := firebase.NewApp(ctx, &config, opt)
if err != nil {
panic(err)
}
// Get an auth client from the firebase.App
client, err := app.Firestore(ctx)
if err != nil {
panic(err)
}
_, _, err = client.Collection("forecast").Add(ctx, configuration)
if err != nil {
panic(err)
}
}
The above code only works for elements that are not in the nested structure.
Any help on this would be appreciated
Update 1
Status is not a sub collection but an object, something like:
{
"name": "Test",
"status": {
"title": "hello",
"message": "hi"
}
}
Firestore is optimized as a hash entry and retrieval datastore. As a consequence, it's better to create maps out of your structs. Structs are good for Go data modeling but when it's time to submit to the database, convert it to a map.
I usually just use Fatih's struct to map converter
It makes it easy to reason about your data on the Go side and still be able to submit it for storage.
Posting this as Community Wiki answer, based in the discussion of the comments.
The solution for this case seems to be adding values manually in a field of type Map. The steps to achieve that are the following: Access the Firebase console -> Firestore -> Create a document -> Add field of type Map. Following this order it's possible to create a field of type Map, that has the format needed to add data as described in the description of the question.
More information about this type, how it works, sort options, etc., can be found in the official documentation here: Supported data types.

How to convert entire nested struct into map

I have a scenario where I have to convert the entire struct into a map.
I know we have a library structs.Map(s) which will convert the struct to a map. But I want to know is there a way where i can convert multiple struct inside struct to map[string]interface.
for example we have below
package main
import (
"log"
"github.com/fatih/structs"
)
type Community struct {
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
Sources []Source `json:"sources,omitempty"`
Moderators []string `json:"moderators,omitempty"`
}
type Source struct {
SourceName string `json:"sourceName,omitempty"`
Region []State `json:"region,omitempty"`
}
type State struct {
State1 string `json:"state1,omitempty"`
State2 string `json:"state2,omitempty"`
}
func main() {
compareData := Community{
Name: "A",
Description: "this belong to A community",
Sources: []Source{
{
SourceName: "SourceA",
Region: []State{
{
State1: "State1",
},
{
State2: "State2",
},
},
},
},
}
m := structs.Map(compareData)
log.Println(m)
}
this will give result as below ,that is it is creating map for the inside struct again
map[Description:this belong to A community
Moderators:[]
Name:A Sources:[map[SourceName:SourceA Region:[map[State1:State1 State2:] map[State1: State2:State2]]]]]
my expectation is get only a single map[string]interface{}
map[
Description:this belong to A community
Moderators:[]
Name:A
SourceName:SourceA
State1:State1
State2:State2
]
my purpose of creating a single map is to compare the value to a different map based on key .
My struct also varies as per the different response so i want to have a map where i can get all the key value pairs for easy comparison . If someone has a suggestion to this please let me know .
You can use mapstructure package.
Sample of usage :
package main
import (
"fmt"
"github.com/mitchellh/mapstructure"
)
func main() {
type Emails struct {
Mail []string
}
type Person struct {
Name string
Age int
Emails Emails
Extra map[string]string
}
// This input can come from anywhere, but typically comes from
// something like decoding JSON where we're not quite sure of the
// struct initially.
mails := []string{"foo#bar.com", "foo2#bar.com"}
input := Person{
Name: "foo",
Age: 25,
Emails: Emails{Mail: mails},
Extra: map[string]string{"family": "bar"},
}
result := map[string]interface{}{}
err := mapstructure.Decode(input, &result)
if err != nil {
panic(err)
}
fmt.Printf("%#v", result)
}
playground

Why is a map value becoming non-existent?

I am using a map to store random string keys to *os.File objects. Users will be uploading a file and I want to hold a reference to the file in a global map so I can delete it later.
I have an http handler to process the upload and at the end, I map a random key from the OS uuidgen to "logBundleFile" which is type *os.File.
var db = map[string]*os.File{}
func uploadHandler(w http.ResponseWriter, r *http.Request) {
r.ParseMultipartForm(5 << 30)
file, handler, err := r.FormFile("file")
if err != nil {
log.Fatalf("Error retrieving the file: %v", err)
return
}
defer file.Close()
logBundleFile, err := ioutil.TempFile("", handler.Filename)
if err != nil {
log.Fatal(err)
}
defer logBundleFile.Close()
fileBytes, err := ioutil.ReadAll(file)
if err != nil {
log.Fatalf("Error reading file: %v", err)
}
logBundleFile.Write(fileBytes)
id, err := exec.Command("uuidgen").Output()
idStr := string(id[:])
//id := "1"
if err != nil {
log.Fatal(err)
}
db[idStr] = logBundleFile
log.Printf("ID: %v Type: %T\n", idStr, idStr)
log.Printf("val: %v Type: %T\n\n", db[idStr], db[idStr])
http.Redirect(w, r, fmt.Sprintf("/%s", idStr), http.StatusMovedPermanently)
}
Once that is done, you get redirected to this sessionHandler. It will check if the ID in the body is valid, i.e, mapped to a *os.File. The "ok" bool is always returning false.
func sessionHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
id := vars["id"]
log.Printf("ID: %v Type: %T\n", id, id)
log.Printf("val: %v Type: %T\n", db[id], db[id])
if val, ok := db[id]; ok {
w.Write([]byte(fmt.Sprintf("Session %s %v", id, val)))
} else {
http.Redirect(w, r, "/", http.StatusMovedPermanently)
}
}
Here is an output from the prints. In the uploadHandler, we can see that we have a string key mapped to a non-nil *os.File.
But in the session handler, the same string key maps to a nil *os.File. I don't know what is going on.
2019/08/27 19:49:49 ID: BA06C157-451E-48B5-85F9-8069D9A4EFCE
Type: string
2019/08/27 19:49:49 val: &{0xc000160120} Type: *os.File
2019/08/27 19:49:49 ID: BA06C157-451E-48B5-85F9-8069D9A4EFCE Type: string
2019/08/27 19:49:49 val: <nil> Type: *os.File
It's because in the uploadHandler, the id variable contains newline. If we take a look closely on the log we can see it. somehow Type: string text is printed in the 2nd line.
2019/08/27 19:49:49 ID: BA06C157-451E-48B5-85F9-8069D9A4EFCE // <-- newline
Type: string
2019/08/27 19:49:49 ID: BA06C157-451E-48B5-85F9-8069D9A4EFCE Type: string
Putting trim operation on the idStr should solve the problem.
idStr := strings.TrimSpace(string(id[:]))

How to check value in nested pointer

How can I check nested pointer easily?
type inner1 struct {
value string
}
type inner2 struct {
value *inner1
}
type outter struct {
value *inner2
}
I have data like this:
o := &outter{
value: &inner2{
value: &inner1{
value: "I need this data",
},
},
}
If I want to get data from this, I need to check for nil pointer.
func printValue(o *outter) (string, bool) {
if o.value != nil {
v1 := o.value
if v1.value != nil {
v2 := v1.value
return v2.value, true
}
}
return "", false
}
Yea, I can do this. But in my real scenario, this nested pointer part is much longer. I will prefer another way.
I have checked this answer, Test for nil values in golang nested stucts. But I need alternative.
How can I accomplish this. Any efficient and better option?
Panic and recover is for exceptional cases. Checking if a pointer is nil is usually not, so you should stay away from it. Checking if a pointer is nil using an if statement is much cleaner than to introduce a deferred function with recover(). Especially as this recover() will stop all other kinds of panic, even those that would result from other reasons than trying to dereference a nil pointer. (Using defer and recover() is also slower than a simple nil check using if.)
If the data is always required to be a non-nil value, you should consider not using a pointer for it. And if the data is always required to be non-nil but for some reason you are required to use a pointer, then this may be a valid case to let your code panic if any of the pointers are still nil.
If it may be nil, then you have to check the nil case and handle it appropriately. You have to be explicit about this, Go doesn't help you omit this check.
To avoid having to check nil in every place, a utility function or method is reasonable. Note that methods can be called even if the receiver is nil which may be useful in such cases.
For example you may attach the following methods:
func (i *inner1) Get() (string, bool) {
if i == nil {
return "", false
}
return i.value, true
}
func (i *inner2) Get() (string, bool) {
if i == nil {
return "", false
}
return i.value.Get()
}
func (o *outter) Get() (string, bool) {
if o == nil {
return "", false
}
return o.value.Get()
}
Note that each Get() method requires to check a single pointer, doesn't matter how complex the data structure is.
Testing it:
o := &outter{
value: &inner2{
value: &inner1{
value: "I need this data",
},
},
}
fmt.Println(o.Get())
o.value.value = nil
fmt.Println(o.Get())
o.value = nil
fmt.Println(o.Get())
o = nil
fmt.Println(o.Get())
Output (try it on the Go Playground):
I need this data true
false
false
false
The above solution hides the internals of outter which is useful for those using outter (doesn't need updating the clients if internals of outter change, just the outter.Get() method).
A similar approach would be to add methods that only return the field of the receiver struct:
func (i *inner1) Value() (string, bool) {
if i == nil {
return "", false
}
return i.value, true
}
func (i *inner2) Inner1() *inner1 {
if i == nil {
return nil
}
return i.value
}
func (o *outter) Inner2() *inner2 {
if o == nil {
return nil
}
return o.value
}
This approach requires clients to know internals of outter, but similarly it does not require any nil checks when using it:
o := &outter{
value: &inner2{
value: &inner1{
value: "I need this data",
},
},
}
fmt.Println(o.Inner2().Inner1().Value())
o.value.value = nil
fmt.Println(o.Inner2().Inner1().Value())
o.value = nil
fmt.Println(o.Inner2().Inner1().Value())
o = nil
fmt.Println(o.Inner2().Inner1().Value())
Output is the same. Try this one on the Go Playground.
I can use panic recovery method. This will solve my problem. But this seems hacky to me.
func printValue(o *outter) (string, bool) {
defer func() (string, bool) {
if r := recover(); r != nil {
return "", false
}
return "", false
}()
return o.value.value.value, true
}
There is no easy way. You may go recover way but it's not idiomatic IMO and you should check that you don't catch other unrelated errors.
I prefer a single if instead of multiple. I don't think of the code below ugly or verbose.
func printValue(o *outter) (string, bool) {
if o.value != nil and o.value.value != nil and o.value.value.value != nil {
return *o.value.value.value, true
}
return "", false
}
Using reflection
func getFieldByName(v interface{}, fields string, sep string) interface{} {
r := reflect.ValueOf(v)
s := strings.Split(fields, sep)
for _, field := range s {
r = reflect.Indirect(r)
if r.IsValid() {
r = reflect.Indirect(r).FieldByName(field)
} else {
return nil
}
}
if r.IsValid() {
return r.Interface()
}
return nil
}
Using Panic Recovery
type safeGetFn func() interface{}
func safeGet(f safeGetFn, d interface{}) (ret interface{}) {
defer func() interface{} {
if r := recover(); r != nil {
ret = d
}
return ret
}()
return f()
}
Example Reflection: https://play.golang.org/p/m0_zQqJm7MY
Example Panic Recovery: https://play.golang.org/p/PNPPBXCvHxJ

Asynchronous Testing With Stream Processing

I'm very new to Go, so I may be misunderstanding something foundational about Go's async/stream handling, but here goes...
I'm trying to write some tests using ginkgo on a function I wrote that processes streams.
The processing side reads in newline-delimited text from a File until it encounters a special delimiter line at which point it tries to parse the text as JSON. The code looks like this:
func ParseConfig(inStream *os.File) (Config, error){
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break;
}
header += line
}
// parse JSON here and return
}
My test looks something like this
Describe("ParseConfig()", func() {
It("should pass for a valid header", func(){
_, err := io.WriteString(stream, "{\"Key\": \"key\", \"File\": \"file\"}\n|||\n")
Expect(err).NotTo(HaveOccurred())
conf, err := parser.ParseConfig(stream)
Expect(err).NotTo(HaveOccurred())
Expect(conf.Key).To(Equal("key"))
})
})
Unfortunately, this yields a JSON parsing error, as it's trying to parse an empty string. I'm assuming that my problem is that I'm sending the string on the stream before I've told the ParseConfig() function to listen on that string for data? But I'm not entirely clear how I could refactor this to use proper go routines to first listen for data then send it.
Some of the potential solutions I saw were around the use of "channels" (with which I'm unfamiliar) but I was worried that this one need might not be worth a major refactor to introduce a whole new paradigm of concurrency.
Thanks!
Not sure if I understood correctly, but your ParseConfig should probably take an io.Reader instead of a *os.File. That way you can test it directly without worrying about concurrency.
file t_test.go:
package main
import (
"strings"
"testing"
"github.com/onsi/ginkgo"
"github.com/onsi/gomega"
)
var _ = ginkgo.Describe("ParseConfig()", func() {
ginkgo.It("should pass for a valid header", func() {
// really don't know what you were doing with your 'stream' variable
// This is a test, you should forge a test scenario and pass it to your config function
stream := strings.NewReader(`{"Key": "key", "File": "file"}` + "\n|||\n")
conf, err := ParseConfig(stream)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
gomega.Expect(conf.Key).To(gomega.Equal("key"))
})
})
func TestParseConfig(t *testing.T) {
ginkgo.RunSpecs(t, "Parse Config")
}
file main.go
package main
import (
"bufio"
"encoding/json"
"io"
"log"
"os"
)
type Config struct {
Key string
File string
}
func ParseConfig(inStream io.Reader) (*Config, error) {
var header string
var stdin = bufio.NewScanner(inStream)
for stdin.Scan() {
line := stdin.Text()
if line == "|||" {
break
}
header += line
}
c := &Config{}
// parse JSON here and return
if err := json.Unmarshal([]byte(header), c); err != nil {
return nil, err
}
return c, nil
}
func main() {
f, err := os.Open("config.json")
if err != nil {
log.Fatal(err)
}
ParseConfig(f)
}

Resources