Slice juggling in golang - pointers

To be short, here is a deal:
http://play.golang.org/p/ePiZcFfPZP
If I use commented lines, everything works, but there is no
any control on allocation sizes (cap), so the slices,
if I got it correct, realloc every time they exceed their limit
and moreover, they start with zero capacity.
Passing a reference of newSlice in setSlice() don't work too.
So, I need ideomatic, elegant, go-ish method to do the job.
Thanks in advance, at least for attention and your time.
UPD:
solution was to make SLICE and STASH *[]byte typed
and make assigns to them like:
var slicePtr *[]byte
tmp := make([]byte, 256)
slicePtr = &tmp // Tmp is needed because we can't take adress of make() rval.

For example,
package main
import "fmt"
var SLICE, STASH []byte
func init() {
SLICE = make([]byte, 0, 5)
}
func setSlice(slice []byte) {
STASH = SLICE
SLICE = slice
}
func restoreSlice() {
SLICE = STASH
}
func appendToSlice(parts ...byte) []byte {
SLICE = append(SLICE, parts...)
return SLICE
}
func main() {
appendToSlice('f', 'o', 'o')
fmt.Printf("Everything is fine: {'%s'}\n", SLICE)
newSlice := make([]byte, 0, 5)
setSlice(newSlice)
newSlice = appendToSlice('b', 'a', 'r')
fmt.Printf("Bar? No! {'%s'}\n", newSlice) // <- I need "bar" appear in newSlice.
fmt.Printf("Bar is here: {'%s'}\n", SLICE)
restoreSlice()
fmt.Printf("Back to origin. {'%s'}\n", SLICE)
}
Output:
Everything is fine: {'foo'}
Bar? No! {'bar'}
Bar is here: {'bar'}
Back to origin. {'foo'}
Like the Go append built-in function, your appendToSlice function needs to return the result of the append.
func appendToSlice(parts ...byte) []byte {
SLICE = append(SLICE, parts...)
return SLICE
}
and
newSlice = appendToSlice('b', 'a', 'r')
The Go Programming Language Specification
Appending to and copying slices
The built-in functions append and copy assist in common slice
operations. For both functions, the result is independent of whether
the memory referenced by the arguments overlaps.
The variadic function append appends zero or more values x to s of
type S, which must be a slice type, and returns the resulting slice,
also of type S.
If the capacity of s is not large enough to fit the additional values,
append allocates a new, sufficiently large underlying array that fits
both the existing slice elements and the additional values. Otherwise,
append re-uses the underlying array.
Example:
var b []byte
b = append(b, "bar"...) // append string contents; b == []byte{'b', 'a', 'r' }

Related

Appending to a slice with enough capacity using value receiver

can someone help me understand what happens here?
package main
import (
"fmt"
)
func appendString(slice []string, newString string) {
slice = append(slice, newString)
}
func main() {
slice := make([]string, 0, 1)
appendString(slice, "a")
fmt.Println(slice)
}
I know about the slice header and the need to use a pointer receiver. But here, as the underlying array has enough capacity I would expect append to work anyways (just adding the new value to the underlying array and the original [copied] header working as expected)
What is wrong with my assumptions?
Let's add a final print statement to see the result:
slice := make([]string, 0, 1)
fmt.Println(cap(slice))
appendString(slice, "a")
fmt.Println(slice)
And the output will be (try it on the Go Playground):
1
[]
Which is correct. One could expect the output to be:
1
[a]
The reason why this is not the case is because even though a new backing array will not be allocated, the slice header in the slice variable inside main() is not changed, it will still hold length = 0. Only the slice header stored in the slice local variable inside appendString() (the parameter) is changed, but this variable is independent from main's slice.
If you were to reslice main's slice, you will see that the backing array does contain the new string:
slice := make([]string, 0, 1)
fmt.Println(cap(slice))
appendString(slice, "a")
fmt.Println(slice)
slice = slice[:1]
fmt.Println(slice)
Now output will be (try it on the Go Playground):
1
[]
[a]
This is why the builtin append() has to return the new slice: because even if no new backing array is needed, the slice header (which contains the length) will have to be changed (increased) if more than 0 elements are appended.
This is why appendString() should also return the new slice:
func appendString(slice []string, newString string) []string {
slice = append(slice, newString)
return slice
}
Or short:
func appendString(slice []string, newString string) []string {
return append(slice, newString)
}
Which you have to reassign where you use it:
slice := make([]string, 0, 1)
fmt.Println(cap(slice))
slice = appendString(slice, "a")
fmt.Println(slice)
And then you get the expected outcome right away (try it on the Go Playground):
1
[a]

Cannot append to slice inside a function [duplicate]

This question already has an answer here:
Why does append() modify the provided slice? (See example)
(1 answer)
Closed 4 years ago.
I tried to add an element to my slice inside a function. I can change the element of the slice but cannot add a new element to it. Since slices act like reference why can't I change it?
Below is the code I have tried:
package main
import (
"fmt"
)
func main() {
a := []int{1, 2, 3}
change(a)
fmt.Println(a)
}
func change(a []int) {
a[0] = 4
a = append(a, 5)
}
Slice are pointers to underlying array. It is described in Golang:
Map and slice values behave like pointers: they are descriptors that
contain pointers to the underlying map or slice data. Copying a map or
slice value doesn't copy the data it points to. Copying an interface
value makes a copy of the thing stored in the interface value. If the
interface value holds a struct, copying the interface value makes a
copy of the struct. If the interface value holds a pointer, copying
the interface value makes a copy of the pointer, but again not the
data it points to.
you are passing a copy of the slice not the original slice. Return the value after appending to the slice and then assign it to the original slice as
package main
import (
"fmt"
)
func main() {
a := []int{1, 2, 3}
a = change(a)
fmt.Println(a)
}
func change(a []int) []int{
a = append(a, 5)
return a
}
Playground Example
Or you can pass a pointer to slice of int but it is not recommended since slice it self is a pointer to bootstrap array.
package main
import (
"fmt"
)
func main() {
a := []int{1, 2, 3}
change(&a)
fmt.Println(a)
}
func change(a *[]int){
*a = append(*a, 5)
}
Note: Everything in Golang is pass by value.
One thing to be considered is even if you are returning the updated slice and assigning to the same value, its original len and cap will change, which will lead to a new underlying array of different len. Try to print the length and cap before and after changing the slice to see the difference.
fmt.Println(len(a), cap(a))
The length is the number of elements referred to by the slice. The capacity is the number of elements in the underlying array (beginning at the element referred to by the slice pointer).
Since the underlying array will check you can check it using reflect and unsafe for fetching the underlying array which is going to be different if cap of a slice change after appending data which is your case.
package main
import (
"fmt"
"reflect"
"unsafe"
)
func main() {
a := []int{1, 2, 3}
hdr := (*reflect.SliceHeader)(unsafe.Pointer(&a))
data := *(*[3]int)(unsafe.Pointer(hdr.Data))
fmt.Println(data)
a = change(a)
hdr = (*reflect.SliceHeader)(unsafe.Pointer(&a))
newData := *(*[4]int)(unsafe.Pointer(hdr.Data))
fmt.Println(newData)
}
func change(a []int) []int {
a = append(a, 5)
return a
}
Playground Example
This is the best part of slices that you need to worry about its capacity when appending data more than its capacity, since it will point to a new array allocated in the memory of bigger length.

Why can't you append a pointer to an array?

package main
import "fmt"
type Circle struct {
x string
}
func main() {
circle := Circle{x: "blah"}
results := make([]*Circle, 1)
results = append(results, &circle)
fmt.Printf("Here: %s\n", results[0].x)
}
If I change the line results = append(results, &circle) to results[0] = &circle works fine. Couldn't find any reason as to why that would be.
You can, it's just that you are appending to the slice, which means that the element you add is in results[1], and results[0] is a nil pointer (the default value for a pointer).
You could do results := make([]*Circle, 0, 1) to give it a capacity of 1 but a length of zero, or you could do results := []*Circle{} (most compact), or you could simply keep the version where you assign to results[0], since it works just fine.
You can, but you are using make with length 1, so the [0] index is already set (to a nil Circle...)
Changing your results[0].x to results[1].x works, but without using make works as expected:
package main
import "fmt"
type Circle struct {
x string
}
func main() {
circle := Circle{x: "blah"}
results := []*Circle{} // initialize empty slice of Circle pointers
results = append(results, &circle)
fmt.Printf("Here: %s\n", results[0].x)
}
Run in playground: https://play.golang.org/p/7vMOfAXvgI

Go: Passing pointers of an array to gob without copying?

I have a very, very large array (not slice) of maps that I am then trying to encode. I really need to avoid making a copy of the array but I can't figure out how to do this.
So I far I have this:
func doSomething() {
var mygiantvar [5]map[string]Searcher
mygiantvar = Load()
Save(`file.gob.gz`, &mygiantvar)
}
func Save(filename string, variable *[5]map[string]Searcher) error {
// Open file for writing
fi, err := os.Create(filename)
if err !=nil {
return err
}
defer fi.Close()
// Attach gzip writer
fz := gzip.NewWriter(fi)
defer fz.Close()
// Push from the gob encoder
encoder := gob.NewEncoder(fz)
err = encoder.Encode(*variable)
if err !=nil {
return err
}
return nil
}
From my understanding that will pass a pointer of mygiantvar to Save, which saves the first copy. But then the entire array will surely be copied into encoder.Encode which will then copy it around many more functions, right?
This mygiantvar variable will be something like 10GB in size. So it must avoid being copied ever.
But then again perhaps only the actual array [5] part is copied but the maps inside of this are pointers inside an array, so the array of pointers to maps would be copied instead of the maps themselves? I have no idea about this - it's all very confusing.
Any ideas?
Note that Encoder.Encode will pass around an interface{}.
func (enc *Encoder) Encode(v interface{}) error {
That means a kind of a pointer to whatever you will be passing to it, as I described in "what is the meaning of interface{} in golang?"
(see also "Why can't I assign a *Struct to an *Interface?")
An interface value isn't the value of the concrete struct (as it has a variable size, this wouldn't be possible), but it's a kind of pointer (to be more precise a pointer to the struct and a pointer to the type)
That means it won't copy the full content of your map (or here of your array).
Since array is a value, you could slice it to avoid any copy during the call to Encode():
err = encoder.Encode(*variable[:])
See "Go Slices: usage and internals"
This is also the syntax to create a slice given an array:
x := [3]string{"Лайка", "Белка", "Стрелка"}
s := x[:] // a slice referencing the storage of x
If that doesn't work, you can keep *variable (here an array: [5]map[string]Searcher), as map types are reference types, like pointers or slices: the copy won't be huge.
See "Go maps in action".
While the array will be copied when passed to interface{}, the map content won't be copied.
See this play.golang.org example:
package main
import "fmt"
func main() {
var a [1]map[string]int
a[0] = make(map[string]int)
a[0]["test"] = 0
modify(a)
fmt.Println(a)
}
func modify(arr interface{}) {
a := arr.([1]map[string]int)
a[0]["test"] = -1
}
Output:
[map[test:-1]]

How can I merge two maps in go?

I have a recursive function that creates objects representing file paths (the keys are paths and the values are info about the file). It's recursive as it's only meant to handle files, so if a directory is encountered, the function is recursively called on the directory.
All that being said, I'd like to do the equivalent of a set union on two maps (i.e. the "main" map updated with the values from the recursive call). Is there an idiomatic way to do this aside from iterating over one map and assigning each key, value in it to the same thing in the other map?
That is: given a,b are of type map [string] *SomeObject, and a and b are eventually populated, is there any way to update a with all the values in b?
There is no built in way, nor any method in the standard packages to do such a merge.
The idomatic way is to simply iterate:
for k, v := range b {
a[k] = v
}
Since Go 1.18, you can simply use the Copy function from the golang.org/x/exp/maps package:
package main
import (
"fmt"
"golang.org/x/exp/maps"
)
func main() {
src := map[string]int{
"one": 1,
"two": 2,
}
dst := map[string]int{
"two": 2,
"three": 3,
}
maps.Copy(dst, src)
fmt.Println("src:", src)
fmt.Println("dst:", dst)
}
(Playground)
Output:
src: map[one:1 two:2]
dst: map[one:1 three:3 two:2]
One caveat of this approach is that, in Go versions 1.18.x to 1.19.x, your map's key type must be concrete, i.e. not an interface type. For instance, the compiler won't allow you to pass values of type map[io.Reader]int to the Copy function:
package main
import (
"fmt"
"io"
"golang.org/x/exp/maps"
)
func main() {
var src, dst map[io.Reader]int
maps.Copy(dst, src)
fmt.Println("src:", src)
fmt.Println("dst:", dst)
}
(Playground)
Compiler output:
go: finding module for package golang.org/x/exp/maps
go: downloading golang.org/x/exp v0.0.0-20220328175248-053ad81199eb
./prog.go:12:11: io.Reader does not implement comparable
Go build failed.
This limitation was lifted in Go 1.20.
(Playground)
Starting at go 1.18, thanks to the release of the Generics feature, there are now generic functions that union maps!
You can use a package like https://github.com/samber/lo in order to do so.
Note that the key can be of any "comparable" type, while the value can be of any type.
Example:
package main
import (
"fmt"
"github.com/samber/lo"
)
func main() {
map1 := map[string]interface{}{"k1": "v1", "k2": 2}
map2 := map[string]interface{}{"k2": "v2new", "k3": true}
map1 = lo.Assign(map1, map2)
fmt.Printf("%v", map1)
}
The result is:
map[k1:v1 k2:v2new k3:true]
If you have a couple of nested maps, left and right, then this function will recursively add the items from right into left. If the key is already in left then we recurse deeper into the structure and attempt only add keys to left (e.g. never replace them).
type m = map[string]interface{}
// Given two maps, recursively merge right into left, NEVER replacing any key that already exists in left
func mergeKeys(left, right m) m {
for key, rightVal := range right {
if leftVal, present := left[key]; present {
//then we don't want to replace it - recurse
left[key] = mergeKeys(leftVal.(m), rightVal.(m))
} else {
// key not in left so we can just shove it in
left[key] = rightVal
}
}
return left
}
NOTE: I do not handle the case in which the value is not itself a map[string]interface{}. So if you have left["x"] = 1 and right["x"] = 2 then the above code will panic when attempting leftVal.(m).
Here is another option,
in case you are trying to limit the number of third-party dependencies such github.com/samber/lo, OR
you are not comfortable with the experimental nature of golang.org/x/exp (read the warning), OR
you would rather the convenience of an append()-like API instead of exp.Copy() from golang.org/x/exp (append accepts any number of lists, whereas Copy() accepts only 2).
However it requires Go 1.18+ as it uses go generics.
Save the following in one of your modules/packages:
func MergeMaps[M ~map[K]V, K comparable, V any](src ...M) M {
merged := make(M)
for _, m := range src {
for k, v := range m {
merged[k] = v
}
}
return merged
}
Then you can use it very similarly to append():
func main() {
mergedMaps := MergeMaps(
map[string]int{"a": 1, "b": 2},
map[string]int{"b": 3, "c": 4},
map[string]int{"c": 3, "d": 4},
)
fmt.Println(mergedMaps)
}
Go is limited by what type of map it is. I'd suspect that there isn't built in functions because of the infinite number of type declarations that could exist for a map. So you have to build your own Merge functions depending on what type of map you are using:
func MergeJSONMaps(maps ...map[string]interface{}) (result map[string]interface{}) {
result = make(map[string]interface{})
for _, m := range maps {
for k, v := range m {
result[k] = v
}
}
return result
}

Resources