Is it possible to convert a pointer to certain value to a slice?
For example, I want to read single byte from io.Reader into uint8 variable. io.Reader.Read accepts a slice as its argument, so I cannot simply provide it a pointer to my variable as I'd do in C.
I think that creating a slice of length 1, capacity 1 from a pointer is safe operation. Obviously, it should be the same as creating a slice from an array of length 1, which is allowed operation. Is there an easy way to do this with plain variable? Or maybe I do not understand something and there are reasons why this is prohibited?
A slice is not only a pointer, like an array in C. It also contains the length and capacity of the data, like this:
struct {
ptr *uint8
len int
cap int
}
So, yes, you will need to create a slice. Simplest way to create a slice of the var a uint8 would be []uint8{a}
a := uint8(42)
fmt.Printf("%#v\n", []uint8{a})
(But after rereading your question, this is not a solution as all)
But if you wish to create the slice from the variable, pointing to the same space of memory, you could use the unsafe package. This is most likely to be discouraged.
fmt.Printf("%#v\n", (*[1]uint8)(unsafe.Pointer(&a))[:] )
Instead of (over)complicating this trivial task, why not to use the simple solution? I.e. pass .Read a length-1 slice and then assign its zeroth element to your variable.
I found a way to overcome my case when I want to supply a variable to io.Reader. Go standard library is wonderful!
import (
"io"
"encoding/binary"
)
...
var x uint8
binary.Read(reader, LittleEndian, &x)
As a side effect this works for any basic type and even for some non-basic.
Related
It seems that both ways to create a new object pointer with all "0" member values, both returns a pointer:
type T struct{}
...
t1:=&T{}
t2:=new(T)
So what is the core difference between t1 and t2, or is there anything that "new" can do while &T{} cannot, or vice versa?
[…] is there anything that "new" can do while &T{} cannot, or vice versa?
I can think of three differences:
The "composite literal" syntax (the T{} part of &T{}) only works for "structs, arrays, slices, and maps" [link], whereas the new function works for any type [link].
For a struct or array type, the new function always generates zero values for its elements, whereas the composite literal syntax lets you initialize some of the elements to non-zero values if you like.
For a slice or map type, the new function always returns a pointer to nil, whereas the composite literal syntax always returns an initialized slice or map. (For maps this is very significant, because you can't add elements to nil.) Furthermore, the composite literal syntax can even create a non-empty slice or map.
(The second and third bullet-points are actually two aspects of the same thing — that the new function always creates zero values — but I list them separately because the implications are a bit different for the different types.)
For structs and other composites, both are same.
t1:=&T{}
t2:=new(T)
//Both are same
You cannot return the address of un-named variable initialised to zero value of other basic types like int without using new. You would need to create a named variable and then take its address.
func newInt() *int {
return new(int)
}
func newInt() *int {
// return &int{} --> invalid
var dummy int
return &dummy
}
See ruakh's answer. I want to point out some of the internal implementation details, though. You should not make use of them in production code, but they help illuminate what really happens behind the scenes, in the Go runtime.
Essentially, a slice is represented by three values. The reflect package exports a type, SliceHeader:
SliceHeader is the runtime representation of a slice. It cannot be used safely or portably and its representation may change in a later release. Moreover, the Data field is not sufficient to guarantee the data it references will not be garbage collected, so programs must keep a separate, correctly typed pointer to the underlying data.
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
If we use this to inspect a variable of type []T (for any type T), we can see the three parts: the pointer to the underlying array, the length, and the capacity. Internally, a slice value v always has all three of these parts. There's a general condition that I think should hold, and if you don't use unsafe to break it, it seems by inspection that it will hold (based on limited testing anyway):
either the Data field is not zero (in which case Len and Cap can but need not be nonzero), or
the Data field is zero (in which case the Len and Cap should both be zero).
That slice value v is nil if the Data field is zero.
By using the unsafe package, we can break it deliberately (and then put it all back—and hopefully nothing goes wrong while we have it broken) and thus inspect the pieces. When this code on the Go Playground is run (there's a copy below as well), it prints:
via &literal: base of array is 0x1e52bc; len is 0; cap is 0.
Go calls this non-nil.
via new: base of array is 0x0; len is 0; cap is 0.
Go calls this nil even though we clobbered len() and cap()
Making it non-nil by unsafe hackery, we get [42] (with cap=1).
after setting *p1=nil: base of array is 0x0; len is 0; cap is 0.
Go calls this nil even though we clobbered len() and cap()
Making it non-nil by unsafe hackery, we get [42] (with cap=1).
The code itself is a bit long so I have left it to the end (or use the above link to the Playground). But it shows that the actual p == nil test in the source compiles to just an inspection of the Data field.
When you do:
p2 := new([]int)
the new function actually allocates only the slice header. It sets all three parts to zero and returns the pointer to the resulting header. So *p2 has three zero fields in it, which makes it a correct nil value.
On the other hand, when you do:
p1 := &[]int{}
the Go compiler builds an empty array (of size zero, holding zero ints) and then builds a slice header: the pointer part points to the empty array, and the length and capacity are set to zero. Then p1 points to this header, with the non-nil Data field. A later assignment, *p1 = nil, writes zeros into all three fields.
Let me repeat this with boldface: these are not promised by the language specification, they're just the actual implementation in action.
Maps work very similarly. A map variable is actually a pointer to a map header. The details of map headers are even less accessible than those of slice headers: there is no reflect type for them. The actual implementation is viewable here under type hmap (note that it is not exported).
What this means is that m2 := new(map[T1]T2) really only allocates one pointer, and set that pointer itself to nil. There is no actual map! The new function returns the nil pointer, and m2 is then nil. Likewise var m1 map[T1]T2 just sets a simple pointer value in m1 to nil. But var m3 map[T1]T2{} allocates an actual hmap structure, fills it in, and makes m3 point to it. We can once again peek behind the curtain on the Go Playground, with code that is not guaranteed to work tomorrow, to see this in effect.
As someone writing Go programs, you don't need to know any of this. But if you have worked with lower-level languages (assembly and C for instance), these explain a lot. In particular, these explain why you cannot insert into a nil map: the map variable itself holds a pointer value, and until the map variable itself has a non-nil pointer to a (possibly empty) map-header, there is no way to do the insertion. An insertion could allocate a new map and insert the data, but the map variable wouldn't point to the correct hmap header object.
(The language authors could have made this work by using a second level of indirection: a map variable could be a pointer pointing to the variable that points to the map header. Or they could have made map variables always point to a header, and made new actually allocate a header, the way make does; then there would never be a nil map. But they didn't do either of these, and we get what we get, which is fine: you just need to know to initialize the map.)
Here's the slice inspector. (Use the playground link to view the map inspector: given that I had to copy hmap's definition out of the runtime, I expect it to be particularly fragile and not worth showing. The slice header's structure seems far less likely to change over time.)
package main
import (
"fmt"
"reflect"
"unsafe"
)
func main() {
p1 := &[]int{}
p2 := new([]int)
show("via &literal", *p1)
show("\nvia new", *p2)
*p1 = nil
show("\nafter setting *p1=nil", *p1)
}
// This demonstrates that given a slice (p), the test
// if p == nil
// is really a test on p.Data. If it's zero (nil),
// the slice as a whole is nil. If it's nonzero, the
// slice as a whole is non-nil.
func show(what string, p []int) {
pp := unsafe.Pointer(&p)
sh := (*reflect.SliceHeader)(pp)
fmt.Printf("%s: base of array is %#x; len is %d; cap is %d.\n",
what, sh.Data, sh.Len, sh.Cap)
olen, ocap := len(p), cap(p)
sh.Len, sh.Cap = 1, 1 // evil
if p == nil {
fmt.Println(" Go calls this nil even though we clobbered len() and cap()")
answer := 42
sh.Data = uintptr(unsafe.Pointer(&answer))
fmt.Printf(" Making it non-nil by unsafe hackery, we get %v (with cap=%d).\n",
p, cap(p))
sh.Data = 0 // restore nil-ness
} else {
fmt.Println("Go calls this non-nil.")
}
sh.Len, sh.Cap = olen, ocap // undo evil
}
The following code serves to create a counter for each pair of float64.
Because the keys of a map cannot be a slice, I have to use arrays as keys, which forces me to to define a dimension with a constant.
counter := make( map[ [2]float64 ] int )
for _, comb := range combinations{ //combinations is a [n][2]float64
for _, row := range data{
counter[ [...]float64{ row[comb[0]] , row[comb[1]] } ]++
}
}
Having said that, is there a way to make this map dependent on the length of the keys (dependent on the dimensions of combinations?
I tried using a struct as key, but as far as I remember (I might be wrong), it was a bit slower... For my purposes (to apply this for all combinations ~n!) this is not the ideal solution.
Right now I'm only considering combinations of size 2 and 3, and I had to split this in two separate functions, which makes my code very verbose and harder to maintain.
Can you find a way to simplify this, so I can scale it to more dimensions?
Thanks for any input
Why not use the pointer to a slice as key?
You could create a slice with make with a big enough capacity, while you do not surpass it's capacity, the pointer will remain the same.
Take a look here https://play.golang.org/p/333tRMpBLv , it exemplifies my suggestion. see that while len < cap the pointer of the slice is not changed, append only creates a new slice when len exceeds cap.
When I run the below code, I get the compiler error saying that indexing is not supported.
txs := make([]string, 2)
txs[0] = "A"
p := &txs
fmt.Println(p[0])
I'm trying to understand why indexing on the slice pointer is not supported. I can make a copy of the pointer variable to value variable and then do indexing, but I'm curious to understand why the language is not supporting the indexing on slice pointer; it'd be so much convenient to do so. or is there a way to do it that i'm not aware? please let me know your thoughts.
Write (*p) to dereference the pointer p:
package main
import (
"fmt"
)
func main() {
txs := make([]string, 2)
txs[0] = "A"
p := &txs
fmt.Println((*p)[0])
}
Playground: https://play.golang.org/p/6Ex-3jtmw44
Output:
A
There's an abstraction happening there and the language designer chose not to apply it to the pointer. To give some practical reason, this is likely due to the fact that the pointer doesn't point to the beginning of an array (like the block of memory. If you're familiar with indexing this is generally done with something like startingAddress + index * sizeof(dataType)). So when you have the value type, it's already providing an abstraction to hide the extra layer of indirection that occurs. I assume the language authors didn't think it made sense to do this when you have a pointer to the slice object, given that points off to the actual memory that would be a pretty misleading. It already causes some confusion as is, but for a lot of developers, they probably will never realize this abstraction exists at all (like in most cases there is no noticeable difference in syntax when operating on a slice vs and array).
This link: http://research.swtch.com/godata
It says (third paragraph of section Slices):
Because slices are multiword structures, not pointers, the slicing
operation does not need to allocate memory, not even for the slice
header, which can usually be kept on the stack. This representation
makes slices about as cheap to use as passing around explicit pointer
and length pairs in C. Go originally represented a slice as a pointer
to the structure shown above, but doing so meant that every slice
operation allocated a new memory object. Even with a fast allocator,
that creates a lot of unnecessary work for the garbage collector, and
we found that, as was the case with strings above, programs avoided
slicing operations in favor of passing explicit indices. Removing the
indirection and the allocation made slices cheap enough to avoid
passing explicit indices in most cases.
What...? Why does it not allocate any memory? If it is a multiword structure or a pointer? Does it not need to allocate memory? Then it mentions that it was originally a pointer to that slice structure, and it needed to allocate memory for a new object. Why does it not need to do that now? Very confused
To expand on Pravin Mishra's answer:
the slicing operation does not need to allocate memory.
"Slicing operation" refers to things like s1[x:y] and not slice initialization or make([]int, x). For example:
var s1 = []int{0, 1, 2, 3, 4, 5} // <<- allocates (or put on stack)
s2 := s1[1:3] // <<- does not (normally) allocate
That is, the second line is similar to:
type SliceHeader struct {
Data uintptr
Len int
Cap int
}
…
example := SliceHeader{&s1[1], 2, 5}
Usually local variables like example get put onto the stack. It's just like if this was done instead of using a struct:
var exampleData uintptr
var exampleLen, exampleCap int
Those example* variables go onto the stack.
Only if the code does return &example or otherFunc(&example) or otherwise allows a pointer to this to escape will the compiler be forced to allocate the struct (or slice header) on the heap.
Then it mentions that it was originally a pointer to that slice structure, and it needed to allocate memory for a new object. Why does it not need to do that now?
Imagine that instead of the above you did:
example2 := &SliceHeader{…same…}
// or
example3 := new(SliceHeader)
example3.Data = …
example3.Len = …
example3.Cap = …
i.e. the type is *SliceHeader rather than SliceHeader.
This is effectively what slices used to be (pre Go 1.0) according to what you mention.
It also used to be that both example2 and example3 would have to be allocated on the heap. That is the "memory for a new object" being refered to. I think that now escape analysis will try and put both of these onto the stack as long as the pointer(s) are kept local to the function so it's not as big of an issue anymore. Either way though, avoiding one level of indirection is good, it's almost always faster to copy three ints compared to copying a pointer and dereferencing it repeatedly.
Every data type allocates memory when it's initialized. In blog, he clearly mention
the slicing operation does not need to allocate memory.
And he is right. Now see, how slice works in golang.
Slices hold references to an underlying array, and if you assign one
slice to another, both refer to the same array. If a function takes a
slice argument, changes it makes to the elements of the slice will be
visible to the caller, analogous to passing a pointer to the
underlying array.
Edit: Jeremy Wall helped me realize I had asked a question more specific than I intended; here's a better version.
Say I want to represent a table associating of values of some type B to sequences of values of some type A for which equality is defined. What is the best way to do that in Go?
Obviously for the table I'd want to use a Go map, but what can I use for the sequences of values of type A? Slices cannot be used as keys for maps in Go; arrays can, but the length of an array is a part of it's type and I'm interested in being able to use sequences of length determined at runtime. I could (1) use arrays of A declaring a maximum length for them or (2) use slices of A, serialize them to strings for use as keys (this technique is familiar to Awk and Lua programmers...). Is there a better work around for this "feature" of Go than the ones I've described?
As pointed out by Jeremy Wall in answer to my original version of the question, where I had A = int, option (2) is pretty good for integers, since you can use slices of runes for which conversion to string is just a cast.
Will a sequence of rune instead of integers work for you? runes are uint32 and the conversion to a string is just a cast:
package main
import "fmt"
type myKey struct {
seq []int
}
func main() {
m := make(map[string]string)
key := []rune{1, 2}
m[string(key)] = "foo"
fmt.Print("lookup: ", m[string(key)])
}
You can play with this code here: http://play.golang.org/p/Kct1dum8A0