Related
I'm trying to use the GetRawInputDeviceList function in Go and I keep getting the following error:
The parameter is incorrect.
Per the official documentation: the first parameter needs to be an array of RAWINPUTDEVICELIST structures for the devices attached to the system. I don't quite understand what combination of unsafe.Pointer, pointer arithmetic(?), and other things I need to do in order to get this to work correctly.
I found this Medium article that offers some guidance, but it's not directly applicable to my use case. I don't have enough experience working with pointers and manual memory management to apply it to my problem. I don't know how to translate this C++ example to Go, and I got so desperate that I tried to convert a working VBA solution to Go with no success.
I have two questions regarding this matter:
How do I convert an array of structs in Go to the appropriate type expected for a Windows API call?
How do I convert the result of the Windows API call back to an array of structs with populated data?
Environment
Here's my system/language details:
macOS Mojave v10.14.6
Go v1.10.7 (required to run executables on Windows XP)
I'm targeting Windows XP, so I run the following command to compile it:
env GOOS=windows GOARCH=386 go1.10.7 build -o example.exe example.go
Code
Here's the code I'm trying to get working. I'm not doing anything with devices yet, but the goal would be to use the handle (DeviceHandle from rawInputDeviceList) to get information about the input device.
package main
import (
"fmt"
"syscall"
"unsafe"
)
// RAWINPUTDEVICELIST structure
type rawInputDeviceList struct {
DeviceHandle uintptr
Type uint32
}
var (
user32 = syscall.NewLazyDLL("user32.dll")
getRawInputDeviceListProc = user32.NewProc("GetRawInputDeviceList")
)
func main() {
dl := rawInputDeviceList{}
size := uint32(unsafe.Sizeof(dl))
// First I determine how many input devices are on the system, which
// gets assigned to `devCount`
var devCount uint32
_ = getRawInputDeviceList(nil, &devCount, size)
if devCount > 0 {
size = size * devCount
devices := make([]rawInputDeviceList, size) // <- This is definitely wrong
for i := 0; i < int(devCount); i++ {
devices[i] = rawInputDeviceList{}
}
// Here is where I get the "The parameter is incorrect." error:
err := getRawInputDeviceList(&devices, &devCount, size)
if err != nil {
fmt.Printf("Error: %v", err)
}
}
}
// Enumerates the raw input devices attached to the system.
func getRawInputDeviceList(
rawInputDeviceList *[]rawInputDeviceList, // <- This is probably wrong
numDevices *uint32,
size uint32,
) error {
_, _, err := getRawInputDeviceListProc.Call(
uintptr(unsafe.Pointer(rawInputDeviceList)),
uintptr(unsafe.Pointer(numDevices)),
uintptr(size))
if err != syscall.Errno(0) {
return err
}
return nil
}
First, the ERROR_INVALID_PARAMETER error is cause by the last parameter: cbSize, According to the document, it should always be set to size of RAWINPUTDEVICELIST.
Then you will pass the compiler but still get the runtime error. because you have passed a pointer of array.
The following code works for me:
package main
import (
"fmt"
"syscall"
"unsafe"
)
// RAWINPUTDEVICELIST structure
type rawInputDeviceList struct {
DeviceHandle uintptr
Type uint32
}
var (
user32 = syscall.NewLazyDLL("user32.dll")
getRawInputDeviceListProc = user32.NewProc("GetRawInputDeviceList")
)
func main() {
dl := rawInputDeviceList{}
size := uint32(unsafe.Sizeof(dl))
// First I determine how many input devices are on the system, which
// gets assigned to `devCount`
var devCount uint32
_ = getRawInputDeviceList(nil, &devCount, size)
if devCount > 0 {
devices := make([]rawInputDeviceList, size * devCount) // <- This is definitely wrong
for i := 0; i < int(devCount); i++ {
devices[i] = rawInputDeviceList{}
}
// Here is where I get the "The parameter is incorrect." error:
err := getRawInputDeviceList(&devices[0], &devCount, size)
if err != nil {
fmt.Printf("Error: %v", err)
}
for i := 0; i < int(devCount); i++ {
fmt.Printf("Type: %v", devices[i].Type)
}
}
}
// Enumerates the raw input devices attached to the system.
func getRawInputDeviceList(
rawInputDeviceList *rawInputDeviceList, // <- This is probably wrong
numDevices *uint32,
size uint32,
) error {
_, _, err := getRawInputDeviceListProc.Call(
uintptr(unsafe.Pointer(rawInputDeviceList)),
uintptr(unsafe.Pointer(numDevices)),
uintptr(size))
if err != syscall.Errno(0) {
return err
}
return nil
}
I noticed, quite by accident, that I can successfully pass both a pointer to a struct, and a pointer to a pointer to a struct to json.Unmarshal(), and both work just fine:
package main
import (
"testing"
"encoding/json"
)
type Person struct {
Name string
Age int
}
func TestMarshaling(t *testing.T) {
foo := &Person{Name: "bob", Age: 23}
// marshal it to bytes
b, err := json.Marshal(foo)
if err != nil {
t.Error(err)
}
bar := &Person{} // pointer to new, empty struct
err = json.Unmarshal(b, bar) // unmarshal to bar, which is a *Person
if err != nil {
t.Error(err)
}
testBob(t, bar) // ok
bar = &Person{} // pointer to new, empty struct
err = json.Unmarshal(b, &bar) // wait a minute, passing in a **Person, yet it still works?
if err != nil {
t.Error(err)
}
testBob(t, bar) // ok
}
func testBob(t *testing.T, person *Person) {
if person.Name != "bob" || person.Age != 23 {
t.Error("not equal")
}
}
I was really surprised that the second one (unmarshal to **Person) worked.
What's going on in json.Unmarshal()? Is it dereferencing the pointers until it finds a struct?
The documentation offers:
To unmarshal JSON into a pointer, Unmarshal first handles the case of
the JSON being the JSON literal null. In that case, Unmarshal sets
the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the
value pointed at by the pointer
It seems to be doing a bit more than that. What's really going on?
Fleshing out my question more: how does it know to automatically dereference my pointer to a pointer? The documentation says it will unmarshal "into the value pointed at by the pointer". Since the value of my pointer is in fact another pointer, and has no Name/Age fields, I expected it to stop there.
To be clear: I'm not saying there's a bug or misfeature in Unmarshal(); I'm trying to satisfy my astonishment that it works at all when given a ptr-to-ptr, and avoid any potential pitfalls in my use of it.
The json package has no reason to "stop at a pointer", since a pointer means nothing in json. It has to keep walking the tree in order to find a value to write. Since the json package is going to allow unmarshaling the same value into Type or *Type, it stands to reason that it should be able to unmarshal that into **Type, which is also a valid type in Go.
For a example, if Person were defined using pointers to differentiate between nil and zero values, and you were unmarshaling into a slice of []*Person, the json package needs to follow those pointers, and allocate values if necessary. The same applies if a field in Person were defined as a **string.
type Person struct {
Name **string
Age *int
}
type People []*Person
http://play.golang.org/p/vLq0nJPG5M
The json.Unmarshal implementation takes multiple indirection into account. Check the source here, in particular the decodeState.indirect method:
// indirect walks down v allocating pointers as needed,
// until it gets to a non-pointer.
// if it encounters an Unmarshaler, indirect stops and returns that.
// if decodingNull is true, indirect stops at the last pointer so it can be set to nil.
func (d *decodeState) indirect(v reflect.Value, decodingNull bool) (Unmarshaler, encoding.TextUnmarshaler, reflect.Value) {
// If v is a named type and is addressable,
// start with its address, so that if the type has pointer methods,
// we find them.
if v.Kind() != reflect.Ptr && v.Type().Name() != "" && v.CanAddr() {
v = v.Addr()
}
for {
if v.Kind() == reflect.Interface && !v.IsNil() {
e := v.Elem()
if e.Kind() == reflect.Ptr && !e.IsNil() && (!decodingNull || e.Elem().Kind() == reflect.Ptr) {
v = e
continue
}
}
if v.Kind() != reflect.Ptr {
break
}
//and so on
}
return nil, nil, v
The same method is called when unmarshaling arrays:
func (d *decodeState) array(v reflect.Value) {
u, ut, pv := d.indirect(v, false)
//...
That would have me believe that go can handle double indirection just fine. If nothing else, the json package source is a great example of what the reflect package is all about.
In short, values are checked, if the decoder is dealing with pointers, it will use reflection to work out how many levels of indirection there are, and determine what type the target has/is. The place to start from in the decode source is this: func (d *decodeState) unmarshal(v interface{}) (err error) {, from that point on, it's pretty self-explanatory.
As other answers have said, pointers are followed.
A little weird that this errors (nil pointer), but makes sense when you think about it.
package main
import (
"encoding/json"
"fmt"
"log"
)
type MyStruct struct {
A string `json:"a"`
}
func main() {
data := []byte(`{"a":"foo"}`)
var a *MyStruct
err := json.Unmarshal(data, a) // nil ptr
if err != nil {
log.Fatal(err)
}
fmt.Println(a)
}
But this doesn't error (pointer to nil pointer).
package main
import (
"encoding/json"
"fmt"
"log"
)
type MyStruct struct {
A string `json:"a"`
}
func main() {
data := []byte(`{"a":"foo"}`)
var a *MyStruct
err := json.Unmarshal(data, &a) // **MyStruct, ptr to nil ptr
if err != nil {
log.Fatal(err)
}
fmt.Println(a)
}
https://play.golang.org/p/eI8jqWZOmGW
I started picking Go in the past couple of days, relying mostly on the language specification and package documentation, however I have problem deciphering the correct usage of net.LookupNS.
Since it's a pointer type, returning an array of memory addresses of NS server values, I want to access the actual values / dereference the array.
The Program:
package main
import "fmt"
import "net"
import "os"
var host string
func args() {
if len(os.Args) != 2 {
fmt.Println("You need to enter a host!")
} else {
host = os.Args[1]
}
if host == "" {
os.Exit(0)
}
}
func nslookup() []*net.NS {
nserv, err := net.LookupNS(host)
if err != nil {
fmt.Println("Error occured during NS lookup", err)
}
return *&nserv
}
func main() {
args()
fmt.Println("Nameserver information:", host)
fmt.Println(" NS records:", nslookup())
}
Given e.g. google.com, it displays the following:
Nameserver information: google.com
NS records: [0xc2100376f0 0xc210037700 0xc210037710 0xc210037720]
Instead of the memory address locations, I would like to see the dereferenced values, e.g:
NS records: ["ns1.google.com", "ns2.google.com", "ns3.google.com", "ns4.google.com"]
Now obviously, I would prefer them as an array/slice of strings, but the problem is that the only way I can get an actual nameserver out is as follows:
func nslookup() *net.NS {
// The rest of the function
return *&nserv[0] // This returns the first nameserver
The above returns the following:
Nameserver information: google.com
NS records: &{ns1.google.com.}
While this at least returns the actual value instead of a memory address, it requires indexing, which isn't very flexible and it's not formatted in a very user-friendly format.
Also, direct conversion of the []*net.NS struct to string is not possible.
The Problem:
How do I get an array of nameservers, instead of memory addresses out, preferably as an array/slice of strings?
Ok few problems :
Why are you returning *&nserv? Go is NOT C, please stop everything you're doing and read Effective Go.
Your nslookup function returns a slice of *net.NS, that's a slice of pointers, so fmt.Println is printing the right thing, if you want more details you could use fmt.Printf with %#vor %#q modifier to see how the data actually looks.
Example:
package main
import "fmt"
import "net"
import "os"
var host string
func nslookupString(nserv []*net.NS) (hosts []string) {
hosts = make([]string, len(nserv))
for i, host := range nserv {
hosts[i] = host.Host
}
return
}
func nslookupNS(host string) []*net.NS {
nserv, err := net.LookupNS(host)
if err != nil {
fmt.Println("Error occured during NS lookup", err)
}
return nserv
}
func init() { //initilizing global arguments is usually done in init()
if len(os.Args) == 2 {
host = os.Args[1]
}
}
func main() {
if host == "" {
fmt.Println("You need to enter a host!")
os.Exit(1)
}
fmt.Println("Nameserver information:", host)
ns := nslookupNS(host)
fmt.Printf(" NS records String: %#q\n", nslookupString(ns))
fmt.Printf(" NS records net.NS: %q\n", ns)
for _, h := range ns {
fmt.Printf("%#v\n", h)
}
}
I want to convert a struct to map in Golang. It would also be nice if I could use the JSON tags as keys in the created map (otherwise defaulting to field name).
Edit Dec 14, 2020
Since structs repo was archived, you can use mapstructure instead.
Edit TL;DR version, Jun 15, 2015
If you want the fast solution for converting a structure to map, see the accepted answer, upvote it and use that package.
Happy coding! :)
Original Post
So far I have this function, I am using the reflect package but I don't understand well how to use the package, please bear with me.
func ConvertToMap(model interface{}) bson.M {
ret := bson.M{}
modelReflect := reflect.ValueOf(model)
if modelReflect.Kind() == reflect.Ptr {
modelReflect = modelReflect.Elem()
}
modelRefType := modelReflect.Type()
fieldsCount := modelReflect.NumField()
var fieldData interface{}
for i := 0; i < fieldsCount; i++ {
field := modelReflect.Field(i)
switch field.Kind() {
case reflect.Struct:
fallthrough
case reflect.Ptr:
fieldData = ConvertToMap(field.Interface())
default:
fieldData = field.Interface()
}
ret[modelRefType.Field(i).Name] = fieldData
}
return ret
}
Also I looked at JSON package source code, because it should contain my needed implementation (or parts of it) but don't understand too much.
I also had need for something like this. I was using an internal package which was converting a struct to a map. I decided to open source it with other struct based high level functions. Have a look:
https://github.com/fatih/structs
It has support for:
Convert struct to a map
Extract the fields of a struct to a []string
Extract the values of a struct to a []values
Check if a struct is initialized or not
Check if a passed interface is a struct or a pointer to struct
You can see some examples here: http://godoc.org/github.com/fatih/structs#pkg-examples
For example converting a struct to a map is a simple:
type Server struct {
Name string
ID int32
Enabled bool
}
s := &Server{
Name: "gopher",
ID: 123456,
Enabled: true,
}
// => {"Name":"gopher", "ID":123456, "Enabled":true}
m := structs.Map(s)
The structs package has support for anonymous (embedded) fields and nested structs. The package provides to filter certain fields via field tags.
From struct to map[string]interface{}
package main
import (
"fmt"
"encoding/json"
)
type MyData struct {
One int
Two string
Three int
}
func main() {
in := &MyData{One: 1, Two: "second"}
var inInterface map[string]interface{}
inrec, _ := json.Marshal(in)
json.Unmarshal(inrec, &inInterface)
// iterate through inrecs
for field, val := range inInterface {
fmt.Println("KV Pair: ", field, val)
}
}
go playground here
Here is a function I've written in the past to convert a struct to a map, using tags as keys
// ToMap converts a struct to a map using the struct's tags.
//
// ToMap uses tags on struct fields to decide which fields to add to the
// returned map.
func ToMap(in interface{}, tag string) (map[string]interface{}, error){
out := make(map[string]interface{})
v := reflect.ValueOf(in)
if v.Kind() == reflect.Ptr {
v = v.Elem()
}
// we only accept structs
if v.Kind() != reflect.Struct {
return nil, fmt.Errorf("ToMap only accepts structs; got %T", v)
}
typ := v.Type()
for i := 0; i < v.NumField(); i++ {
// gets us a StructField
fi := typ.Field(i)
if tagv := fi.Tag.Get(tag); tagv != "" {
// set key of map to value in struct field
out[tagv] = v.Field(i).Interface()
}
}
return out, nil
}
Runnable example here.
Note, if you have multiple fields with the same tag value, then you will obviously not be able to store them all within a map. It might be prudent to return an error if that happens.
I like the importable package for the accepted answer, but it does not translate my json aliases. Most of my projects have a helper function/class that I import.
Here is a function that solves my specific problem.
// Converts a struct to a map while maintaining the json alias as keys
func StructToMap(obj interface{}) (newMap map[string]interface{}, err error) {
data, err := json.Marshal(obj) // Convert to a json string
if err != nil {
return
}
err = json.Unmarshal(data, &newMap) // Convert to a map
return
}
And in the main, this is how it would be called...
package main
import (
"fmt"
"encoding/json"
"github.com/fatih/structs"
)
type MyStructObject struct {
Email string `json:"email_address"`
}
func main() {
obj := &MyStructObject{Email: "test#test.com"}
// My solution
fmt.Println(StructToMap(obj)) // prints {"email_address": "test#test.com"}
// The currently accepted solution
fmt.Println(structs.Map(obj)) // prints {"Email": "test#test.com"}
}
package main
import (
"fmt"
"reflect"
)
type bill struct {
N1 int
N2 string
n3 string
}
func main() {
a := bill{4, "dhfthf", "fdgdf"}
v := reflect.ValueOf(a)
values := make(map[string]interface{}, v.NumField())
for i := 0; i < v.NumField(); i++ {
if v.Field(i).CanInterface() {
values[v.Type().Field(i).Name] = v.Field(i).Interface()
} else {
fmt.Printf("sorry you have a unexported field (lower case) value you are trying to sneak past. I will not allow it: %v\n", v.Type().Field(i).Name)
}
}
fmt.Println(values)
passObject(&values)
}
func passObject(v1 *map[string]interface{}) {
fmt.Println("yoyo")
}
I'm a bit late but I needed this kind of feature so I wrote this. Can resolve nested structs. By default, uses field names but can also use custom tags. A side effect is that if you set the tagTitle const to json, you could use the json tags you already have.
package main
import (
"fmt"
"reflect"
)
func StructToMap(val interface{}) map[string]interface{} {
//The name of the tag you will use for fields of struct
const tagTitle = "kelvin"
var data map[string]interface{} = make(map[string]interface{})
varType := reflect.TypeOf(val)
if varType.Kind() != reflect.Struct {
// Provided value is not an interface, do what you will with that here
fmt.Println("Not a struct")
return nil
}
value := reflect.ValueOf(val)
for i := 0; i < varType.NumField(); i++ {
if !value.Field(i).CanInterface() {
//Skip unexported fields
continue
}
tag, ok := varType.Field(i).Tag.Lookup(tagTitle)
var fieldName string
if ok && len(tag) > 0 {
fieldName = tag
} else {
fieldName = varType.Field(i).Name
}
if varType.Field(i).Type.Kind() != reflect.Struct {
data[fieldName] = value.Field(i).Interface()
} else {
data[fieldName] = StructToMap(value.Field(i).Interface())
}
}
return data
}
map := Structpb.AsMap()
// map is the map[string]interface{}
I have a quick question about serving files in Go. There is the great timesaving FileServer handler, but for my use case, I only have 2 or 3 files (js and css) that go with my app and I dont want to complicate the deployment to have to think about those.
Do you think there is an easy way to build those couple of files into the binary and serve them from there. For example base64 encode the data of the files as constants and server the files from the constants. This would work in its most simple form, but I dont want to go through the pain of doing everything that a file server does (headers, expiries, mime-types, etc) on my own. So would there be an easy way to bake those static files into the binary in some form and serve them that way?
The FileServer requires a FileSystem object in its constructor. Usually, you would provide something based on http.Dir to make that FileSystem for you from the actual file system, but nothing prevents you from implementing your own:
package main
import "os"
import "time"
import "net/http"
type InMemoryFS map[string]http.File
// Implements FileSystem interface
func (fs InMemoryFS) Open(name string) (http.File, error) {
if f, ok := fs[name]; ok {
return f, nil
}
panic("No file")
}
type InMemoryFile struct {
at int64
Name string
data []byte
fs InMemoryFS
}
func LoadFile(name string, val string, fs InMemoryFS) *InMemoryFile {
return &InMemoryFile{at: 0,
Name: name,
data: []byte(val),
fs: fs}
}
// Implements the http.File interface
func (f *InMemoryFile) Close() error {
return nil
}
func (f *InMemoryFile) Stat() (os.FileInfo, error) {
return &InMemoryFileInfo{f}, nil
}
func (f *InMemoryFile) Readdir(count int) ([]os.FileInfo, error) {
res := make([]os.FileInfo, len(f.fs))
i := 0
for _, file := range f.fs {
res[i], _ = file.Stat()
i++
}
return res, nil
}
func (f *InMemoryFile) Read(b []byte) (int, error) {
i := 0
for f.at < int64(len(f.data)) && i < len(b) {
b[i] = f.data[f.at]
i++
f.at++
}
return i, nil
}
func (f *InMemoryFile) Seek(offset int64, whence int) (int64, error) {
switch whence {
case 0:
f.at = offset
case 1:
f.at += offset
case 2:
f.at = int64(len(f.data)) + offset
}
return f.at, nil
}
type InMemoryFileInfo struct {
file *InMemoryFile
}
// Implements os.FileInfo
func (s *InMemoryFileInfo) Name() string { return s.file.Name }
func (s *InMemoryFileInfo) Size() int64 { return int64(len(s.file.data)) }
func (s *InMemoryFileInfo) Mode() os.FileMode { return os.ModeTemporary }
func (s *InMemoryFileInfo) ModTime() time.Time { return time.Time{} }
func (s *InMemoryFileInfo) IsDir() bool { return false }
func (s *InMemoryFileInfo) Sys() interface{} { return nil }
const HTML = `<html>
Hello world !
</html>
`
const CSS = `
p {
color:red;
text-align:center;
}
`
func main() {
FS := make(InMemoryFS)
FS["foo.html"] = LoadFile("foo.html", HTML, FS)
FS["bar.css"] = LoadFile("bar.css", CSS, FS)
http.Handle("/", http.FileServer(FS))
http.ListenAndServe(":8080", nil)
}
This implementation is very buggy at best, and you should probably never ever use it, but it should show you how the FileSystem interface can be implemented for arbitrary 'files'.
A more credible (and certainly less dangerous) implementation of something similar is available here. This is the one used to fake the filesystem on Go playground, so it should be a good reference (much better than mine anyway).
Whether it is simpler to reimplement this FileSystem interface or a custom FileServer as other suggested, is entirely up to you and your project ! I suspect however that for serving a couple of predefined files, rewriting the serving part might be easier than emulating a full file-system.
The "go.rice" package takes care of this for you - embedding resources in your binaries, and providing an http.FileSystem implementation.
It is not very difficult to do what you request. You don't have to base64 encode it or anything (it will just make it harder for you to edit.).
Below is an example of how to output a javascript file with correct mime type:
package main
import (
"fmt"
"log"
"net/http"
)
const jsFile = `alert('Hello World!');`
func main() {
http.HandleFunc("/file.js", JsHandler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
func JsHandler(w http.ResponseWriter, r *http.Request) {
// Getting the headers so we can set the correct mime type
headers := w.Header()
headers["Content-Type"] = []string{"application/javascript"}
fmt.Fprint(w, jsFile)
}
I would store the files in variable as plain text.
Something like this:
package main
import (
"fmt"
"log"
"net/http"
)
var files = map[string]string{}
func init() {
files["style.css"] = `
/* css file content */
body { background-color: pink; }
`
}
func init() {
files["index.html"] = `
<!-- Html content -->
<html><head>
<link rel="stylesheet" type="text/css" href="style.css">
</head><body>Hello world!</body></html>
`
}
func main() {
for fileName, content := range files {
contentCpy := content
http.HandleFunc("/"+fileName, func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "%s\n", contentCpy)
})
}
log.Fatal(http.ListenAndServe(":8080", nil))
}
That way, it is pretty easy to have your makefile or build script so something like:
for file in index.html style.css; do echo "package main\nfunc init() { files[\"$file\"] = \`$(cat $file)\` }" | gofmt -s > $file.go; done; go build && ./httptest