Why is DynamoDB adding a NULL suffix to attribute names? - amazon-dynamodb

I've been working on a toy project and noticed that dynamo added a (NULL) suffix in two of my attribute names. I had not noticed before, so I assume it must have happened after one of my code changes. I could not find any reference to this behavior online.
The script I'm running is a simple PutItem got from the official Dynamodb documentation, where I insert a few mock users in a table.
func InsertModel(m interface{}) error {
av, err := dynamodbattribute.MarshalMap(m)
if err != nil {
return fmt.Errorf("handlers: Got error marshalling map: %v", err)
}
input := &dynamodb.PutItemInput{
Item: av,
TableName: aws.String(appTableName),
ConditionExpression: aws.String("attribute_not_exists(PK) AND attribute_not_exists(SK)"),
}
_, err = svc.PutItem(input)
if err != nil {
return fmt.Errorf("handlers: Got error calling PutItem: %v", err)
}
return nil
}
m (user mock data) has all fields type string:
UserModel{PK: "910cc6d8-b7e2-dfg6-d8d4-sh6d0e3fde6b", SK: "user_info", Name: "bla", ImageURI: "aaa"},
When I remove the fields "Name" and "ImageURI", the PutItem inserts a boolean true to the field value as seen below.
Here is the value in av after the MarshalMap operation.
with populated "Name" and "ImageURI" fields:
map[ImageURI:{
S: "aaa"
} Name:{
S: "bla"
} PK:{
S: "910cc6d8-b7e2-dfg6-d8d4-sh6d0e3fde6b"
} SK:{
S: "user_info"
}]
and here without "Name" and "ImageURI" as in UserModel{PK: "910cc6d8-b7e2-dfg6-d8d4-sh6d0e3fde6b", SK: "user_info"}
map[ImageURI:{
NULL: true
} Name:{
NULL: true
} PK:{
S: "910cc6d8-b7e2-dfg6-d8d4-sh6d0e3fde6b"
} SK:{
S: "user_info"
}]
I have tried to delete all the records from the table and insert again but the behavior continues. Also, I did the same process for an int type attribute (inserting the object with the int attribute populated and not populated) and I get 0 when it's not populated (which is what I'd expect). I tried replicating this with a different string type attribute, and I get the same behavior (true when empty), but the attribute name doesn't get the suffix NULL.
So in summary, it seems this behavior is mostly happening with type string in my case, and I only get a NULL suffix in the attributes "Name" and "ImageURI", but not on the other string attribute I've tried (nor the int one).

I had the same issue for one of the fields of my table items.
For that field I was doing an update using the NameMap option, useful when you want to use a name that, for some other reasons, is reserved by dynamo.
I just tried not to use the NameMap option, giving another name for my field, and that suffix disappeared.
I hope my experience could be somehow helpful.
Regards

Option A
The dynamodbattribute.MarshalMap method reads json struct tags if present. By adding omitempty it will leave the attribute off of the item. Then when reading it back later, it will default to the empty value in the struct.
type Foo struct {
Bar string `json:"bar,omitempty"`
}
Option B
You can explicitly set empty struct values by either creating your own dynamodb.AttributeValue or implement the marshaller interface on your struct. For example:
item := map[string]*dynamodb.AttributeValue{
"Foo": {
S: aws.String("")
}
}
Making the (NULL) suffix to go away
After deleting all the rows containing a NULL value in a column with the (NULL) suffix, it seems to take some time for the suffix to go away from the AWS UI. When I tested this, it took roughly 12 hours.
Discussion
Continue the discussion of null behaviors on github.
https://github.com/aws/aws-sdk-go/pull/2419
https://github.com/aws/aws-sdk-go/issues?q=MarshalMap+null

Related

Reflection struct field.Set with a Flag pointer value

I have a bunch of flags parsed, and I'm then trying to assign those values to fields in a struct, but I'm struggling to get a parsed flag value set into the struct because I can't type assert it or cast it.
Here is a snippet of the code I have. It's not important to worry too much about the IterFields function, basically the third argument is called for each field in the struct...
Note: there are comments in the code below which highlight the error(s).
flag.Parse()
IterFields(st, v, func(field reflect.Value, sf reflect.StructField) {
flag.VisitAll(func(f *flag.Flag) {
if f.Name == strings.ToLower(sf.Name) || f.Name == sf.Tag.Get("short") {
fmt.Printf("%+v, %T\n", f.Value, f.Value)
// PRINTS: true, *flag.boolValue
if v, ok := f.Value.(bool); ok {
fmt.Println("ok")
} else {
fmt.Println("not ok")
}
// ERROR: impossible type assertion: bool does not implement flag.Value (missing Set method)
field.Set(reflect.ValueOf(f.Value))
// PANIC: value of type *flag.boolValue is not assignable to type bool
}
})
})
f.Value is an interface type flag.Value abstracting all kinds of flag values. As your code indicates, it's not of type bool but some non-exported *flag.boolValue. You shouldn't be concerned about its dynamic type.
You may use the Value.String() method to get its value as a string, which will be either "false" or "true" for bool types, you may use simple comparison to obtain a bool from it like f.Value.String() == "true".
But a better approach would be: all flag.Value values originating from the flag package also implement flag.Getter which also has a Get() method that will directly return a bool value in case of a bool flag (wrapped in interface{} of course). Just use that:
field.Set(reflect.ValueOf(f.Value.(flag.Getter).Get()))
The above works for fields of any type (given that the flag's value type is assignable to the field's type).
For bool fields only, alternatively you may also use:
field.SetBool(f.Value.(flag.Getter).Get().(bool))

How do I handle optional query parameters for a Go using Mux properly?

I'm making an API server using GoLang and I am using Mux as the HTTP router. I have one endpoint in my app, /tasks. When I hit that endpoint, I get back an array of results, which is as expected. Now, I want to build upon that and add an optional query parameter into the URL to only return N results, like so: /tasks?num=5 which would only return 5 results. I have accomplished this using the handler below:
vars := r.URL.Query()
t := task{}
if numOfTasksParam, ok := vars["num"]; ok {
fmt.Printf("%+v", numOfTasksParam[0])
numOfTasks, err := strconv.Atoi(vars.Get("num"))
//return only n number of results
} else {
//return the entire result set
}
I devised this solution because I discovered that URL.Query() returns a map of the query parameters and therefore, I can just check to see if that map contained the key of "num". If so, the client wants N number of results. If not, the client must want the whole result set.
The main issue I have with this approach is that when I go to check if the keys exists, I make a temporary variable called numOfTasksParam which holds the query parameter value, but it holds the value as a string and I need a int. Therefore, I must use the numOfTasksParam somehow and then create another variable to convert that to an integer value.
Is there more succinct or convenient way of checking if a query parameter exists in the request URL?
This is probably the most succinct, and works because Get returns an empty string if the parameter isn't set, which Atoi will fail to parse:
vars := r.URL.Query()
t := task{}
if numOfTasks, err := strconv.Atoi(vars.Get("num")); err == nil {
//return only numOfTasks number of results
} else {
//return the entire result set
}
The price you pay for having less code is that if a user passes an invalid value like ?num=taco, it will be treated as if they passed nothing, rather than telling the user they passed something unusable. This may or may not be what you want. It is also slightly less efficient, because it will run Atoi even if the value is known to be empty.

Is it "bad form" to perform map lookup and type assertion in one statement?

I just realized that it's possible to do a map lookup and a type/interface-assertion in one statement.
m := map[string]interface{}{
"key": "the value",
}
if value, ok := m["key"].(string); ok {
fmt.Printf("value exists and is a string: %s\n", value)
} else {
fmt.Println("value does not exist or is not a string")
}
Is this considered bad?
I've not seen any official documentation commenting this.
edit: I know this code can not distinguish between "key doesn't exist" and "value is of incorrect type".
edit2: ahem, the print in the else-clause was incorrect :(
What you do actually works, because you use the special comma-ok idiom of the type assertion which does not panic if the assertion does not hold, and because maps can be indexed with keys which are not in it (which will result in the zero value of the value type of the map).
It's true that with this you can't tell if the key is in the map or not, or it is but its value is nil, but you already suspected this as if the assertion does not hold, you print "value does not exist or is not a string".
To test all "corner" cases, see this example:
m := map[string]interface{}{
"key": "the value",
"key2": 2,
"key3": nil,
// "key4":"", // Uncommented on purpose
}
for _, k := range []string{"key", "key2", "key3", "key4"} {
if value, ok := m[k].(string); ok {
fmt.Printf("[key: %s] value exists and is a string: %s\n", k, value)
} else {
fmt.Printf("[key: %s] value does not exist or is not a string: %s\n",
k, value)
}
}
Output (try it on the Go Playground):
[key: key] value exists and is a string: the value
[key: key2] value does not exist or is not a string:
[key: key3] value does not exist or is not a string:
[key: key4] value does not exist or is not a string:
So basically you can use this, nothing bad will happen from it (e.g. panic or memory leak), just know its limits (e.g. you couldn't get the value associated for "key2" as it's not of string type).
If your intention is to get the value for a key if it exists and is of type string, then this is what your code exactly does. Although you should avoid data structures and constructs where you need this, as it's harder to understand and maintain in larger projects.
What I mean by this is for example if at some point in your code you expect the key "somekey" to have a string value associated and it does not, you won't immediately know why it is so; is it because the map does not contain that key, or is it because it does but with a value of wrong type (or the value may even be nil)? Further testing / debugging is needed to track down the root cause.
Is this considered bad?
No, this is plain wrong. You cannot combine map lookup and type assertion like this.
What you code does is:
It looks up "key" in the map unconditional. If "key" is not in the map it will yield nil (as this is the zero value of interface{}).
The value retrieved from the map (possible nil if not in the map is type asserted to a string.
Your code is incapable to determine whether "key" is in the map or not.

Reflect on Time type in go

I am trying to write a csv parser using the example provided here. It works great for all native types but I am having trouble with any structs that contain a timestamp of type time.Time. It exits with an error of "cannot convert this type".
This is the code.
//For each field in a given struct...
//Get a field
val := sv.Field(i)
// this is necessary because Kind can't tell
// distinguish between a primitive type
// and a type derived from it. We're looking
// for a Value interface defined on
// the pointer to this value
_, ok := val.Addr().Interface().(Value)
if ok {
val = val.Addr()
kind = value_k
} else {
switch Kind {
case reflect.Int, reflect.Int16, reflect.Int8,
reflect.Int32, reflect.Int64:
kind = int_k
case reflect.Uint, reflect.Uint16, reflect.Uint8,
reflect.Uint32, reflect.Uint64:
kind = uint_k
case reflect.Float32, reflect.Float64:
kind = float_k
case reflect.String:
kind = string_k
default:
// Kind is Struct here
kind = value_k
_, ok := val.Interface().(Value)
if !ok {
err = os.NewError("cannot convert this type ")
this = nil
return
}
}
}
What this code does is take an interface and a reader. It attempts to match the field headers in the reader (csv file) with field names in the interface. It also reflects on the interface (struct) and collects positional a type information for later setting the fields in the iterator. It is this step that is failing for non-native types.
I've tried a few methods to work around this but the only thing that seems to work is changing the timestamp to a string. I am undoubtedly missing something and would greatly appreciate some guidance.

How to check if a map contains a key in Go?

I know I can iterate over a map m with
for k, v := range m { ... }
and look for a key, but is there a more efficient way of testing for a key's existence in a map?
Here's how you check if a map contains a key.
val, ok := myMap["foo"]
// If the key exists
if ok {
// Do something
}
This initializes two variables. val is the value of "foo" from the map if it exists, or a "zero value" if it doesn't (in this case the empty string). ok is a bool that will be set to true if the key existed.
If you want, you can shorten this to a one-liner.
if val, ok := myMap["foo"]; ok {
//do something here
}
Go allows you to put an initializing statement before the condition (notice the semicolon) in the if statement. The consequence of this is that the scope ofval and ok will be limited to the body of the if statement, which is helpful if you only need to access them there.
In addition to The Go Programming Language Specification, you should read Effective Go. In the section on maps, they say, amongst other things:
An attempt to fetch a map value with a key that is not present in the
map will return the zero value for the type of the entries in the map.
For instance, if the map contains integers, looking up a non-existent
key will return 0. A set can be implemented as a map with value type
bool. Set the map entry to true to put the value in the set, and then
test it by simple indexing.
attended := map[string]bool{
"Ann": true,
"Joe": true,
...
}
if attended[person] { // will be false if person is not in the map
fmt.Println(person, "was at the meeting")
}
Sometimes you need to distinguish a missing entry from a zero value.
Is there an entry for "UTC" or is that 0 because it's not in the map
at all? You can discriminate with a form of multiple assignment.
var seconds int
var ok bool
seconds, ok = timeZone[tz]
For obvious reasons this is called the “comma ok” idiom. In this
example, if tz is present, seconds will be set appropriately and ok
will be true; if not, seconds will be set to zero and ok will be
false. Here's a function that puts it together with a nice error
report:
func offset(tz string) int {
if seconds, ok := timeZone[tz]; ok {
return seconds
}
log.Println("unknown time zone:", tz)
return 0
}
To test for presence in the map without worrying about the actual
value, you can use the blank identifier (_) in place of the usual
variable for the value.
_, present := timeZone[tz]
Searched on the go-nuts email list and found a solution posted by Peter Froehlich on 11/15/2009.
package main
import "fmt"
func main() {
dict := map[string]int {"foo" : 1, "bar" : 2}
value, ok := dict["baz"]
if ok {
fmt.Println("value: ", value)
} else {
fmt.Println("key not found")
}
}
Or, more compactly,
if value, ok := dict["baz"]; ok {
fmt.Println("value: ", value)
} else {
fmt.Println("key not found")
}
Note, using this form of the if statement, the value and ok variables are only visible inside the if conditions.
Short Answer
_, exists := timeZone[tz] // Just checks for key existence
val, exists := timeZone[tz] // Checks for key existence and retrieves the value
Example
Here's an example at the Go Playground.
Longer Answer
Per the Maps section of Effective Go:
An attempt to fetch a map value with a key that is not present in the map will return the zero value for the type of the entries in the map. For instance, if the map contains integers, looking up a non-existent key will return 0.
Sometimes you need to distinguish a missing entry from a zero value. Is there an entry for "UTC" or is that the empty string because it's not in the map at all? You can discriminate with a form of multiple assignment.
var seconds int
var ok bool
seconds, ok = timeZone[tz]
For obvious reasons this is called the “comma ok” idiom. In this example, if tz is present, seconds will be set appropriately and ok will be true; if not, seconds will be set to zero and ok will be false. Here's a function that puts it together with a nice error report:
func offset(tz string) int {
if seconds, ok := timeZone[tz]; ok {
return seconds
}
log.Println("unknown time zone:", tz)
return 0
}
To test for presence in the map without worrying about the actual value, you can use the blank identifier (_) in place of the usual variable for the value.
_, present := timeZone[tz]
Have a look at this snippet of code
nameMap := make(map[string]int)
nameMap["river"] = 33
v ,exist := nameMap["river"]
if exist {
fmt.Println("exist ",v)
}
As noted by other answers, the general solution is to use an index expression in an assignment of the special form:
v, ok = a[x]
v, ok := a[x]
var v, ok = a[x]
var v, ok T = a[x]
This is nice and clean. It has some restrictions though: it must be an assignment of special form. Right-hand side expression must be the map index expression only, and the left-hand expression list must contain exactly 2 operands, first to which the value type is assignable, and a second to which a bool value is assignable. The first value of the result of this special form will be the value associated with the key, and the second value will tell if there is actually an entry in the map with the given key (if the key exists in the map). The left-hand side expression list may also contain the blank identifier if one of the results is not needed.
It's important to know that if the indexed map value is nil or does not contain the key, the index expression evaluates to the zero value of the value type of the map. So for example:
m := map[int]string{}
s := m[1] // s will be the empty string ""
var m2 map[int]float64 // m2 is nil!
f := m2[2] // f will be 0.0
fmt.Printf("%q %f", s, f) // Prints: "" 0.000000
Try it on the Go Playground.
So if we know that we don't use the zero value in our map, we can take advantage of this.
For example if the value type is string, and we know we never store entries in the map where the value is the empty string (zero value for the string type), we can also test if the key is in the map by comparing the non-special form of the (result of the) index expression to the zero value:
m := map[int]string{
0: "zero",
1: "one",
}
fmt.Printf("Key 0 exists: %t\nKey 1 exists: %t\nKey 2 exists: %t",
m[0] != "", m[1] != "", m[2] != "")
Output (try it on the Go Playground):
Key 0 exists: true
Key 1 exists: true
Key 2 exists: false
In practice there are many cases where we don't store the zero-value value in the map, so this can be used quite often. For example interfaces and function types have a zero value nil, which we often don't store in maps. So testing if a key is in the map can be achieved by comparing it to nil.
Using this "technique" has another advantage too: you can check existence of multiple keys in a compact way (you can't do that with the special "comma ok" form). More about this: Check if key exists in multiple maps in one condition
Getting the zero value of the value type when indexing with a non-existing key also allows us to use maps with bool values conveniently as sets. For example:
set := map[string]bool{
"one": true,
"two": true,
}
fmt.Println("Contains 'one':", set["one"])
if set["two"] {
fmt.Println("'two' is in the set")
}
if !set["three"] {
fmt.Println("'three' is not in the set")
}
It outputs (try it on the Go Playground):
Contains 'one': true
'two' is in the set
'three' is not in the set
See related: How can I create an array that contains unique strings?
var d map[string]string
value, ok := d["key"]
if ok {
fmt.Println("Key Present ", value)
} else {
fmt.Println(" Key Not Present ")
}
var empty struct{}
var ok bool
var m map[string]struct{}
m = make(map[string]struct{})
m["somestring"] = empty
_, ok = m["somestring"]
fmt.Println("somestring exists?", ok)
_, ok = m["not"]
fmt.Println("not exists?", ok)
Then, go run maps.go
somestring exists? true
not exists? false
It is mentioned under "Index expressions".
An index expression on a map a of type map[K]V used in an assignment
or initialization of the special form
v, ok = a[x]
v, ok := a[x]
var v, ok = a[x]
yields an additional untyped boolean value. The value of ok is true if
the key x is present in the map, and false otherwise.
A two value assignment can be used for this purpose. Please check my sample program below
package main
import (
"fmt"
)
func main() {
//creating a map with 3 key-value pairs
sampleMap := map[string]int{"key1": 100, "key2": 500, "key3": 999}
//A two value assignment can be used to check existence of a key.
value, isKeyPresent := sampleMap["key2"]
//isKeyPresent will be true if key present in sampleMap
if isKeyPresent {
//key exist
fmt.Println("key present, value = ", value)
} else {
//key does not exist
fmt.Println("key does not exist")
}
}
Example usage: Looping through a slice, for pairMap checking if key exists.
It an algorithm to find all pairs that adds to a specific sum.
func findPairs(slice1 []int, sum int) {
pairMap := make(map[int]int)
for i, v := range slice1 {
if valuei, ok := pairMap[v]; ok {
fmt.Println("Pair Found", i, valuei)
} else {
pairMap[sum-v] = i
}
}
}

Resources