I have some data in form of map and I'm converting it to []byt and signing it and when verifying, it gives True value even when data used for verifying and signing are different.
Here is what I did-
func main(){
n, _ := ioutil.ReadFile("privatekey")
private_key,_ := x509.ParseECPrivateKey(n)
public_key := private_key.PublicKey
data := map[string]string{
"data1": "somestring",
"data2": "12312",
"data3": "34fs4",
}
json_data, _ := json.Marshal(data)
data_2 := map[string]string{
"data1": "somestring",
"data2": "13312",
"data4": "fh34",
}
json_data_2,_ := json.Marshal(data_2)
r, s, _ := ecdsa.Sign(rand.Reader, private_key, json_data)
verifystatus := ecdsa.Verify(&public_key, json_data_2, r, s)
fmt.Println(verifystatus)
}
It is printing true. I tried changing the data and it seems that If json_data and json_data_2 have first 32 bytes common, then Verify returns true.
Is there some limit over the length of byte array I can send to ecdsa.Verify()? If so how can I use it for larger data?
The golang ecdsa.Sign and ecdsa.Verify functions are expected to take the output of a cryptographic hash function, rather than the message itself. So you are correct that only the first 32 bytes are being examined, in this case.
To resolve the problem first hash the messages using a cryptographic hash function such as SHA-2
Related
Labstack Echo v4 has recently been updated to include ModifyResponse hook (ala golang's httputil ReverseProxy).
Most of the examples of using this seem to leverage ioutil.ReadAll().
For example:
func UpdateResponse(r *http.Response) error {
b, _ := ioutil.ReadAll(r.Body)
buf := bytes.NewBufferString("Monkey")
buf.Write(b)
r.Body = ioutil.NopCloser(buf)
r.Header["Content-Length"] = []string{fmt.Sprint(buf.Len())}
return nil
}
What I am looking to do is to avoid waiting for the entire response (from ReadAll) and monitor the stream for certain content (i.e. class='blue') then replace it with different text (i.e. class='blue-green')
How can this be done using streams efficiently and with as little allocations as possible?
How can one remove selected keys from a map?
Is it safe to combine delete() with range, as in the code below?
package main
import "fmt"
type Info struct {
value string
}
func main() {
table := make(map[string]*Info)
for i := 0; i < 10; i++ {
str := fmt.Sprintf("%v", i)
table[str] = &Info{str}
}
for key, value := range table {
fmt.Printf("deleting %v=>%v\n", key, value.value)
delete(table, key)
}
}
https://play.golang.org/p/u1vufvEjSw
This is safe! You can also find a similar sample in Effective Go:
for key := range m {
if key.expired() {
delete(m, key)
}
}
And the language specification:
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next. If map entries that have not yet been reached are removed during iteration, the corresponding iteration values will not be produced. If map entries are created during iteration, that entry may be produced during the iteration or may be skipped. The choice may vary for each entry created and from one iteration to the next. If the map is nil, the number of iterations is 0.
Sebastian's answer is accurate, but I wanted to know why it was safe, so I did some digging into the Map source code. It looks like on a call to delete(k, v), it basically just sets a flag (as well as changing the count value) instead of actually deleting the value:
b->tophash[i] = Empty;
(Empty is a constant for the value 0)
What the map appears to actually be doing is allocating a set number of buckets depending on the size of the map, which grows as you perform inserts at the rate of 2^B (from this source code):
byte *buckets; // array of 2^B Buckets. may be nil if count==0.
So there are almost always more buckets allocated than you're using, and when you do a range over the map, it checks that tophash value of each bucket in that 2^B to see if it can skip over it.
To summarize, the delete within a range is safe because the data is technically still there, but when it checks the tophash it sees that it can just skip over it and not include it in whatever range operation you're performing. The source code even includes a TODO:
// TODO: consolidate buckets if they are mostly empty
// can only consolidate if there are no live iterators at this size.
This explains why using the delete(k,v) function doesn't actually free up memory, just removes it from the list of buckets you're allowed to access. If you want to free up the actual memory you'll need to make the entire map unreachable so that garbage collection will step in. You can do this using a line like
map = nil
I was wondering if a memory leak could happen. So I wrote a test program:
package main
import (
log "github.com/Sirupsen/logrus"
"os/signal"
"os"
"math/rand"
"time"
)
func main() {
log.Info("=== START ===")
defer func() { log.Info("=== DONE ===") }()
go func() {
m := make(map[string]string)
for {
k := GenerateRandStr(1024)
m[k] = GenerateRandStr(1024*1024)
for k2, _ := range m {
delete(m, k2)
break
}
}
}()
osSignals := make(chan os.Signal, 1)
signal.Notify(osSignals, os.Interrupt)
for {
select {
case <-osSignals:
log.Info("Recieved ^C command. Exit")
return
}
}
}
func GenerateRandStr(n int) string {
rand.Seed(time.Now().UnixNano())
const letterBytes = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
b := make([]byte, n)
for i := range b {
b[i] = letterBytes[rand.Int63() % int64(len(letterBytes))]
}
return string(b)
}
Looks like GC do frees the memory. So it's okay.
In short, yes. See previous answers.
And also this, from here:
ianlancetaylor commented on Feb 18, 2015
I think the key to understanding this is to realize that while executing the body of a for/range statement, there is no current iteration. There is a set of values that have been seen, and a set of values that have not been seen. While executing the body, one of the key/value pairs that has been seen--the most recent pair--was assigned to the variable(s) of the range statement. There is nothing special about that key/value pair, it's just one of the ones that has already been seen during the iteration.
The question he's answering is about modifying map elements in place during a range operation, which is why he mentions the "current iteration". But it's also relevant here: you can delete keys during a range, and that just means that you won't see them later on in the range (and if you already saw them, that's okay).
I have some trouble building a function that can dynamically use parametrized structs. For that reason my code has 20+ functions that are similar except basically for one type that gets used. Most of my experience is with Java, and I'd just develop basic generic functions, or use plain Object as parameter to function (and reflection from that point on). I would need something similar, using Go.
I have several types like:
// The List structs are mostly needed for json marshalling
type OrangeList struct {
Oranges []Orange
}
type BananaList struct {
Bananas []Banana
}
type Orange struct {
Orange_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
type Banana struct {
Banana_id string
Field_1 int
// The fields are different for different types, I am simplifying the code example
}
Then I have function, basically for each list type:
// In the end there are 20+ of these, the only difference is basically in two types!
// This is very un-DRY!
func buildOranges(rows *sqlx.Rows) ([]byte, error) {
oranges := OrangeList{} // This type changes
for rows.Next() {
orange := Orange{} // This type changes
err := rows.StructScan(&orange) // This can handle each case already, could also use reflect myself too
checkError(err, "rows.Scan")
oranges.Oranges = append(oranges.Oranges,orange)
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(oranges)
return jsontext, err
}
Yes, I could change the sql library to use more intelligent ORM or framework, but that's besides the point. I want to learn on how to build generic function that can handle similar function for all my different types.
I got this far, but it still doesn't work properly (target isn't expected struct I think):
func buildWhatever(rows *sqlx.Rows, tgt interface{}) ([]byte, error) {
tgtValueOf := reflect.ValueOf(tgt)
tgtType := tgtValueOf.Type()
targets := reflect.SliceOf(tgtValueOf.Type())
for rows.Next() {
target := reflect.New(tgtType)
err := rows.StructScan(&target) // At this stage target still isn't 1:1 smilar struct so the StructScan fails... It's some perverted "Value" object instead. Meh.
// Removed appending to the list because the solutions for that would be similar
checkError(err, "rows.Scan")
}
checkError(rows.Err(), "rows.Err")
jsontext, err := json.Marshal(targets)
return jsontext, err
}
So umm, I would need to give the list type, and the vanilla type as parameters, then build one of each, and the rest of my logic would be probably fixable quite easily.
Turns out there's an sqlx.StructScan(rows, &destSlice) function that will do your inner loop, given a slice of the appropriate type. The sqlx docs refer to caching results of reflection operations, so it may have some additional optimizations compared to writing one.
Sounds like the immediate question you're actually asking is "how do I get something out of my reflect.Value that rows.StructScan will accept?" And the direct answer is reflect.Interface(target); it should return an interface{} representing an *Orange you can pass directly to StructScan (no additional & operation needed). Then, I think targets = reflect.Append(targets, target.Indirect()) will turn your target into a reflect.Value representing an Orange and append it to the slice. targets.Interface() should get you an interface{} representing an []Orange that json.Marshal understands. I say all these 'should's and 'I think's because I haven't tried that route.
Reflection, in general, is verbose and slow. Sometimes it's the best or only way to get something done, but it's often worth looking for a way to get your task done without it when you can.
So, if it works in your app, you can also convert Rows straight to JSON, without going through intermediate structs. Here's a sample program (requires sqlite3 of course) that turns sql.Rows into map[string]string and then into JSON. (Note it doesn't try to handle NULL, represent numbers as JSON numbers, or generally handle anything that won't fit in a map[string]string.)
package main
import (
_ "code.google.com/p/go-sqlite/go1/sqlite3"
"database/sql"
"encoding/json"
"os"
)
func main() {
db, err := sql.Open("sqlite3", "foo")
if err != nil {
panic(err)
}
tryQuery := func(query string, args ...interface{}) *sql.Rows {
rows, err := db.Query(query, args...)
if err != nil {
panic(err)
}
return rows
}
tryQuery("drop table if exists t")
tryQuery("create table t(i integer, j integer)")
tryQuery("insert into t values(?, ?)", 1, 2)
tryQuery("insert into t values(?, ?)", 3, 1)
// now query and serialize
rows := tryQuery("select * from t")
names, err := rows.Columns()
if err != nil {
panic(err)
}
// vals stores the values from one row
vals := make([]interface{}, 0, len(names))
for _, _ = range names {
vals = append(vals, new(string))
}
// rowMaps stores all rows
rowMaps := make([]map[string]string, 0)
for rows.Next() {
rows.Scan(vals...)
// now make value list into name=>value map
currRow := make(map[string]string)
for i, name := range names {
currRow[name] = *(vals[i].(*string))
}
// accumulating rowMaps is the easy way out
rowMaps = append(rowMaps, currRow)
}
json, err := json.Marshal(rowMaps)
if err != nil {
panic(err)
}
os.Stdout.Write(json)
}
In theory, you could build this to do fewer allocations by not reusing the same rowMap each time and using a json.Encoder to append each row's JSON to a buffer. You could go a step further and not use a rowMap at all, just the lists of names and values. I should say I haven't compared the speed against a reflect-based approach, though I know reflect is slow enough it might be worth comparing them if you can put up with either strategy.
The Go Programming Language Specification says:
3. The iteration order over maps is not specified. [...]
That's to be expected since a map type can be implemented as a hash table, or as a search tree, or as some other data structure. But how is map actually implemented in Go?
Put differently, what determines the iteration order of the keys in
for k, _ := range m { fmt.Println(k) }
I started wondering about this after I saw that a map with string keys apparently do have a certain iteration order. A program like
package main
import ("fmt"; "time"; "rand")
func main() {
rand.Seed(time.Seconds())
words := [...]string{"foo", "bar", "a", "b", "c", "hello", "world",
"0", "1", "10", "100", "123"}
stringMap := make(map[string]byte)
for i := range rand.Perm(len(words)) {
stringMap[words[i]] = byte(rand.Int())
}
fmt.Print("stringMap keys:")
for k, _ := range stringMap { fmt.Print(" ", k) }
fmt.Println()
}
prints the following on my machine:
stringMap keys: a c b 100 hello foo bar 10 world 123 1 0
regardless of the insertion order.
The equivalent program with a map[byte]byte map also prints the keys in a shuffled order, but here the key order depends on the insertion order.
How is all this implemented? Is the map specialized for integers and for strings?
Map is implemented in Go as a hashmap.
The Go run-time uses a common hashmap implementation which is implemented in C. The only implementation differences between map[string]T and map[byte]T are: hash function, equivalence function and copy function.
Unlike (some) C++ maps, Go maps aren't fully specialized for integers and for strings.
In Go release.r60, the iteration order is independent from insertion order as long as there are no key collisions. If there are collisions, iteration order is affected by insertion order. This holds true regardless of key type. There is no difference between keys of type string and keys of type byte in this respect, so it is only a coincidence that your program always printed the string keys in the same order. The iteration order is always the same unless the map is modified.
However, in the newest Go weekly release (and in Go1 which may be expected to be released this month), the iteration order is randomized (it starts at a pseudo-randomly chosen key, and the hashcode computation is seeded with a pseudo-random number). If you compile your program with the weekly release (and with Go1), the iteration order will be different each time you run your program. That said, running your program an infinite number of times probably wouldn't print all possible permutations of the key set. Example outputs:
stringMap keys: b 0 hello c world 10 1 123 bar foo 100 a
stringMap keys: hello world c 1 10 bar foo 123 100 a b 0
stringMap keys: bar foo 123 100 world c 1 10 b 0 hello a
...
If the specs say the iteration order is not specified then a specific order in specific cases is not ruled out.
The point is one cannot rely on that order in any case, not even in some special case. The implementation is free to change this behavior at any given moment, run time included.
Note that it is not that odd for order to be stable regardless of insertion order if there is a total order over the keys (as there frequently may be if they are of a homogenous type); if nothing else, it can allow efficient searching over keys which generate the same hash.
This may well also reflect a different underlying implementation - in particular, this is something people might want for strings, but for integers you could use a sparse array instead.
To extend #user811773 answer. A semi-random range clause iteration does not mean that the chance of returning each element in a map is also equal. See https://medium.com/i0exception/map-iteration-in-go-275abb76f721 and https://play.golang.org/p/GpSd8i7XZoG.
package main
import "fmt"
type intSet map[int]struct{}
func (s intSet) put(v int) { s[v] = struct{}{} }
func (s intSet) get() (int, bool) {
for k := range s {
return k, true
}
return 0, false
}
func main() {
s := make(intSet)
for i := 0; i < 4; i++ {
s.put(i)
}
counts := make(map[int]int)
for i := 0; i < 1024*1024; i++ {
v, ok := s.get()
if !ok {return}
counts[v]++
}
for k, v := range counts {
fmt.Printf("Value: %v, Count: %v\n", k, v)
}
}
/*
Value: 1, Count: 130752
Value: 2, Count: 130833
Value: 0, Count: 655840
Value: 3, Count: 131151
*/
I know I can iterate over a map m with
for k, v := range m { ... }
and look for a key, but is there a more efficient way of testing for a key's existence in a map?
Here's how you check if a map contains a key.
val, ok := myMap["foo"]
// If the key exists
if ok {
// Do something
}
This initializes two variables. val is the value of "foo" from the map if it exists, or a "zero value" if it doesn't (in this case the empty string). ok is a bool that will be set to true if the key existed.
If you want, you can shorten this to a one-liner.
if val, ok := myMap["foo"]; ok {
//do something here
}
Go allows you to put an initializing statement before the condition (notice the semicolon) in the if statement. The consequence of this is that the scope ofval and ok will be limited to the body of the if statement, which is helpful if you only need to access them there.
In addition to The Go Programming Language Specification, you should read Effective Go. In the section on maps, they say, amongst other things:
An attempt to fetch a map value with a key that is not present in the
map will return the zero value for the type of the entries in the map.
For instance, if the map contains integers, looking up a non-existent
key will return 0. A set can be implemented as a map with value type
bool. Set the map entry to true to put the value in the set, and then
test it by simple indexing.
attended := map[string]bool{
"Ann": true,
"Joe": true,
...
}
if attended[person] { // will be false if person is not in the map
fmt.Println(person, "was at the meeting")
}
Sometimes you need to distinguish a missing entry from a zero value.
Is there an entry for "UTC" or is that 0 because it's not in the map
at all? You can discriminate with a form of multiple assignment.
var seconds int
var ok bool
seconds, ok = timeZone[tz]
For obvious reasons this is called the “comma ok” idiom. In this
example, if tz is present, seconds will be set appropriately and ok
will be true; if not, seconds will be set to zero and ok will be
false. Here's a function that puts it together with a nice error
report:
func offset(tz string) int {
if seconds, ok := timeZone[tz]; ok {
return seconds
}
log.Println("unknown time zone:", tz)
return 0
}
To test for presence in the map without worrying about the actual
value, you can use the blank identifier (_) in place of the usual
variable for the value.
_, present := timeZone[tz]
Searched on the go-nuts email list and found a solution posted by Peter Froehlich on 11/15/2009.
package main
import "fmt"
func main() {
dict := map[string]int {"foo" : 1, "bar" : 2}
value, ok := dict["baz"]
if ok {
fmt.Println("value: ", value)
} else {
fmt.Println("key not found")
}
}
Or, more compactly,
if value, ok := dict["baz"]; ok {
fmt.Println("value: ", value)
} else {
fmt.Println("key not found")
}
Note, using this form of the if statement, the value and ok variables are only visible inside the if conditions.
Short Answer
_, exists := timeZone[tz] // Just checks for key existence
val, exists := timeZone[tz] // Checks for key existence and retrieves the value
Example
Here's an example at the Go Playground.
Longer Answer
Per the Maps section of Effective Go:
An attempt to fetch a map value with a key that is not present in the map will return the zero value for the type of the entries in the map. For instance, if the map contains integers, looking up a non-existent key will return 0.
Sometimes you need to distinguish a missing entry from a zero value. Is there an entry for "UTC" or is that the empty string because it's not in the map at all? You can discriminate with a form of multiple assignment.
var seconds int
var ok bool
seconds, ok = timeZone[tz]
For obvious reasons this is called the “comma ok” idiom. In this example, if tz is present, seconds will be set appropriately and ok will be true; if not, seconds will be set to zero and ok will be false. Here's a function that puts it together with a nice error report:
func offset(tz string) int {
if seconds, ok := timeZone[tz]; ok {
return seconds
}
log.Println("unknown time zone:", tz)
return 0
}
To test for presence in the map without worrying about the actual value, you can use the blank identifier (_) in place of the usual variable for the value.
_, present := timeZone[tz]
Have a look at this snippet of code
nameMap := make(map[string]int)
nameMap["river"] = 33
v ,exist := nameMap["river"]
if exist {
fmt.Println("exist ",v)
}
As noted by other answers, the general solution is to use an index expression in an assignment of the special form:
v, ok = a[x]
v, ok := a[x]
var v, ok = a[x]
var v, ok T = a[x]
This is nice and clean. It has some restrictions though: it must be an assignment of special form. Right-hand side expression must be the map index expression only, and the left-hand expression list must contain exactly 2 operands, first to which the value type is assignable, and a second to which a bool value is assignable. The first value of the result of this special form will be the value associated with the key, and the second value will tell if there is actually an entry in the map with the given key (if the key exists in the map). The left-hand side expression list may also contain the blank identifier if one of the results is not needed.
It's important to know that if the indexed map value is nil or does not contain the key, the index expression evaluates to the zero value of the value type of the map. So for example:
m := map[int]string{}
s := m[1] // s will be the empty string ""
var m2 map[int]float64 // m2 is nil!
f := m2[2] // f will be 0.0
fmt.Printf("%q %f", s, f) // Prints: "" 0.000000
Try it on the Go Playground.
So if we know that we don't use the zero value in our map, we can take advantage of this.
For example if the value type is string, and we know we never store entries in the map where the value is the empty string (zero value for the string type), we can also test if the key is in the map by comparing the non-special form of the (result of the) index expression to the zero value:
m := map[int]string{
0: "zero",
1: "one",
}
fmt.Printf("Key 0 exists: %t\nKey 1 exists: %t\nKey 2 exists: %t",
m[0] != "", m[1] != "", m[2] != "")
Output (try it on the Go Playground):
Key 0 exists: true
Key 1 exists: true
Key 2 exists: false
In practice there are many cases where we don't store the zero-value value in the map, so this can be used quite often. For example interfaces and function types have a zero value nil, which we often don't store in maps. So testing if a key is in the map can be achieved by comparing it to nil.
Using this "technique" has another advantage too: you can check existence of multiple keys in a compact way (you can't do that with the special "comma ok" form). More about this: Check if key exists in multiple maps in one condition
Getting the zero value of the value type when indexing with a non-existing key also allows us to use maps with bool values conveniently as sets. For example:
set := map[string]bool{
"one": true,
"two": true,
}
fmt.Println("Contains 'one':", set["one"])
if set["two"] {
fmt.Println("'two' is in the set")
}
if !set["three"] {
fmt.Println("'three' is not in the set")
}
It outputs (try it on the Go Playground):
Contains 'one': true
'two' is in the set
'three' is not in the set
See related: How can I create an array that contains unique strings?
var d map[string]string
value, ok := d["key"]
if ok {
fmt.Println("Key Present ", value)
} else {
fmt.Println(" Key Not Present ")
}
var empty struct{}
var ok bool
var m map[string]struct{}
m = make(map[string]struct{})
m["somestring"] = empty
_, ok = m["somestring"]
fmt.Println("somestring exists?", ok)
_, ok = m["not"]
fmt.Println("not exists?", ok)
Then, go run maps.go
somestring exists? true
not exists? false
It is mentioned under "Index expressions".
An index expression on a map a of type map[K]V used in an assignment
or initialization of the special form
v, ok = a[x]
v, ok := a[x]
var v, ok = a[x]
yields an additional untyped boolean value. The value of ok is true if
the key x is present in the map, and false otherwise.
A two value assignment can be used for this purpose. Please check my sample program below
package main
import (
"fmt"
)
func main() {
//creating a map with 3 key-value pairs
sampleMap := map[string]int{"key1": 100, "key2": 500, "key3": 999}
//A two value assignment can be used to check existence of a key.
value, isKeyPresent := sampleMap["key2"]
//isKeyPresent will be true if key present in sampleMap
if isKeyPresent {
//key exist
fmt.Println("key present, value = ", value)
} else {
//key does not exist
fmt.Println("key does not exist")
}
}
Example usage: Looping through a slice, for pairMap checking if key exists.
It an algorithm to find all pairs that adds to a specific sum.
func findPairs(slice1 []int, sum int) {
pairMap := make(map[int]int)
for i, v := range slice1 {
if valuei, ok := pairMap[v]; ok {
fmt.Println("Pair Found", i, valuei)
} else {
pairMap[sum-v] = i
}
}
}