I am trying to understand the channel in GO. For that written simple code
func test(names []string) {
ch := make(chan string)
for _, s := range names {
go func(s string) {
fmt.Println("sending : ", s)
ch <- s
}(s)
}
for range ch {
fmt.Println("receiving ", <-ch)
}
}
func main() {
test([]string{"test1", "test2"})
}
When I ran the code it out put the following
sending : test2
sending : test1
receiving test1
fatal error: all goroutines are asleep - deadlock!
According to my understanding <-ch will receive as soon as 2 different go routines send test2 and test1 .
so according to my understanding first test2 is sent then test2 should be received then test1 is sent and test1 is received .
However test2 is never received.
What is wrong here ?
While trying to reason about I tried following
func test(names []string) {
ch := make(chan string)
for _, s := range names {
go func(s string) {
fmt.Println("sending : ", s)
ch <- s
}(s)
}
for range names {
fmt.Println("receiving ", <-ch)
}
}
I got the result as expected
sending : test2
receiving test2
sending : test1
receiving test1
Not able to understand for range names works as expected but for range ch does not work . Could you please help me to understand what is going on here?
Related
I have set up a basic project on Github: https://github.com/kounelios13/range-download.
Essentially this project tries to download a file using HTTP Range requests, assemble it, and save it back to disk. I am trying to follow this article (apart from the goroutines for the time being). When I try to download the file using range requests the final size, after all requests data are combined, is bigger than the original size I would get have and the final file is corrupted.
Here is the code responsible for downloading the file
type Manager struct{
limit int
}
func NewManager(limit int) *Manager{
return &Manager{
limit: limit,
}
}
func (m *Manager) DownloadBody(url string ) ([]byte ,error){
// First we need to determine the filesize
body := make([]byte ,0)
response , err := http.Head(url) // We perform a Head request to get header information
if response.StatusCode != http.StatusOK{
return nil ,fmt.Errorf("received code %d",response.StatusCode)
}
if err != nil{
return nil , err
}
maxConnections := m.limit // Number of maximum concurrent co routines
bodySize , _ := strconv.Atoi(response.Header.Get("Content-Length"))
bufferSize :=(bodySize) / (maxConnections)
diff := bodySize % maxConnections
read := 0
for i:=0;i<maxConnections;i++{
min := bufferSize * i
max := bufferSize * (i+1)
if i==maxConnections-1{
max+=diff // Check to see if we have any leftover data to retrieve for the last request
}
req , _ := http.NewRequest("GET" , url, nil)
req.Header.Add("Range" ,fmt.Sprintf("bytes=%d-%d",min,max))
res , e := http.DefaultClient.Do(req)
if e != nil{
return body , e
}
log.Printf("Index:%d . Range:bytes=%d-%d",i,min,max)
data , e :=ioutil.ReadAll(res.Body)
res.Body.Close()
if e != nil{
return body,e
}
log.Println("Data for request: ",len(data))
read = read + len(data)
body = append(body, data...)
}
log.Println("File size:",bodySize , "Downloaded size:",len(body)," Actual read:",read)
return body, nil
}
Also I noticed that the bigger the limit I set the more the difference between the original file content length and the actual size of all request bodies combined is.
Here is my main.go
func main() {
imgUrl := "https://media.wired.com/photos/5a593a7ff11e325008172bc2/16:9/w_2400,h_1350,c_limit/pulsar-831502910.jpg"
maxConnections := 4
manager := lib.NewManager(maxConnections)
data , e:= manager.DownloadBody(imgUrl)
if e!= nil{
log.Fatalln(e)
}
ioutil.WriteFile("foo.jpg" , data,0777)
}
Note: for the time being I am not interested in making the code concurrent.
Any ideas what I could be missing?
Note: I have confirmed that server returns a 206 partial content using the curl command below:
curl -I https://media.wired.com/photos/5a593a7ff11e325008172bc2/16:9/w_2400,h_1350,c_limit/pulsar-831502910.jpg
Thanks to #mh-cbon I managed to write a simple test that helped me find the solution . Here is the fixed code
for i:=0;i<maxConnections;i++{
min := bufferSize * i
if i != 0{
min++
}
max := bufferSize * (i+1)
if i==maxConnections-1{
max+=diff // Check to see if we have any leftover data to retrieve for the last request
}
req , _ := http.NewRequest("GET" , url, nil)
req.Header.Add("Range" ,fmt.Sprintf("bytes=%d-%d",min,max))
res , e := http.DefaultClient.Do(req)
if e != nil{
return body , e
}
log.Printf("Index:%d . Range:bytes=%d-%d",i,min,max)
data , e :=ioutil.ReadAll(res.Body)
res.Body.Close()
if e != nil{
return body,e
}
log.Println("Data for request: ",len(data))
read = read + len(data)
body = append(body, data...)
}
The problem was that I didn't have a correct min value to begin with . So lets say I have the following ranges to download :
0-100
101 - 200
My code would download bytes from 0-100 and then again from 100-200 instead of 101-200
So I made sure on every iteration (except the first one) to increment the min by 1 so as not to overlap with the previous range
Here is a simple test I managed to fix from the docs provided as comments:
func TestManager_DownloadBody(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
http.ServeContent(writer,request ,"hey" ,time.Now() ,bytes.NewReader([]byte(`hello world!!!!`)))
}))
defer ts.Close()
m := NewManager(4)
data , err := m.DownloadBody(ts.URL)
if err != nil{
t.Errorf("%s",err)
}
if string(data) != "hello world!!!!"{
t.Errorf("Expected hello world!!!! . received : [%s]",data)
}
}
Sure there are more tests to be written but it is a good start
Further to this question, I want to parse a date/time passed on the command line to a Go program. At the moment, I use the flag package to populate a string variable ts and then the following code:
if ts == "" {
config.Until = time.Now()
} else {
const layout = "2006-01-02T15:04:05"
if config.Until, err = time.Parse(layout, ts); err != nil {
log.Errorf("Could not parse %s as a time string: %s. Using current date/time instead.", ts, err.Error())
config.Until = time.Now()
}
}
This works OK, provided the user passes exactly the right format - e.g. 2019-05-20T09:07:33 or some such.
However, what I would like, if possible, is the flexibility to pass e.g. 2019-05-20T09:07 or 2019-05-20T09 or maybe even 2019-05-20 and have the hours, minutes and seconds default to 0 where appropriate.
Is there a sane1 way to do this?
1 not requiring me to essentially write my own parser
UPDATE
I've kind of got a solution to this, although it's not particularly elegant, it does appear to work for most of the cases I am likely to encounter.
package main
import (
"fmt"
"time"
)
func main() {
const layout = "2006-01-02T15:04:05"
var l string
var input string
for _, input = range []string{"2019-05-30", "2019-05-30T16", "2019-05-30T16:04", "2019-05-30T16:04:34",
"This won't work", "This is extravagantly long and won't work either"} {
if len(input) < len(layout) {
l = layout[:len(input)]
} else {
l = layout
}
if d, err := time.Parse(l, input); err != nil {
fmt.Printf("Error %s\n", err.Error())
} else {
fmt.Printf("Layout %-20s gives time %v\n", l, d)
}
}
}
Just try each format, until one works. If none work, return an error.
var formats = []string{"2006-01-02T15:04:05", "2006-01-02", ...}
func parseTime(input string) (time.Time, error) {
for _, format := range formats {
t, err := time.Parse(format, input)
if err == nil {
return t, nil
}
}
return time.Time{}, errors.New("Unrecognized time format")
}
I think this library is what you are looking for https://github.com/araddon/dateparse
Parse many date strings without knowing the format in advance. Uses a scanner to read bytes and use a state machine to find format.
t, err := dateparse.ParseAny("3/1/2014")
In the specific scenario you describe, you could check the length of the input datestamp string, and add the proper length of zero stuff at the end of it to correspond to your layout. So basically you could append as much of the string "T00:00:00" (counting from the end), to the input as is missing in length compared to the layout format string.
I have a trouble for gonna get a value from map stored in struct. Please, look at the next part of code (some strings skipped):
type Settings struct {
...
LcInfoData *[]LcInfodb
LcInfoLog *MapLcInfoLL
}
type MapLcInfoLL map[string]LcInfoLL
type LcInfoLL struct {
EnableLog string
FileLogPtr *os.File
}
...
func updLogInfo(cnf *Settings)(err) {
for _, t := range *cnf.LcInfoData {
fpPtr, err := logInit(t.FilepLog);
if err != nil {
exitMsg(1, err.Error());
}
lcMapVal := LcInfoLL{EnableLog: t.EnableLog, FileLogPtr: fpPtr}
lcMap[t.LocationID] = lcMapVal
}
cnf.uLcInfoLog(&lcMap) // at the end
...
}
At the end I got filled structure for using in another function (it's global settings). But. I can't get an access to elements inside a map (which stored in structure). I mean something like that:
v := *cnf.LcInfoLog["index"]
log.Println("ABOUT LOCATION: ", v.FileLogPtr)
Can you help me?
Thank you!
TL;DR Somehow, I am appending a pointer to a list instead of the object within a for loop of objects so at the end the entire slice is composed of the same object multiple times. I just don't know how to fix that.
The Long Way
I am still having a super hard time trying to figure out pointers in go. I posted a question yesterday and got some help but now I am stuck on a slightly different issue in the same piece of code.
I am working with gocql and cqlr go packages to try and bit a small object mapper for my Cassandra queries. Essentially the problem I am having is I am appending what appears to be a pointer to an object, not a new instance of the obj to the array. How do I fix that? I have tried adding & and * in front of value but that doesn't seem to work. How do I fix these? The bind function needs an & according to their docs.
Code
type Query struct {
query string
values interface{}
attempts int
maxAttempts int
structType reflect.Type
}
func (query Query) RetryingQuery() (results []interface{}) {
var q *gocql.Query
if query.values != nil {
q = c.Session.Query(query.query, query.values)
} else {
q = c.Session.Query(query.query)
}
bindQuery := cqlr.BindQuery(q)
value := reflect.New(query.structType).Interface()
for bindQuery.Scan(value) {
fmt.Println(value)
results = append(results, value)
}
return
}
The docs ask for var value type then in bind you would pass &value. I quoted the docs below.
var t Tweet
var s []Tweet
for b.Scan(&t) {
// Application specific code goes here
append(s, t)
}
The issue is I cannot directly go var value query.structType to define its type then pass the reference of that to bindQuery.Scan().
What is printed
&{result1 x86_64 24 3.2.0-74-generic Linux}
&{result2 x86_64 24 3.19.0-25-generic Linux}
&{result3 x86_64 4 3.13.0-48-generic Linux}
&{result4 x86_64 2 3.13.0-62-generic Linux}
&{result5 x86_64 4 3.13.0-48-generic Linux}
What is in the slice
Spoiler, it is result5 repeated over and over. I understand that I am just appending the pointer to same object to the list and that every loop iteration the object is changed and that changes all the results in the slice to that new object. I just don't know how to fix it.
[{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"},{"hostname":"result5","machine":"x86_64","num_cpus":4,"release":"3.13.0-48-generic","sysname":"Linux"}]
Well I can at least tell you what you're doing. bindQuery takes a pointer. It changes the value stored at the address.
What you're essentially doing is this:
package main
import "fmt"
func main() {
var q int
myInts := make([]*int, 0, 5)
for i := 0; i < 5; i++ {
q = i
fmt.Printf("%d ", q)
myInts = append(myInts, &q)
}
fmt.Printf("\n")
for _, value := range myInts {
fmt.Printf("%d ", *value)
}
fmt.Printf("\n")
fmt.Println(myInts)
}
Which, as you can probably guess, gives you this:
0 1 2 3 4
4 4 4 4 4
[0x104382e0 0x104382e0 0x104382e0 0x104382e0 0x104382e0]
Things get a little more confusing with reflect. You can get your type as an interface, but that is it (unless you want to play with unsafe). An interface, in simple terms, contains a pointer to the original type underneath (and some other stuff). So in your function you are passing a pointer (and some other stuff). Then you're appending the pointer. It might be nice just to get concrete and type switch your interface. I assume you know what types it could be. In which case you'd have to have something along these lines:
package main
import (
"fmt"
"reflect"
)
type foo struct {
fooval string
}
type bar struct {
barval string
}
func main() {
f1 := foo{"hi"}
f2 := &foo{"hi"}
b1 := bar{"bye"}
b2 := &bar{"bye"}
doSomething(f1)
doSomething(f2)
doSomething(b1)
doSomething(b2)
}
func doSomething(i interface{}) {
n := reflect.TypeOf(i)
// get a new one
newn := reflect.New(n).Interface()
// find out what we got and handle each case
switch t := newn.(type) {
case **foo:
*t = &foo{"hi!"}
fmt.Printf("It was a **foo, here is the address %p and here is the value %v\n", *t, **t)
case **bar:
*t = &bar{"bye :("}
fmt.Printf("It was a **bar, here is the address %p and here is the value %v\n", *t, **t)
case *foo:
t = &foo{"hey!"}
fmt.Printf("It was a *foo, here is the address %p and here is the value %v\n", t, *t)
case *bar:
t = &bar{"ahh!"}
fmt.Printf("It was a *bar, here is the address %p and here is the value %v\n", t, *t)
default:
panic("AHHHH")
}
}
You could also just keep calling value = reflect.New(query.structType).Interface() inside of the loop which will give you new interfaces every time. Reassigning value after every append. Last time through the loop would make one extra though..
Under what circumstances could this code:
v, ok := value.(int64)
if !ok {
panic("NOPE "+reflect.TypeOf(value).Kind().String())
} else {
fmt.Printf("VAL: %d\n",v)
}
produce the panic with message panic: NOPE int64?
Is this a bug or is there something basic about the numeric types that I'm missing?
This can happen if you're using type declaration on numeric types. If you do something like this:
type T int64
...
var value interface{} = T(1)
and put it into your code, you'll get the exact same error. But if you don't check the kind, but the type, you'll see what's happening here:
v, ok := value.(int64)
if !ok {
panic("NOPE " + reflect.TypeOf(value).String())
} else {
fmt.Printf("VAL: %d\n", v)
}
produces the message:
panic: NOPE main.T
The kind of T is int64, but value is not int64.