Golang Chan is hanging when looping through the results - asynchronous

I am pretty new to GoLang channels, but it isn't working as I would expect.
I have a function that I want to call 3 separate goroutines on and then wait for them all to complete. If I get an error then I am trying to put it on the CHAN and then handle the error after wg.Wait() completes.
Unfortunately, when looping through the CHAN results it will hang. I hypothesize as it is still waiting for the CHAN to get filled but not all goroutines will throw errors.
What is the best way to handle looping through channels that won't always be populated? I also know that I don't have to use CHANs here, but I wanted to make sure that I understood them.
Below is my code.
func createWorkoutPlanForUserPreconditionCheck(planID, userID, transactionID *string) (*sharedstructs.Plan, *sharedstructs.User, *sharedstructs.Profile, error) {
if planID == nil || userID == nil || transactionID == nil {
return nil, nil, nil, sharedstructs.InvalidData{Msg: "Cannot pass in Nil Parameters"}
}
plan := sharedstructs.Plan{}
user := sharedstructs.User{}
profile := sharedstructs.Profile{}
//myError := sharedstructs.InvalidData{}
ch := make(chan sharedstructs.InvalidData, 3)
var wg sync.WaitGroup
wg.Add(3)
//Get the Plan from the Plan ID
go func() {
defer wg.Done()
returnedPlan, readError := readPlan(*planID)
if readError != nil || returnedPlan == nil {
ch <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
plan = *returnedPlan
}
}()
//Get the User
go func() {
defer wg.Done()
returnedUser, getUserError := userdomain.GetUserByID(*userID, *transactionID)
if getUserError != nil || &returnedUser == nil {
ch <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
user = returnedUser
}
}()
//Get the Profile
go func() {
defer wg.Done()
readProfile, getProfileError := profiledomain.GetProfile(*userID, *transactionID)
if getProfileError != nil || readProfile == nil {
ch <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
profile = *readProfile
}
}()
wg.Wait()
////"Hangs Here" - PUT MY ERROR HANDLING LOGIC HERE
for err := range ch {
fmt.Println(err.Error())
}
return &plan, &user, &profile, nil
}

So, not long after posting I figured a solution. My problem was really two fold:
Closing my channel so that it knew when to stop listening
Using inconsistent methods to get my data from my goroutine to my calling function. For errors I was using chan's but for custom structs I was just setting it. I generisized my chan to an interface{} and then did a type switch on it when processing to determine the type of struct that it was.
^^^ fixing these issues made my code work but here is what my code ended up as..
func createWorkoutPlanForUserPreconditionCheck(planID, userID, transactionID *string) (*sharedstructs.Plan, *sharedstructs.User, *sharedstructs.Profile, error) {
if planID == nil || userID == nil || transactionID == nil {
return nil, nil, nil, sharedstructs.InvalidData{Msg: "Cannot pass in Nil Parameters"}
}
outputChannel := make(chan interface{}, 3)
var wg sync.WaitGroup
wg.Add(3)
//Get the Plan from the Plan ID
go func() {
defer wg.Done()
returnedPlan, readError := readPlan(*planID)
if readError != nil || returnedPlan == nil {
outputChannel <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
outputChannel <- *returnedPlan
}
}()
//Get the User
go func() {
defer wg.Done()
returnedUser, getUserError := userdomain.GetUserByID(*userID, *transactionID)
if getUserError != nil || &returnedUser == nil {
outputChannel <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
outputChannel <- returnedUser
}
}()
//Get the Profile
go func() {
defer wg.Done()
readProfile, getProfileError := profiledomain.GetProfile(*userID, *transactionID)
if getProfileError != nil || readProfile == nil {
outputChannel <- sharedstructs.InvalidData{Msg: "Could Not Retreive the User with ID: " + *userID}
} else {
outputChannel <- *readProfile
}
}()
wg.Wait()
close(outputChannel)
plan := sharedstructs.Plan{}
user := sharedstructs.User{}
profile := sharedstructs.Profile{}
for result := range outputChannel {
switch result.(type) {
case sharedstructs.InvalidData:
return nil, nil, nil, result.(sharedstructs.InvalidData)
case sharedstructs.Plan:
plan = result.(sharedstructs.Plan)
case sharedstructs.User:
user = result.(sharedstructs.User)
case sharedstructs.Profile:
profile = result.(sharedstructs.Profile)
}
}
return &plan, &user, &profile, nil
}

Related

When writing an http handler, do we have to listen for request context cancellation?

Supposed that I'm writing an http handler, that do something else before returning a response, do I have to setup a listener to check wether the http request context has been canceled? so that it can return immediately, or is there any other way to exit the handler when the request context cancelled?
func handleSomething(w http.ResponseWriter, r *http.Request) {
done := make(chan error)
go func() {
if err := doSomething(r.Context()); err != nil {
done <- err
return
}
done <- nil
}()
select {
case <-r.Context().Done():
http.Error(w, r.Context().Err().Error(), http.StatusInternalServerError)
return
case err := <-done:
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
}
}
func doSomething(ctx context.Context) error {
// simulate doing something for 1 second.
time.Sleep(time.Second)
return nil
}
I tried making a test for it, but after the context got cancelled, doSomething function didn't stop and still running in the background.
func TestHandler(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/something", handleSomething)
srv := http.Server{
Addr: ":8989",
Handler: mux,
}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
if err := srv.ListenAndServe(); err != nil {
log.Println(err)
}
}()
time.Sleep(time.Second)
req, err := http.NewRequest(http.MethodGet, "http://localhost:8989/something", nil)
if err != nil {
t.Fatal(err)
}
cl := http.Client{
Timeout: 3 * time.Second,
}
res, err := cl.Do(req)
if err != nil {
t.Logf("error: %s", err.Error())
} else {
t.Logf("request is done with status code %d", res.StatusCode)
}
go func() {
<-time.After(10 * time.Second)
shutdown, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
srv.Shutdown(shutdown)
}()
wg.Wait()
}
func handleSomething(w http.ResponseWriter, r *http.Request) {
done := make(chan error)
go func() {
if err := doSomething(r.Context()); err != nil {
log.Println(err)
done <- err
}
done <- nil
}()
select {
case <-r.Context().Done():
log.Println("context is done!")
return
case err := <-done:
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
}
}
func doSomething(ctx context.Context) error {
return runInContext(ctx, func() {
log.Println("doing something")
defer log.Println("done doing something")
time.Sleep(10 * time.Second)
})
}
func runInContext(ctx context.Context, fn func()) error {
ch := make(chan struct{})
go func() {
defer close(ch)
fn()
}()
select {
case <-ctx.Done():
return ctx.Err()
case <-ch:
return nil
}
}
I've just refactored the solution provided a little bit and now it should work. Let me guide you through the relevant changes.
The doSomething function
func doSomething(ctx context.Context) error {
fmt.Printf("%v - doSomething: start\n", time.Now())
select {
case <-ctx.Done():
fmt.Printf("%v - doSomething: cancelled\n", time.Now())
return ctx.Err()
case <-time.After(3 * time.Second):
fmt.Printf("%v - doSomething: processed\n", time.Now())
return nil
}
}
It waits for a cancellation input or after a delay of 3 seconds it returns to the caller. It accepts a context to listen for.
The handleSomething function
func handleSomething(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
fmt.Printf("%v - handleRequestCtx: start\n", time.Now())
done := make(chan error)
go func() {
if err := doSomething(ctx); err != nil {
fmt.Printf("%v - handleRequestCtx: error %v\n", time.Now(), err)
done <- err
}
done <- nil
}()
select {
case <-ctx.Done():
fmt.Printf("%v - handleRequestCtx: cancelled\n", time.Now())
return
case err := <-done:
if err != nil {
fmt.Printf("%v - handleRequestCtx: error: %v\n", time.Now(), err)
w.WriteHeader(http.StatusInternalServerError)
return
}
fmt.Printf("%v - handleRequestCtx: processed\n", time.Now())
}
}
Here, the logic is very similar to yours. In the select, we check whether the received error is nil or not, and based on this we return to the proper HTTP status code to the caller. If we receive a cancellation input, we cancel all the context chain.
The TestHandler function
func TestHandler(t *testing.T) {
r := mux.NewRouter()
r.HandleFunc("/demo", handleSomething)
srv := http.Server{
Addr: ":8000",
Handler: r,
}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
if err := srv.ListenAndServe(); err != nil {
fmt.Println(err.Error())
}
}()
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 1*time.Second) // request canceled
// ctx, cancel := context.WithTimeout(ctx, 5*time.Second) // request processed
defer cancel()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "http://localhost:8000/demo", nil)
client := http.Client{}
res, err := client.Do(req)
if err != nil {
fmt.Println(err.Error())
} else {
fmt.Printf("res status code: %d\n", res.StatusCode)
}
srv.Shutdown(ctx)
wg.Wait()
}
Here, we spin up an HTTP server and issue an HTTP request to it through an http.Client. You can see that there are two statements to set the context timeout. If you use the one with the comment // request canceled, everything will be canceled, otherwise, if you use the other the request will be processed.
I Hope that this clarifies your question!

How should I handle the errors when I try to write in DB

Not sure how should I deal with errors when I try to write in DB in this particular case:
So I use this func to insert in DB
func SaveToDB(dateid string, content string) {
db, err := sql.Open("mysql", dbLink)
if err != nil {
log.Fatal(err)
}
queryString := fmt.Sprintf("INSERT INTO balances (dateid, content) VALUES('%v','%v');", dateid, content)
rows, err := db.Query(queryString)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
}
But I would like to don't stop the server when error is duplicate so I tried this version:
func SaveToDB(dateid string, content string) {
db, err := sql.Open("mysql", dbLink)
if err != nil {
log.Fatal(err)
}
queryString := fmt.Sprintf("INSERT INTO balances (dateid, content) VALUES('%v','%v');", dateid, content)
rows, err1 := db.Query(queryString)
if err1 != nil {
thisErr, err2 := regexp.MatchString("Error 1062: Duplicate entry", err.Error())
if err2 != nil {
log.Fatal("ERROR: error occured while trying to perform regex on SaveToDB", err2)
}
if thisErr == true {
log.Println("ERROR: Not able to save in DB due to ducplicate: ", err1)
}else{log.Fatal("ERROR: error occured when trying to save to DB: ", err1)}
}
defer rows.Close()
}
But in this situation I receive panic. So how I can stop this function from executing before it reaches "defer rows.Close()"? I guess that is the reason for panic...
Don't use log.Fatal and install deferred code once you have the correct data:
rows, err := db.Query(queryString)
if err != nil {
log.Error(err)
return
}
defer rows.Close()

Server returning 400 when user exists

I'm working on some tests in Go and I have spent the past 2 days trying to make it work but I couldn't. My problem is that the test returns 400 even when the user does exist.
This is my getUser function
func (handler *UserHandler) getUser(w http.ResponseWriter, ID int) {
logfile, err := os.OpenFile("events.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Fatalf("Error opening file: %v", err)
}
defer logfile.Close()
log.SetOutput(logfile)
user := db.Fetch(ID)
userJSON, err := json.Marshal(user)
if err != nil {
log.Printf("Error while marshaling the user into JSON: %v", err)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
// userJSON is sent as http Response
w.Write(userJSON)
}
This is my UserHandler
type UserHandler struct{}
func (handle *UserHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
var head string
head, r.URL.Path = ShiftPath(r.URL.Path)
id, err := strconv.Atoi(head)
if err != nil {
http.Error(w, fmt.Sprintf("Invalid user ID %q", head), http.StatusBadRequest)
return
}
switch r.Method {
case "GET":
handle.getUser(w, id)
default:
http.Error(w, "Only GET is allowed", http.StatusMethodNotAllowed)
}
}
func ShiftPath(p string) (head, tail string) {
p = path.Clean("/" + p)
i := strings.Index(p[1:], "/") + 1
if i <= 0 {
return p[1:], "/"
}
return p[1:i], p[i:]
}
And this is my test
func TestGetUser(t *testing.T) {
handler := new(UserHandler)
mux := http.NewServeMux()
mux.HandleFunc("/user/", handler.ServeHTTP)
writer := httptest.NewRecorder()
request, _ := http.NewRequest("GET", "/user/12", nil)
mux.ServeHTTP(writer, request)
if writer.Code != 200 {
t.Errorf("Response code is %v", writer.Code)
}
}
Issue with code ====> id, err := strconv.Atoi(head)
Due to error you see a return and hence you see 400 error.
Have your server code fully functional with valid logic.
Suggestion: Always print or debug line by line. You can find the issue and root cause.

Recursion in golang is giving deadlock or negative WaitGroup counter when using goroutines, channels and sync.Waitgroup

I am trying to find the list of all directories using a recursive function. The code to the function is
func FindDirs(dir string, nativePartitions []int64, wg *sync.WaitGroup, dirlistchan chan string) {
// defer wg.Done here will give negative waitgroup panic, commenting it will give negative waitgroup counter panic
fd, err := os.Open(dir)
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir(){
dirlistchan <- buff.String()
FindDirs(buff.String(), nativePartitions, wg, dirlistchan) //recursion happens here
} else {
//fmt.Println(i.Name(), "is not native")
}
}
}
and in the main function, I am calling it as
wg := new(sync.WaitGroup)
dirlistchan := make(chan string, 1000)
wg.Add(1)
go func() {
filtermounts.FindDirs(parsedConfig.ScanFrom, []int64{filtermounts.EXT4_SUPER_MAGIC}, wg, dirlistchan)
}()
go func() {
wg.Wait()
close(dirlistchan)
}()
for i := range dirlistchan {
fmt.Println(i)
}
wg.Wait()
and I am getting a
fatal error: all goroutines are asleep - deadlock!
I was able to get this working if I am printing the result instead of using channels, or append to a slice using mutex. (verified with the linux find command to see if the results are same.) Please find the function after omitting channels and using sync.Mutex and append.
func FindDirs(dir string, nativePartitions []int64, dirlist *[]string, mutex *sync.Mutex) []string{
fd, err := os.Open(dir)
defer fd.Close()
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir(){
//dirlistchan <- buff.String()
mutex.Lock()
*dirlist = append(*dirlist, buff.String())
mutex.Unlock()
//fmt.Println(buff.String())
FindDirs(buff.String(), nativePartitions, dirlist, mutex)
} else {
//fmt.Println(i.Name(), "is not native")
}
}
return *dirlist
}
But I cannot think of a way to make this work with channels and goroutines. Any help is greatly appreciated.
Note: Here is a link to the golang playground with the code. I couldn't find a workaround to get the syscall thing to work on the playground either. It works on my system though.
Thanks.
Short answer : You are not closing the channel.
Fix : add defer wg.Done() at beginning of the go routine that calls FindDirs
go func() {
defer wg.Done()
filtermounts.FindDirs(parsedConfig.ScanFrom, []int64{filtermounts.EXT4_SUPER_MAGIC}, wg, dirlistchan)
}()
Why did it happen
The go routine that is responsponsible for closing the channel waits for wg there is no wg.Done in the code above. So close never happens
Now the for loop blocks on the channel for close or a value for ever, this cause the error
fatal error: all goroutines are asleep - deadlock!
So here is your code ,this may be run as
go run filename.go /path/to/folder
Code
package main
import (
"bytes"
"fmt"
"os"
"sync"
"syscall"
)
func main() {
wg := new(sync.WaitGroup)
dirlistchan := make(chan string, 1000)
wg.Add(1)
go func() {
defer wg.Done()
FindDirs(os.Args[1], []int64{61267}, wg, dirlistchan)
}()
go func() {
wg.Wait()
close(dirlistchan)
}()
for i := range dirlistchan {
fmt.Println(i)
}
wg.Wait()
}
func FindDirs(dir string, nativePartitions []int64, wg *sync.WaitGroup, dirlistchan chan string) {
fd, err := os.Open(dir)
if err != nil {
panic(err)
}
filenames, err := fd.Readdir(0)
if err != nil {
panic(err)
}
for _, i := range filenames {
var buff bytes.Buffer
buff.WriteString(dir)
switch dir {
case "/":
default:
buff.WriteString("/")
}
buff.WriteString(i.Name())
/*err := os.Chdir(dir)
if err != nil {
return err
}*/
t := new(syscall.Statfs_t)
err = syscall.Statfs(buff.String(), t)
if err != nil {
//fmt.Println("Error accessing", buff.String())
}
if checkDirIsNative(t.Type, nativePartitions) && i.IsDir() {
dirlistchan <- buff.String()
FindDirs(buff.String(), nativePartitions, wg, dirlistchan) //recursion happens here
} else {
//fmt.Println(i.Name(), "is not native")
}
}
}
func checkDirIsNative(dirtype int64, nativetypes []int64) bool {
for _, i := range nativetypes {
if dirtype == i {
return true
}
}
return false
}
Find the go.play link here
As has been stated already you should close the channel if you want the main goroutine to exit.
Example of implementation :
In function func FindDirs you could make an additional channel for every recursive func FindDirs call that this function is going to make and pass that new channel in the argument. Then simultaneously listen to all those new channels and forward the strings back to the channel that function got in the argument.
After all new channels has been closed close the channel given in the argument.
In other words every func call should have its own channel that it sends to. The string is then forwarded all the way to main function.
Dynamic select described here : how to listen to N channels? (dynamic select statement)

Reconnect TCP on EOF in Go

I have the following:
//In an init func
if logStashHost != "" {
lsconn, err = net.Dial("tcp", logStashHost)
}
...
ToLogStash(rec, lsconn)
Then Two functions:
func ReadLogStash(conn net.Conn) {
buffer := make([]byte, 256)
for {
_, err := conn.Read(buffer)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(buffer)
}
}
}
func ToLogStash(r *logrow.Record, conn net.Conn) {
b, err := json.Marshal(r)
if err != nil {
fmt.Println(err)
return
}
_, err = fmt.Fprintln(conn, string(b))
if err != nil {
fmt.Println(err)
}
}
Where ReadLogStash is a running goroutine. If the other side closes, I get EOF. What would be a good implementation in ReadLogStash to have it attempt to reestablish the connection every X seconds when it gets an EOF?
Go has channels for synchronization and communication, use them!
Make your connection in a loop, and have it wait for some sort of message to come back on a channel.
...
errCh := make(chan error)
for {
lsconn, err = net.Dial("tcp", logStashHost)
// check error!
go ReadLogStash(lsconn, errCh)
err = <-errCh
if err != nil {
// bad error
break
}
// sleep to backoff on retries?
}
...
func ReadLogStash(conn net.Conn, errCh chan error) {
_, err := io.Copy(os.Stderr, conn)
if err != nil {
fmt.Println(err)
}
// a nil error from io.Copy means you reached EOF.
errCh <- err
}
Unless you have more functionality in ReadLogStash, you can probably just use io.Copy inline, and forget the entire function, but this pattern may come in useful for you anyway.
Here is what I ended up going with, a channel was the right direction:
if logStashHost != "" {
lsc = make(chan *logrow.Record)
go ToLogStash(lsc, logStashHost)
}
...
if lsc != nil {
lsc <- rec
}
...
func ToLogStash(c chan *logrow.Record, logStashHost string) {
var lsconn net.Conn
var enc *json.Encoder
var err error
connect := func() {
for {
lsconn, err = net.Dial("tcp", logStashHost)
if err == nil {
enc = json.NewEncoder(lsconn)
break
}
log.Println(err)
time.Sleep(time.Second)
}
}
connect()
for r := range c {
err = enc.Encode(r)
if err != nil {
lsconn.Close()
log.Println(err)
connect()
}
}
}

Resources