Send async response and don't get zombie pids - http

Question

I'm aiming for curl client sends http req to Go API and Go API does (1. run a background shell, 2. returns response to client instantly but 3. keeps on running the point 1 server-side command in the background). Problem is point 2 is not being returned to client instantly, client is only getting a response after point 3 finishes
I tried:
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"os/exec"
"log"
"strings"
"flag"
"strconv"
"crypto/tls"
"crypto/x509"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
"github.com/go-ldap/ldap"
"regexp"
"errors"
"encoding/base64"
"time"
)
func insert(w http.ResponseWriter, r *http.Request) (error) {
fullcmd := fmt.Sprintf("/home/ec2-user/spark_home/bin/spark-submit %s", "dgs")
cmd := exec.Command("/bin/sh", "-c", fullcmd)
err4 := cmd.Start()
if err4 != nil {
e1 := fmt.Sprintf("Error")
l.Printf(e1)
http.Error(w, e1, http.StatusInternalServerError)
return err4
} else {
l.Printf("The data is being ingested asynchronously in the background \n")
fmt.Fprintf(w, "request received. The data is being ingested asynchronously in the background \n")
w.Header().Set("Content-Type", "text/plain")
w.Write([]byte(fmt.Sprintf("request received. The data is being ingested asynchronously in the background \n")))
}
//wait for the spark command to finish, need to Wait() otherwise zombie/orphan pid is created
cmd.Wait()
//do bunch of other commands here that take 30+ seconds
l.Printf("success")
return nil
}
r := mux.NewRouter()
r.HandleFunc("/test", insert).Methods(http.MethodPost)
http.Handle("/", r)
server := &http.Server{
Addr: ":" + strconv.Itoa(*port),
Handler: handlers.LoggingHandler(os.Stdout, http.DefaultServeMux),
TLSConfig: tlsConfig,
}
server.ListenAndServeTLS(TLS_SERVER_CERTFILE, TLS_SERVER_KEYFILE)

Answer 1

The response will be completed when the HTTP handler returns, so if you want to start a job that will continue, you have to do that in a separate goroutine. You can start the goroutine as soon as the shell process starts, using something like this:
func insert(w http.ResponseWriter, r *http.Request) (error) {
...
err4 := cmd.Start()
if err4 != nil {
...
}
w.Header().Set("Content-Type", "text/plain")
fmt.Fprintf(w, "request received. The data is being ingested asynchronously in the background \n")
go func() {
cmd.Wait()
// Do other stuff
}()
return nil
}