Diagnosing root cause long HTTP response turnaround in Golang - http

Question

So my HTTP client initialisation and send request code looks like this.
package http_util
import (
"crypto/tls"
"net/http"
"time"
)
var httpClient *http.Client
func Init() {
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
MaxIdleConnsPerHost: 200,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
}
httpClient = &http.Client{Transport: tr, Timeout: 30 * time.Second}
}
func SendRequest(ctx context.Context, request *http.Request) (*SomeRespStruct, error) {
httpResponse, err := httpClient.Do(request)
if err != nil {
return nil, err
}
responseBody, err := ioutil.ReadAll(httpResponse.Body)
defer httpResponse.Body.Close()
if err != nil {
return nil, err
}
response := &SomeRespStruct{}
err = json.Unmarshal(responseBody, response)
if err != nil {
return nil, err
}
return response, nil
}
When I launch my server, I call http_util.Init().
The issue arises when I receive multiple requests (20+) at once to call this external server. In one of my functions I do
package external_api
import (
"context"
"log"
)
func SomeAPICall(ctx context.Context) (SomeRespStruct, error) {
// Build request
request := buildHTTPRequest(...)
log.Printf("Send request: %v", request)
response, err := http_util.SendRequest(ctx, request)
// Error checks
if err != nil {
log.Printf("HTTP request timed out: %v", err)
return nil, err
}
log.Printf("Received response: %v", response)
return response, nil
}
My issue is that I get a 15~20s lag in between the Send request and Received response logs based on the output timestamp when there is high request volume. Upon checking with the server that's handling my requests, I found out that on their end, processing time from end-to-end takes less than a second (the same exact request that had a long turnaround time according to my own logs), so I'm not too sure what is the root cause of this high turnaround time. I also did a traceroute and a ping to the server as well and there was no delay, so this should not be a network error.
I've looked around and it seems like the suggested solutions are:
to increase the MaxIdleConnsPerHost
to read the HTTP response body in full and close it
Both of which I have already done.
I'm not sure if there is more tuning to be done regarding the configuration of my HTTP client to resolve this issue, or if I should investigate other workarounds, for instance retry or perhaps scaling (but my CPU and memory utilisation are at the 2-3% range).