I built a small testcase, that checks that code on my end timesout after a set amount of time has passed. But this isnt working as expected
I am hitting an server sided endpoint which works fine, but what happens if it is slower than usual, I will need code on my end to timeout, this has been implemented but I need to test that I implemented it correctly.
This is what I have so far
func TestTimeout(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(time.Second * 15)
}))
defer ts.Close()
client := &http.Client{
Timeout: time.Second * 10,
}
myRequest,err := createMyRequest(Somedata,SomeMoreData)
res, err := client.Do(myRequest)
if err != nil {
t.Fatal(err)
}
res.Body.Close()}
However my code successfully runs without giving a timeout error (as in I am not waiting for 10 seconds, where am I going wrong?
As already pointed out in some of the comments, you need to make a request to the test server's endpoint (using ts.URL) as follows:
myRequest, err := http.NewRequest(http.MethodPost, ts.URL, bytes.NewBuffer([]byte("test")))
if err != nil {
t.Fatal(err)
}
res, err := client.Do(myRequest)
if err != nil {
t.Fatal(err)
}
[...]
Related
Hi I have this following GoLang code snippet.
func executeAuthorisationRequest(request http.Request) (*AuthorisationResponse, error) {
var response AuthResponse
client := &http.Client{
Timeout: time.Second * 10
}
requestResult, requestError := client.Do(&request)
if requestError != nil {
log.Error(fmt.Sprintf("Some error %s", request.Error()))
}
}
The request is created here
func creatRequest(url string, body url.Values) (*http.Request, error){
req,reqError := http.NewRequest(http.MethodPost,url,strings.NewReader(body.Encode()))
if reqError != nil {
//Error handle
}
req.Header.Add("Content-Type","some business logic")
return request,nil
}
I am trying to create a testcase that my client side code will timeout after 10 seconds if the server sided API is taking too long, how do I simulate/create a testcase like this
I do not have access to the server sided code
Any guidance will be highly appreciate please point me in right direction.
You need lib httptest
example:
func TestTimeout(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(time.Second * 15)
}))
defer ts.Close()
client := &http.Client{
Timeout: time.Second * 10,
}
res, err := client.Get(ts.URL)
if err != nil {
t.Fatal(err)
}
res.Body.Close()}
I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.
Here is the schema :
Client sends a POST request to server A
server A process this and sends a GET to server B
server B sends a response through A to the client
I though the best idea was to make a pipe which would read the response of the GET, and write into the response of the POST, but I got many types problems.
func main() {
r := mux.NewRouter()
r.HandleFunc("/test/{hash}", testHandler)
log.Fatal(http.ListenAndServe(":9095", r))
}
func handleErr(err error) {
if err != nil {
log.Fatalf("%s\n", err)
}
}
func testHandler(w http.ResponseWriter, r *http.Request){
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
write, _ = http.Get("http://localhost:9090/test/" + hash)
defer write.Close()
}()
w.Write(read)
}
When I run this I get the following error:
./ReverseProxy.go:61: cannot use read (type *io.PipeReader) as type []byte in argument to w.Write
Is there a way, to properly insert a io.PipeReader format into an http response?
Or am I doing this in a totally wrong way?
You are not actually writing to it, you're replacing the pipe's write.
Something along the lines of:
func testHandler(w http.ResponseWriter, r *http.Request) {
fmt.Println("FIRST REQUEST RECEIVED")
vars := mux.Vars(r)
hash := vars["hash"]
read, write := io.Pipe()
// writing without a reader will deadlock so write in a goroutine
go func() {
defer write.Close()
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
return
}
defer resp.Body.Close()
io.Copy(write, resp.Body)
}()
io.Copy(w, read)
}
Although, I agree with #JimB, for this instance, the pipe isn't even needed, something like this should be more efficient:
func testHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
hash := vars["hash"]
resp, err := http.Get("http://localhost:9090/test/" + hash)
if err != nil {
// handle error
return
}
defer resp.Body.Close()
io.Copy(w, resp.Body)
}
Is there any other better way to ping websites and check if the website is available or not?
I just need to get the status code not get(download) all websites...
func Ping(domain string) int {
timeout := time.Duration(2 * time.Second)
dialTimeout := func(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
url := "http://" + domain
req, _ := http.NewRequest("GET", url, nil)
resp, _ := client.Do(req)
return resp.StatusCode
}
This function is too slow and when I run with goroutines, it goes over the limits and gives me the errors...
Thanks!
Use a single transport. Because the transport maintains a pool of connections, you should not create and ignore transports willy nilly.
Close the response body as described at the beginning of the net/http doc.
Use HEAD if you are only interested in the status.
Check errors.
Code:
var client = http.Client{
Transport: &http.Transport{
Dial: net.Dialer{Timeout: 2 * time.Second}.Dial,
},
}
func Ping(domain string) (int, error) {
url := "http://" + domain
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return 0, err
}
resp, err := client.Do(req)
if err != nil {
return 0, err
}
resp.Body.Close()
return resp.StatusCode, nil
}
Since this is the top result on Google for Pinging in Go, just know there have been several packages written for this purpose, but if you plan to use this answer, I had to make some changes for this to work.
import (
"time"
"net/http"
)
var client = http.Client{
Timeout: 2 * time.Second,
}
But otherwise keeping the same with the accepted answer.
But I'm a beginner in Go so there may be a better way to do this.
I know the usual method of specifying a timeout with HTTP requests by doing:
httpClient := http.Client{
Timeout: time.Duration(5 * time.Second),
}
However, I can't seem to figure out how to do the same when tracing HTTP requests. Here is the piece of code I am working with:
func timeGet(url string) (httpTimingBreakDown, error) {
req, _ := http.NewRequest("GET", url, nil)
var start, connect, dns, tlsHandshake time.Time
var timingData httpTimingBreakDown
timingData.url = url
trace := &httptrace.ClientTrace{
TLSHandshakeStart: func() { tlsHandshake = time.Now() },
TLSHandshakeDone: func(cs tls.ConnectionState, err error) { timingData.tls = time.Since(tlsHandshake) },
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
start = time.Now()
http.DefaultTransport.(*http.Transport).ResponseHeaderTimeout = time.Second * 10 // hacky way, worked earlier but don't work anymore
if _, err := http.DefaultTransport.RoundTrip(req); err != nil {
fmt.Println(err)
return timingData, err
}
timingData.total = time.Since(start)
return timingData, nil
}
I am firing this function inside a goroutine. My sample data set is 100 urls. All goroutines fire, but eventually the program ends in 30+ secs as if the timeout is 30secs.
Earlier I made the same to work by using the hacky way of changing the default inside of it to 10 secs and anything that took too long, timed out and the program ended at 10.xxx secs but now its taking 30.xx secs.
What would be a proper way of specifying a timeout in this scenario?
I know the usual method of specifying a timeout with HTTP requests by doing:
httpClient := http.Client{
Timeout: time.Duration(5 * time.Second),
}
Actually, the preferred method is to use a context.Context on the request. The method you've used is just a short-cut suitable for simple use cases.
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
ctx, cancel := context.WithTimeout(context.Background(), 5 * time.Second)
defer cancel()
req = req.WithContext(ctx)
And this method should work nicely for your situation as well.