This question already has answers here:
How to mock http.Client Do method
(4 answers)
Closed 4 months ago.
Hey so I have seen and used this post to help mock my http.Client but when I try to pass the mock request I get the following error: cannot use mockClient (variable of type *MockClient) as *"net/http".Client value in argument to api.callAPI.
In one file I have my actual code:
I created the HTTPClient:
type HTTPClient interface {
Do(req *http.Request) (*http.Response, error)
}
I have the function that passes the HTTPClient as an interface (I can't show all of it because it is for work but here's the important pieces):
func (api *API) callAPI(req *http.Request, client HTTPClient) (utils.ErrorWrapper, bool) {
response, err := client.Do(req)
}
I also have another function that calls the callAPI method. In that function I create client variable right before I call the callAPI function
var Client HTTPClient = &http.Client{}
response, isRetry := api.callAPI(req, Client)
This all works fine. However, in my testing file I get the error as mentioned above. I am using testify for my mocking framework. Here is what I have in my testing file (both the testing file and the actual code are apart of the same package):
set up my mock client and the Do function using testify
type MockClient struct {
mock.Mock
}
func (m *MockClient) Do(req *http.Request) (*http.Response, error) {
args := m.Called()
resp := args.Get(0)
return resp.(*http.Response), args.Error(1)
}
Then create my test:
func TestCallAPI(t *testing.T) {
mockClient := &MockClient{}
recorder := httptest.NewRecorder()
responseCh := make(chan utils.ErrorWrapper)
c, _ := gin.CreateTestContext(recorder)
id:= "unitTest123"
api := NewAPICaller(responseCh, id, c)
var response = Response{
StatusCode: 200,
}
//setup expectations
mockClient.On("Do").Return(response, nil)
req, _ := http.NewRequest("GET", "URL I Can't Show", nil)
wrapper, isRetry := api.callAPI(req, mockClient)
mockClient.AssertExpectations(t)
assert.Equal(t, "placeholder", wrapper)
assert.Equal(t, false, isRetry)
}
I tried to do a similar thing with the mockclient variable the way I did with the Client variable:
var mockclient HTTPClient = &MockClient{}
but I get this error on the HTTPClient: undeclared name: HTTPClient. Unsure why this is happening because they are a part of the same package so I thought it could be exported easily?
There are 2 ways you can achieve this.
As suggested by #Volker. Setup a HTTP Server. you can use a docker container to run it for your tests.
The other options (my favorite for unit testing) is to wrap your HTTP client in an interface of your own. That interface will expose methods either your domain specific OR probably just plain simple GET, POST. Then use the library like mockgen to generate mock. This library generate mocks based on interface. With this setup in place, you can have proper unit tests on the exposed methods.
Related
Is there any way to prevent httputil.ReverseProxy from sending an incoming request to the target server? For example, if I have a cache and I can respond to the client using only local data. Or after validation, I want to return an error to the client.
A httputil.ReverseProxy has a single exported method, ServeHTTP(rw http.ResponseWriter, req *http.Request) which makes it implement the net/http.Handler interface.
So basically at a place you're now using an vanilla httputil.ReverseProxy instance, instead use an instance of your custom type which implements net/http.Handler as well, keeps a pointer to an instance of httputil.ReverseProxy, and either processes the request itself or calls out to that ReverseProxy instance's ServeHTTP.
You should be able to wrap the http.DefaultTransport with a cache that can either use the cache based on the request or fallback on the http.DefaultTransport.
package main
import (
"net/http"
"net/http/httputil"
)
var _ http.RoundTripper = &CachingTransport{}
type CachingTransport struct {
// put your cache here
}
func (c *CachingTransport) RoundTrip(request *http.Request) (*http.Response, error) {
// determine whether to use the cache and return, or use the default transport
return http.DefaultTransport.RoundTrip(request)
}
func main() {
_ = httputil.ReverseProxy{
Transport: &CachingTransport{},
}
}
I'm doing some requests through some proxy servers. The function that defines which proxy url to use will choose randomly from a list of proxies. I would like to know for a given request, which proxy url is being used. As far as I know, when using a proxy server the http headers remain the same, but the tcp headers are the one that change.
Here's some code illustrating it (no error handling for simplicity):
func main() {
transport := &http.Transport{Proxy: chooseProxy}
client := http.Client{Transport: transport}
request, err := http.NewRequest(http.MethodGet, "https://www.google.com", nil)
checkErr(err)
// How to know here which proxy was used? Suppose the same client will perform several requests to different URL's.
response, err := client.Do(request)
checkErr(err)
dump, _ := httputil.DumpRequest(response.Request, false)
fmt.Println(dump)
}
func chooseProxy(request *http.Request) (*url.URL, error) {
proxies := []string{"proxy1", "proxy2", "proxy3"}
proxyToUse := proxies[rand.Intn(len(proxies))]
return url.Parse(proxyToUse)
}
I'm assuming that the Proxy function in the transport is called for each request even if the same client is used, as per the docs that say "Proxy specifies a function to return a proxy for a given Request". Am I right?
Some HTTP proxies add a Via header that tell who they are.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Via
You can modify your chooseProxy function so that it saves the proxy selected.
To do that, you can transform the chooseProxy func into a method of a type that will be used as storage for the information you want to keep:
type proxySelector string
func (sel *proxySelector) chooseProxy(request *http.Request) (*url.URL, error) {
proxies := []string{"proxy1", "proxy2", "proxy3"}
proxyToUse := proxies[rand.Intn(len(proxies))]
*sel = proxySelector(proxyToUse) // <-----
return url.Parse(proxyToUse)
}
func main() {
var proxy proxySelector
transport := &http.Transport{Proxy: proxy.chooseProxy} // <-----
client := http.Client{Transport: transport}
request, err := http.NewRequest(http.MethodGet, "https://www.google.com", nil)
checkErr(err)
// How to know here which proxy was used? Suppose the same client will perform several requests to different URL's.
response, err := client.Do(request)
checkErr(err)
dump, _ := httputil.DumpRequest(response.Request, false)
fmt.Println(dump)
fmt.Println("Proxy:", string(proxy)) // <-----
}
The request which contains the target URI is given as argument request to chooseProxy. So you can have the correct mapping already inside your chooseProxy function, all you need to to is check proxyToUse vs. request.URL there.
If you don't really trust the code that this mapping is actually done, then you need to look outside the code. For example you can look at the actual network traffic with Wireshark to see which proxy gets accessed.
Are there any options for logging request times in grpc-node? I've been able to log response times using opentelemetry & jaegar (to display response times). I couldn't find any npm packages for this either, but just wanted to ask you guys if you did find any options for grpc-node.
You don't need a package to do it, you can do it using a simple gRPC client interceptor.
This is how you would do it in Golang. Just check how you can create a gRPC interceptor in Node. Sorry for not having a JS example, hope it helps some how.
func UnaryRequestTImeInterceptor() grpc.UnaryClientInterceptor {
return func(
ctx context.Context,
method string,
req interface{},
reply interface{},
cc *grpc.ClientConn,
invoker grpc.UnaryInvoker,
opts ...grpc.CallOption,
) error {
start := time.Now()
err := invoker(ctx, method, req, reply, cc, opts...)
reqTime := time.Since(start)
return err
}
}
I've been using the gocraft-web package so far to do some development on an HTTP service. It's really great because you can stick middleware in it to check for stuff like the presence of a Cookie in the header.
At the moment I am wanting to implement request signing. Getting the client to sign the request is easy enough, but I am wanting to check it for all endpoints with a common piece of middleware. Basically the middleware needs to find the key to check against, compute the request HMAC, and check it against the supplied HMAC (presumably in the Authorization Header).
Computing the actual HMAC is really easy in go.
The problem is: reading the message in middleware makes it unavailable to the final endpoint.
The best solution I have come up with (example shown below) is to read everything from the Request in the middleware and stuffing it back into a bytes.Buffer for later reading. Is there a better way to do this? The current implementation seems a bit hackish.
Reading everything into memory sucks, but I can probably just put my service behind a proxy and limit the size of requests anyways. The actual content will always be pretty small(under 5 kilobytes). The extra copy introduced by this approach is likely to be quite slow, but computing the HMAC of a message is not exactly cheap to begin with.
The advantage to this is that it is transparent: it will work with any other go http code that just expects to read from Request.Body without any magic.
I suppose I could be a bit slicker and use a io.TeeReader.
This is my solution so far. If you post to localhost:3300 some JSON it prints the sha512 to the terminal in the server process, but also the response is able to contain a listing of the keys & values in it.
package main
import "fmt"
import "github.com/gocraft/web"
import "net/http"
import "bytes"
import "crypto/sha512"
import "io"
import "encoding/hex"
import "encoding/json"
type Context struct{}
type echoer struct {
*bytes.Buffer
}
func (e echoer) Close() error {
//Just do nothing to make the interface happy
return nil
}
func middlewareThatLooksAtBody(rw web.ResponseWriter, req *web.Request, next web.NextMiddlewareFunc) {
var replacement echoer
replacement.Buffer = &bytes.Buffer{}
hash := sha512.New()
hash.Write([]byte(req.Method))
reader := req.Body
var bytes []byte = make([]byte, 64)
for {
amount, err := reader.Read(bytes)
fmt.Printf("Read %d bytes\n", amount)
if err != nil {
if err == io.EOF {
break
}
panic(err)
}
if amount == 0 {
break
}
hash.Write(bytes)
replacement.Write(bytes)
}
//Is this needed?
reader.Close()
//replacement.Seek(0, 0)
req.Body = replacement
fmt.Printf("%v\n", hex.EncodeToString(hash.Sum(nil)))
next(rw, req)
}
func echoJson(rw web.ResponseWriter, req *web.Request) {
dec := json.NewDecoder(req.Body)
var obj map[string]interface{}
err := dec.Decode(&obj)
if err != nil {
rw.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(rw, "%v\n", err)
return
}
for k, v := range obj {
fmt.Fprintf(rw, "%v = %v\n", k, v)
}
}
func main() {
router := web.New(Context{})
router.Middleware(middlewareThatLooksAtBody)
router.Post("/", echoJson)
http.ListenAndServe("localhost:3300", router)
}
From your description, it looks like you need to read all the bytes from the request body, regardless of what your handlers will do.
If so, then you have at least a couple of options that would avoid the extra copy:
1) Store the read contents inside your gocraft context.
2) Do all body data processing and validation in the middleware and store the results of the processing in the context.
Granted, this means that your handlers now must know that they should look for the contents in the context instead of the req.Body.
I think it's a decent trade-off though, given your requirements.
In Go, a TCP connection (net.Conn) is a io.ReadWriteCloser. I'd like to test my network code by simulating a TCP connection. There are two requirements that I have:
the data to be read is stored in a string
whenever data is written, I'd like it to be stored in some kind of buffer which I can access later
Is there a data structure for this, or an easy way to make one?
No idea if this existed when the question was asked, but you probably want net.Pipe() which provides you with two full duplex net.Conn instances which are linked to each other
EDIT: I've rolled this answer into a package which makes things a bit simpler - see here: https://github.com/jordwest/mock-conn
While Ivan's solution will work for simple cases, keep in mind that a real TCP connection is actually two buffers, or rather pipes. For example:
Server | Client
---------+---------
reads <=== writes
writes ===> reads
If you use a single buffer that the server both reads from and writes to, you could end up with the server talking to itself.
Here's a solution that allows you to pass a MockConn type as a ReadWriteCloser to the server. The Read, Write and Close functions simply proxy through to the functions on the server's end of the pipes.
type MockConn struct {
ServerReader *io.PipeReader
ServerWriter *io.PipeWriter
ClientReader *io.PipeReader
ClientWriter *io.PipeWriter
}
func (c MockConn) Close() error {
if err := c.ServerWriter.Close(); err != nil {
return err
}
if err := c.ServerReader.Close(); err != nil {
return err
}
return nil
}
func (c MockConn) Read(data []byte) (n int, err error) { return c.ServerReader.Read(data) }
func (c MockConn) Write(data []byte) (n int, err error) { return c.ServerWriter.Write(data) }
func NewMockConn() MockConn {
serverRead, clientWrite := io.Pipe()
clientRead, serverWrite := io.Pipe()
return MockConn{
ServerReader: serverRead,
ServerWriter: serverWrite,
ClientReader: clientRead,
ClientWriter: clientWrite,
}
}
When mocking a 'server' connection, simply pass the MockConn in place of where you would use the net.Conn (this obviously implements the ReadWriteCloser interface only, you could easily add dummy methods for LocalAddr() etc if you need to support the full net.Conn interface)
In your tests you can act as the client by reading and writing to the ClientReader and ClientWriter fields as needed:
func TestTalkToServer(t *testing.T) {
/*
* Assumes that NewMockConn has already been called and
* the server is waiting for incoming data
*/
// Send a message to the server
fmt.Fprintf(mockConn.ClientWriter, "Hello from client!\n")
// Wait for the response from the server
rd := bufio.NewReader(mockConn.ClientReader)
line, err := rd.ReadString('\n')
if line != "Hello from server!" {
t.Errorf("Server response not as expected: %s\n", line)
}
}
Why not using bytes.Buffer? It's an io.ReadWriter and has a String method to get the stored data. If you need to make it an io.ReadWriteCloser, you could define you own type:
type CloseableBuffer struct {
bytes.Buffer
}
and define a Close method:
func (b *CloseableBuffer) Close() error {
return nil
}
In majority of the cases you do not need to mock net.Conn.
You only have to mock stuff that will add time to your tests, prevent tests from running in parallel (using shared resources like the hardcoded file name) or can lead to outages (you can potentially exhaust the connection limit or ports but in most of the cases it is not a concern, when you run your tests in isolation).
Not mocking has an advantage of more precise testing of what you want to test with a real thing.
https://www.accenture.com/us-en/blogs/software-engineering-blog/to-mock-or-not-to-mock-is-that-even-a-question
Instead of mocking net.Conn, you can write a mock server, run it in a goroutine in your test and connect to it using real net.Conn
A quick and dirty example:
port := someRandomPort()
srv := &http.Server{Addr: port}
go func(msg string) {
http.HandleFunc("/hello", myHandleFUnc)
srv.ListenAndServe()
}
myTestCodeUsingConn(port)
srv.Shutdown(context.TODO())