In Node.js, to finish writing to a stream (and in theory with HTTP, tell the client there is no more data), we use response.end(). Using Go, the ResponseWriter interface is like:
type ResponseWriter interface {
Header() Header
Write([]byte) (int, error)
WriteHeader(statusCode int)
}
so my question is twofold:
How can we get the HTTP status code from the ResponseWriter?
more importantly: How does Go (and routers like Mux) know when the programmer is done writing to the ResponseWriter? Is it when the goroutine ends? What if you wanted to finish the response before the goroutine stack is empty? Seems like an implementation flaw to not have an End() method in the ResponseWriter interface.
This is not possible with the standard http.ResponseWriter implementation. But this is an interface, so it's easy to write your own implementation that records the status. The beginning of a simple implementation might be:
type statusRecorder struct {
http.ResponseWriter
status int
}
func (r *statusRecorder) WriteHeader(status int) {
r.status = status
r.ResponseWriter.WriteHeader(status)
}
While this may seem like a limitation of the API, it's actually the opposite. By using an interface, it is possible to create an implementation that does anything, or records any information you want, rather than being limited to whatever functionality the standard library authors may have decided to expose.
When the handler returns, it is done. If you wish to do additional work after sending a response, you can spawn a goroutine to continue operating after the main handler returns.
Related
For example (golang):
type {
Product struct {
Name string
}
Customer struct {
Name string
Products []*Product
}
}
Which is the correct behavior:
GRPC honor the *Product pointer and transfer it only once.
GRPC will transfer the same *Product as many times as it associated to different Customer.
Michael,
It is not clear on your message, but I am assuming that you will send a Customer as part of your request to a gRPC server.
Golang will marshal the struct into []byte (https://godoc.org/github.com/golang/protobuf/proto#Marshal), so the message will not have such thing as a pointer. It will be just an encoded message. (see
https://github.com/golang/protobuf/blob/master/proto/wire.go#L22).
gRPC is not a Golang thing, so a pointer on a side (e.g. server) does not mean it must be a point on the other side (e.g. client).
Finally, answering your question, the expected behavior is 2. However, you may take a deeper look into proto buff serialization (https://developers.google.com/protocol-buffers/docs/encoding). I have no idea how it works, but maybe the message is compressed, so repeated []bytes maybe be discarded.
I have handlers that respond to https requests. In the handlers I call a function F1() which does some application logic and connects to a mysql db and does a query. I want to know how I can use the golang context package to cancel the Db query if the client cancels the request. Do I need to pass the ctx to F1()? Also the code I have now will take 4 seconds even if F1() returns in less then 4. How can I return as soon as F1() returns?
func handler(w http.ResponseWriter, r *http.Request) {
ctx:= r.context()
F1()
select {
case <-ctx.Done():
case <- time.After( 4*time.Second):
}
w.WriteHeader(http.statusOk)
return
}
To begin, I highly recommend taking a look at the Context blog post to familiarize yourself with contexts, in addition to reading over the context documentation itself.
To address your specific questions:
How can you cancel the database query if the user cancels their quest?
To make this work, there are a few things you want to check:
Ensure that your database driver (if you are using database/sql) supports context cancellation.
Ensure you are using the Context variants of all available methods (e.g. db.QueryContext instead of db.Query).
Ensure that you are passing the context (or a derivative of the context) through the stack from your HTTP request through to the database calls.
Do I need to pass the ctx to F1()?
Per #3 above, yes: you will need to pass the context through all intermediate calls to "connect" the database call with the request context.
How can I return as soon as F1() returns?
The code that you have in your question calls F1 in series, rather than concurrently, with your cancellation/timeout select.
If you want to apply a specific deadline to your database call, use context.WithTimeout to limit how long it can take. Otherwise, you do not need to do anything special: just call F1 with your context, and the rest will happen for you, no select is needed.
I need to test an app which uses Google Cloud Pubsub, and so must wrap its types pubsub.Client and pubsub.Subscriber for testing purposes. However, despite several attempts I can't get an interface around them which compiles.
The definitions of the methods I'm trying to wrap are:
func (s *Subscription) Receive(
ctx context.Context, f func(context.Context, *Message)) error
func (c *Client) Subscription(id string) *Subscription
Here is the current code. The Receiver interface (wrapper around Subscriber) seems to work, but I suspect it may need to change in order to fix SubscriptionMaker, so I've include both.
Note: I've tried several variations of where to reference and dereference pointers, so please don't tell me to change that unless you have an explanation of why your suggested configuration is the correct one or you've personally verified it compiles.
import (
"context"
"cloud.google.com/go/pubsub"
)
type Receiver interface {
Receive(context.Context, func(ctx context.Context, msg *pubsub.Message)) (err error)
}
// Pubsub subscriptions implement Receiver
var _ Receiver = &pubsub.Subscription{}
type SubscriptionMaker interface {
Subscription(name string) (s Receiver)
}
// Pubsub clients implement SubscriptionMaker
var _ SubscriptionMaker = pubsub.Client{}
Current error message:
common_types.go:21:5: cannot use "cloud.google.com/go/pubsub".Client literal (type "cloud.google.com/go/pubsub".Client) as type SubscriptionMaker in assignment:
"cloud.google.com/go/pubsub".Client does not implement SubscriptionMaker (wrong type for Subscription method)
have Subscription(string) *"cloud.google.com/go/pubsub".Subscription
want Subscription(string) Receiver
First, for most uses, using the ptest package is probably a much easier approach for testing pubsub. But of course, your specific question can apply to any library, and the below approach can be useful for many things, not just mocking pubsub.
Your broader goal of using interfaces to mock a library like this, is doable. But it is complicated when the library you wish to mock out returns concrete types that you cannot mock (probably due to unreported fields). The approach to be taken is much more involved than is often worth it, as there may be easier ways to test your code.
But if you're intent on doing this, the approach you must take is to not wrap the entire package in interfaces, not just the specific methods you wish to mock.
You would need to wrap any types that you wish to mock which are returned by or accepted by your interface, too. This usually means you also need to modify your production code (not just your test code), so this can sometimes be a deal-breaker for existing code bases.
Where I have usually done this before is when mocking something like the standard library's sql driver, but the same approach can be applied here. In essence, you would need to create a wrapper package for your pubsub library, which you use even in your production code. Again, this can be quite intrusive on existing codebases, but for the sake of illustration. Using your defined interfaces:
package mypubsub
import "cloud.google.com/go/pubsub"
type Receiver interface {
Recieve(context.Context, func(context.Context, *pubsub.Message) error)
}
type SubscriptionMaker interface {
Subscription(string) Receiver
}
You can then wrap the default implementation, for use in production code:
// defaultClient wraps the default pubsub Client functionality.
type defaultClient struct {
*pubsub.Client
}
func (d defaultImplementation) Subscription(name string) Receiver {
return d.Client.Subscription()
}
Naturally, you'd need to expand this package to wrap most or all of the pubsub package you're using. This can be a bit daunting.
But once you've done that, then use your mypubsub package everywhere in your code, instead of directly depending on the pubsub package. And now you can easily swap out a mock implementation anywhere you need for testing.
It can't be done.
When defining the type signature of a method on an interface, it must match exactly. func (c *Client) Subscription(id string) *Subscription returns a *Subscription, and a *Subscription is a valid Receiver, but it does not count as conforming to the interface method Subscription(string) Receiver. Go requires precise matching for function signatures, not the duck-typing style that it usually uses for interfaces.
I want to run some slow routine in another goroutine, is it safe to do it like this:
func someHandler(w http.ResponseWriter, r *http.Request) {
go someReallySlowFunction() // sending mail or something slow
fmt.Fprintf(w,"Mail will be delivered shortly..")
}
func otherHandler(w http.ResponseWriter, r *http.Request) {
foo := int64(0)
bar := func() {
// do slow things with foo
}
go bar()
fmt.Fprintf(w,"Mail will be delivered shortly..")
}
Is there any gotchas by doing this?
Serving each http request runs in its own goroutine (more details on this). You are allowed to start new goroutines from your handler, and they will run concurrently, independently from the goroutine executing the handler.
Some things to look out for:
The new goroutine runs independently from the handler goroutine. This means it may complete before or after the handler goroutine, you cannot (should not) assume anything regarding to this without explicit synchronization.
The http.ResponseWriter and http.Request arguments of the handler are only valid and safe to use until the handler returns! These values (or "parts" of them) may be reused - this is an implementation detail of which you should also not assume anything. Once the handler returns, you should not touch (not even read) these values.
Once the handler returns, the response is committed (or may be committed at any moment). Which means your new goroutine should not attempt to send back any data using the http.ResponseWriter after this. This is true to the extent that even if you don't touch the http.ResponseWriter in your handler, not panicing from the handler is taken as a successful handling of the request and thus HTTP 200 status is sent back (see an example of this).
You are allowed to pass the http.Request and http.ResponseWriter values to other functions and to new goroutines, but care must be taken: you should use explicit synchronization (e.g. locks, channels) if you intend to read / modify these values from multiple goroutines (or you want to send back data from multiple goroutines).
Note that seemingly if both your handler goroutine and your new goroutine just reads / inspects the http.Request, that still may be problematic. Yes, multiple goroutines can read the same variable without synchronization (if nobody modifies it). But calling certain methods of http.Request also modify the http.Request, and without synchronization there is no guarantee what other goroutines would see from this change. For example Request.FormValue() returns a form value associated with the given key. But this method calls ParseMultiPartForm() and ParseForm() if necessary which modify the http.Request (e.g. they set the Request.PostForm and Request.Form struct fields).
So unless you synchronize your goroutines, you should not pass Request and ResponseWriter to the new goroutine, but acquire data needed from the Request in the handler goroutine, and pass only e.g. a struct holding the needed data.
Your second example:
foo := int64(0)
bar := func() {
// do slow things with foo
}
go bar()
This is perfectly fine. This is a closure, and local variables referred by it will survive as long as they are accessible.
Note that alternatively you could pass the value of the local variable to the anonymous function call as an argument like this:
foo := int64(0)
bar := func(foo int64) {
// do slow things with param foo (not the local foo var)
}
go bar(foo)
In this example the anonymous function will see and use its parameter foo and not the local variable foo. This may or may not be what you want (depending on whether the handler also uses the foo and whether changes made by any of the goroutines need to be visible to the other - but that would require synchronization anyway, which would be superseded by a channel solution).
If you care for acknowledgement for the mail, then the posted code won't help. Running the code in separate goroutine makes it independent and the server reply will be success even if the mail is not sent due to some error in the goroutine function.
Im creating a Networking API and want people to be able to route requests to specific endpoints using a ServeMux. Instead of using a Server instance, I need to use my own low level ServerConn. This is because I am receiving both incoming HTTP requests and plain text data from the same port.
The problem, however, is that if I want to forward a request using my ServeMux, I would use it's ServeHTTP method. For this, I need to provide a ResponseWriter, which I don't know how to create an instance of since it is an interface, not a struct. Should a I create my own ResponseWriter struct? Is there one given by the Golang Standard Library? Or is there an alternate solution to this altogether?
I would avoid doing this altogether if at all possible. Mixing protocols on the same connection is bound to lead to hard-to-trace bugs, and unexpected behavior. If you really want to do it, and have all the http/1.1 mechanisms work correctly, leave as much as possible to the http package.
Since ResponseWriter is an interface, you would implement your own type to satisfy it. Look at the unexported response type in the http package for a full example. There's a lot to get right, and using it in combination with a ServerConn (which is documented as "do no use") is probably not a good idea.
The place to do this at a lower level would be in Accept inside the Server's net.Listener. Since you're going to have to parse the start of every request twice, you would need a net.Conn that can be "rewound" partly.
Make yourself a net.Listener that checks the start of the stream on a new connection, and if it looks like an http request, return a net.Conn that replays the first chunk you read off the wire on its first Reads. Something like:
type replayConn struct {
net.Conn
buf []byte
pos int
}
func (c *replayConn) Read(b []byte) (int, error) {
if c.pos < len(c.buf) {
n := copy(b, c.buf[c.pos:])
c.pos += n
return n, nil
}
return c.Conn.Read(b)
}
If the connection isn't http, then send the connection off to your other type of handler, and continue blocking on Accept.