Finally I am posting my first question on StackOverflow. I'm using this site for years now and I always found great answers to all my questions :)
I am implementing a file encryption background daemon which is based on the official Golang cipher example:
func ExampleStreamReader() {
key := []byte("example key 1234")
inFile, err := os.Open("encrypted-file")
if err != nil {
panic(err)
}
defer inFile.Close()
block, err := aes.NewCipher(key)
if err != nil {
panic(err)
}
// If the key is unique for each ciphertext, then it's ok to use a zero
// IV.
var iv [aes.BlockSize]byte
stream := cipher.NewOFB(block, iv[:])
outFile, err := os.OpenFile("decrypted-file", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
panic(err)
}
defer outFile.Close()
reader := &cipher.StreamReader{S: stream, R: inFile}
// Copy the input file to the output file, decrypting as we go.
if _, err := io.Copy(outFile, reader); err != nil {
panic(err)
}
// Note that this example is simplistic in that it omits any
// authentication of the encrypted data. If you were actually to use
// StreamReader in this manner, an attacker could flip arbitrary bits in
// the output.
}
func ExampleStreamWriter() {
key := []byte("example key 1234")
inFile, err := os.Open("plaintext-file")
if err != nil {
panic(err)
}
defer inFile.Close()
block, err := aes.NewCipher(key)
if err != nil {
panic(err)
}
// If the key is unique for each ciphertext, then it's ok to use a zero
// IV.
var iv [aes.BlockSize]byte
stream := cipher.NewOFB(block, iv[:])
outFile, err := os.OpenFile("encrypted-file", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
if err != nil {
panic(err)
}
defer outFile.Close()
writer := &cipher.StreamWriter{S: stream, W: outFile}
// Copy the input file to the output file, encrypting as we go.
if _, err := io.Copy(writer, inFile); err != nil {
panic(err)
}
// Note that this example is simplistic in that it omits any
// authentication of the encrypted data. If you were actually to use
// StreamReader in this manner, an attacker could flip arbitrary bits in
// the decrypted result.
}
What is meant with the following quote. About what should I take care to provide a secure encryption and decryption?
Note that this example is simplistic in that it
omits any authentication of the encrypted data. If you were actually
to use StreamReader in this manner, an attacker could flip arbitrary
bits in the output.
Thanks!
From wikipedia:
The block cipher modes ECB, CBC, OFB, CFB, CTR, and XTS provide confidentiality, but they do not protect against accidental modification or malicious tampering.
A good explanation can be found here: https://security.stackexchange.com/a/33576.
Go has support for other modes which do support integrity and authentication checks. As rossum said you can use GCM or CCM. You can find lots of examples on godoc.org. For example HashiCorp's memberlist library.
Another library worth checking out is the NaCL port in golang.org/x/crypto/nacl:
func Open(out []byte, box []byte, nonce *[24]byte, key *[32]byte) ([]byte, bool)
func Seal(out, message []byte, nonce *[24]byte, key *[32]byte) []byte
If you're working with small messages this API will probably be a lot easier to use.
Related
I am writing an HTTP server in Go, which uses the following pattern to handle API output:
func handler(w http.ResponsWriter, r *http.Request) {
defer reply(w, r, L)() //L is a Logger
//do things...
}
func reply(w http.ResponseWriter, r *http.Request, log Logger) func() {
cid := []byte{0, 0, 0, 0}
if log != nil {
rand.Read(cid)
log.Debug("[%x] %s %s", cid, r.Method, r.URL.String())
}
entry := time.Now()
return func() {
if log != nil {
defer log.Debug("[%x] elapsed %d millis", cid, time.Since(entry).Milliseconds())
}
_, err := w.Write(nil)
if err == http.ErrHijacked {
return //API is a WEBSOCKET entry point, do nothing
}
//handle common output logic for normal HTTP APIs...
}
}
The reason I do this, is that I found this comment in the standard library:
// ErrHijacked is returned by ResponseWriter.Write calls when
// the underlying connection has been hijacked using the
// Hijacker interface. A zero-byte write on a hijacked
// connection will return ErrHijacked without any other side
// effects.
ErrHijacked = errors.New("http: connection has been hijacked")
However following the Write() method, I got this comment:
// Write writes the data to the connection as part of an HTTP reply.
//
// If WriteHeader has not yet been called, Write calls
// WriteHeader(http.StatusOK) before writing the data. If the Header
// does not contain a Content-Type line, Write adds a Content-Type set
// to the result of passing the initial 512 bytes of written data to
// ...
Write([]byte) (int, error)
My questions are:
Is it OK to use my code to safely detect if a HTTP connection is hijacked? I only want to check the connection is hijacked or not, but do NOT want it to add headers for me!
Since the ResponseWriter is an interface, I cannot click through the source code to find out how the standard library implements that method. In general, how can I drill down to the standard library (or any open source code) to find out the implementation of an interface?
Thanks to Cerise, I found the source code of the standard response.Writer:
func (w *response) write(lenData int, dataB []byte, dataS string) (n int, err error) {
if w.conn.hijacked() {
if lenData > 0 {
caller := relevantCaller()
w.conn.server.logf("http: response.Write on hijacked connection from %s (%s:%d)", caller.Function, path.Base(caller.File), caller.Line)
}
return 0, ErrHijacked
}
... ....
So, as said in the document, there is NO side effect.
I am trying to construct a request with the multipart.CreateFormFile and multipart.CreateFormField, but I am not having any success on transferring the data to its destination.
I want to send a form post to a client and the client will do some pre-processing of the data and then send it to its final destination at the server. It should be noted, I am only have problems when a file is being transferred, a normal application/x-www-form-urlencoded I can get it to work fine.
So here is my scenario. Someone submits a form with a file to /post, the /post route will then take the form data and use multipart.CreateFormFile and multipart.CreateFormField to make a new http.newRequest.
Here is my code:
http.HandleFunc("/post", func(w http.ResponseWriter, r *http.Request){
r.ParseMultipartForm(10 << 20)
file, header, _ := r.FormFile("file")
b := bytes.Buffer{}
we := multipart.NewWriter(&b)
var fw io.Writer
defer we.Close()
fw, err := we.CreateFormFile("file", header.Filename) //create the new file field
_, err = io.Copy(fw, file) //copy the file to the new file field
if err != nil {
fmt.Println("error with copy")
}
for key, r0 := range r.Form { //this creates and copies the form values to the new created form
if fw, err = we.CreateFormField(key); err != nil {
return
}
_, err = io.Copy(fw, strings.NewReader(r0[0])) //copies field data to new form field
if err != nil {
fmt.Println("error with copy fields")
}
}
req, err := http.NewRequest("POST", "http://localhost:3000/newpost", &b) //make new request from buffer
CheckError(err, "error with post form req")
req.Header.Set("Content-Type", we.FormDataContentType()) //set content type
resp, err := http.DefaultClient.Do(req) //send request and get response
CheckError(err, "error with post form resp")
fmt.Println(resp.StatusCode)
})
This is stumping me because I am finding minimal explanation on what I am trying to achieve, or I am missing something fundamental.
So in summary, I want to create a new Form with a file to send in an http.NewRequest through POST. If I pass in http.Request.Form to the NewRequest it sends only the form values and not the form file.
I am implementing a retry using http.RoundTripper in Go. Here is an implementation example.
type retryableRoundTripper struct {
tr http.RoundTripper
maxRetryCount int
}
func (t *retryableRoundTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
for count := 0; count < t.maxRetryCount; count++ {
log.Printf("retryableRoundTripper retry: %d\n", count+1)
resp, err = t.tr.RoundTrip(req)
if err != nil || resp.StatusCode != http.StatusTooManyRequests {
return resp, err
}
}
return resp, err
}
Questions
Is it necessary to read and close the response body to reuse the TCP connection on retry?
func (t *retryableRoundTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
for count := 0; count < t.maxRetryCount; count++ {
log.Printf("retryableRoundTripper retry: %d\n", count+1)
resp, err = t.tr.RoundTrip(req)
if err != nil || resp.StatusCode != http.StatusTooManyRequests {
return resp, err
}
}
// add
io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
return resp, err
}
As a side note, I've written a test and have confirmed that retries work as expected. (In Go Playground, it times out, but it works locally.)
https://play.golang.org/p/08YWV0kjaKr
Of course you need to read the connection to ensure that it can be reused, and Closing the connection is required as documented.
As stated in the docs:
The client must close the response body when finished with it
and
The default HTTP client's Transport may not
reuse HTTP/1.x "keep-alive" TCP connections if the Body is
not read to completion and closed.
If the server wants to send more data than fits in the initial read buffers, it is going to be blocked sending the response. This means that if the transport were to attempt to send a new request over that connection the server may not be able to handle it because it never completed the first request. This will usually result in a client error of connection reset by peer and a server error of write: broken pipe.
If you want to make an attempt to reuse the connection, but limit the amount read, use an io.LimitedReader and/or check the ContentLength value. This way you can discard the connection when it's faster to handle the errors and bring up a new connection than to read an unbounded amount of data. See Limiting amount of data read in the response to a HTTP GET request.
I have a function that takes a json decoder and an interface as its arguements, and i am decoding to a struct that is passed on the interface. Like so:
func DecodeJSON(decoder *json.Decoder, i interface{}) bool {
if c, ok := i.(User); ok {
err := decoder.Decode(c)
if err != nil {
fmt.Println(err)
return false //err is not nil
}
}
return false
}
Usage of the function:
// Register test
func HandleRequest(w rest.ResponseWriter, r *rest.Request) {
decoder := json.NewDecoder(r.Body)
user := User{}
if DecodeJSON(decoder, user) {
fmt.Println("success")
}
Error i am getting:
json: Unmarshal(non-pointer main.User)
Bit confused by this message since my DecodeJSON is not taking a pointer for user. So am not sure what i have done wrong with my code. Hope some one can explain so i can understand my mistake.
You need to use a pointer to a user to decode the data in, otherwise the decoded data gets decoded in a copy of the object that gets deleted when the function returns.
func DecodeJSON(decoder *json.Decoder, i interface{}) bool {
if c, ok := i.(*User); ok {
err := decoder.Decode(c)
if err != nil {
fmt.Println(err)
return false //err is not nil
}
}
return false
}
// Register test
func HandleRequest(w rest.ResponseWriter, r *rest.Request) {
decoder := json.NewDecoder(r.Body)
user := &User{}
if DecodeJSON(decoder, user) {
fmt.Println("success")
}
Leave the interface parameter as is, just use a pointer when you pass the User and when you get the user from the interface.
Based on your code, you'd better change your function signature to
func DecodeJSON(decoder *json.Decoder, user *User) bool
This will (1) eliminate explicit runtime cast, and (2) make your code less ambiguous and safe and compile-time checked.
I need to make a request to a server that return different responses at different times. I mean, the server generate different responses and these responses take different execution time, so server return the responses as soon as they are available.
And I want print in the screen (by the moment, I'd settle with that) these responses as soon as the server returns me.
All what I could do until now is print the responses but only when the server returns all the responses. So if the first response take 1sec, and the last response take 10sec, my code needs to wait 10sec to print all the messages.
EDIT: to add code I have:
//Config is gotten from yml file
RestConfig = Config["rest"].(map[string]interface{})
ServerConfig = Config["server"].(map[string]interface{})
RequestUrl := ServerConfig["url"]
RequestReader := bytes.NewReader(body)
Request, _ := http.NewRequest("POST", RequestUrl.(string), RequestReader)
//AppendHeaders append the needing headers to the request
client.AppendHeaders(Request, RestConfig["headers"])
//the type of client.HttpClient is *http.Client
Response, _ := client.HttpClient.Do(Request)
//And to print in the screen
defer Response.Body.Close()
fmt.Println( "-> Receiving response:\n---\n" )
fmt.Println( Response , "\n---\n-> Response body:\n---\n")
body_resp, _ := ioutil.ReadAll(Response.Body)
fmt.Println( string(body_resp) )
fmt.Println( "\n--\n")
Any way to do it??
Thank you very much.
Finally my code , is like this:
package main
import (
"fmt"
"log"
"bytes"
"strings"
"bufio"
"net/http"
)
func main() {
var body = "The body"
RequestReader := bytes.NewReader([]byte(body))
req, err := http.NewRequest("POST", "the_url", RequestReader)
if err != nil {
log.Fatal(err)
}
req.Header.Add("Accept", "application/xml")
req.Header.Add("Content-Type", "application/xml")
req.Header.Add("AG-Authorization", "key")
req.Header.Add("AG-Forwarded-Hosts", "*")
resp, err := (&http.Client{}).Do(req)
if err != nil {
log.Fatal(err)
}
reader := bufio.NewReader(resp.Body)
message := ""
for {
line, err := reader.ReadBytes('\n')
if err != nil {
log.Fatal(err)
}
message = message + string(line)
if strings.Contains(message, "<!-- End mark for each message -->"){
fmt.Println(message)
message = ""
}
}
}
Thank everyone.
The context package is what you are looking for.
The context package is responsible for signal cancelation and operation deadlines for processes and server requests. This has two public methods: WithCancel and WithTimeout. The Context associated with an incoming request is typically canceled when the request handler returns.
For your specific case you can use the WithTimeout method for setting a deadline on requests to backend servers.
// WithTimeout returns a copy of parent whose Done channel is closed as soon as
// parent.Done is closed, cancel is called, or timeout elapses. The new
// Context's Deadline is the sooner of now+timeout and the parent's deadline, if
// any. If the timer is still running, the cancel function releases its
// resources.
func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc)
And here is a snippet taken from https://blog.golang.org/context/server/server.go
timeout, err := time.ParseDuration(req.FormValue("timeout")) // set a time limit in your post
if err == nil {
// The request has a timeout, so create a context that is
// canceled automatically when the timeout expires.
ctx, cancel = context.WithTimeout(context.Background(), timeout)
} else {
ctx, cancel = context.WithCancel(context.Background())
}
defer cancel() // Cancel ctx as soon as handleSearch returns.
For further reading take a look at this article:
https://blog.golang.org/context