Reading image from HTTP request's body in Go - http

I'm playing with Go (first time ever) and I want to build a tool to retrieve images from Internet and cut them (even resize) but I'm stuck on the first step.
package main
import (
"fmt"
"http"
)
var client = http.Client{}
func cutterHandler(res http.ResponseWriter, req *http.Request) {
reqImg, err := client.Get("http://www.google.com/intl/en_com/images/srpr/logo3w.png")
if err != nil {
fmt.Fprintf(res, "Error %d", err)
return
}
buffer := make([]byte, reqImg.ContentLength)
reqImg.Body.Read(buffer)
res.Header().Set("Content-Length", fmt.Sprint(reqImg.ContentLength)) /* value: 7007 */
res.Header().Set("Content-Type", reqImg.Header.Get("Content-Type")) /* value: image/png */
res.Write(buffer)
}
func main() {
http.HandleFunc("/cut", cutterHandler)
http.ListenAndServe(":8080", nil) /* TODO Configurable */
}
I'm able to request an image (let's use Google logo) and to get its kind and size.
Indeed, I'm just re-writing the image (look at this as a toy "proxy"), setting Content-Length and Content-Type and writing the byte slice back but I get it wrong somewhere. See how it looks the final image rendered on Chromium 12.0.742.112 (90304):
Also I checked the downloaded file and it is a 7007 bytes PNG image. It should be working properly if we look at the request:
GET /cut HTTP/1.1
User-Agent: curl/7.22.0 (i486-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.23 libssh2/1.2.8 librtmp/2.3
Host: 127.0.0.1:8080
Accept: /
HTTP/1.1 200 OK
Content-Length: 7007
Content-Type: image/png
Date: Tue, 27 Dec 2011 19:51:53 GMT
[PNG data]
What do you think I'm doing wrong here?
Disclaimer: I'm scratching my own itch, so probably I'm using the wrong tool :) Anyway, I can implement it on Ruby but before I would like to give Go a try.
Update: still scratching itches but... I think this is going to be a good side-of-side project so I'm opening it https://github.com/imdario/go-lazor If it is not useful, at least somebody can find usefulness with the references used to develop it. They were for me.

I think you went too fast to the serve things part.
Focus on the first step, downloading the image.
Here you have a little program that downloads that image to memory.
It works on my 2011-12-22 weekly version, for r60.3 you just need to gofix the imports.
package main
import (
"log"
"io/ioutil"
"net/http"
)
const url = "http://www.google.com/intl/en_com/images/srpr/logo3w.png"
func main() {
// Just a simple GET request to the image URL
// We get back a *Response, and an error
res, err := http.Get(url)
if err != nil {
log.Fatalf("http.Get -> %v", err)
}
// We read all the bytes of the image
// Types: data []byte
data, err = ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ioutil.ReadAll -> %v", err)
}
// You have to manually close the body, check docs
// This is required if you want to use things like
// Keep-Alive and other HTTP sorcery.
res.Body.Close()
// You can now save it to disk or whatever...
ioutil.WriteFile("google_logo.png", data, 0666)
log.Println("I saved your image buddy!")
}
Voilá!
This will get the image to memory inside data.
Once you have that, you can decode it, crop it and serve back to the browser.
Hope this helps.

I tried your code and noticed that the image you were serving was the right size, but the contents of the file past a certain point were all 0x00.
Review the io.Reader documentation. The important thing to remember is that Read reads up to the number of bytes you request. It can read fewer with no error returned. (You should be checking the error too, but that's not an issue here.)
If you want to make sure your buffer is completely full, use io.ReadFull. In this case it's simpler to just copy the entire contents of the Reader with io.Copy.
It's also important to remember to close HTTP request bodies.
I would rewrite the code this way:
package main
import (
"fmt"
"http"
"io"
)
var client = http.Client{}
func cutterHandler(res http.ResponseWriter, req *http.Request) {
reqImg, err := client.Get("http://www.google.com/intl/en_com/images/srpr/logo3w.png")
if err != nil {
fmt.Fprintf(res, "Error %d", err)
return
}
res.Header().Set("Content-Length", fmt.Sprint(reqImg.ContentLength))
res.Header().Set("Content-Type", reqImg.Header.Get("Content-Type"))
if _, err = io.Copy(res, reqImg.Body); err != nil {
// handle error
}
reqImg.Body.Close()
}
func main() {
http.HandleFunc("/cut", cutterHandler)
http.ListenAndServe(":8080", nil) /* TODO Configurable */
}

Related

Downloading just part of a file

I want to download just part of a text-like file from the internet using go. It appears that curl --max-filesize and --range aren't respected by the websites I want to download from. Additionally, I read that the http MaxBytesReader still downloads the entire file, but only stores part of it.
Is there a way to just get the first kb of a file and then close the connection? The equivalent of pressing "x" when a large page is loading, on chrome.
I'm thinking that I can run a thread that reads a website to a file, and then kill the thread after a ms or two. Is this possible?
One simple way to do it without much error handling (which would have to added could be):
import (
"fmt"
"io"
"io/ioutil"
"net/http"
)
const readLimit = 1024 // bytes
func main() {
resp, err := http.Get("http://example.com/")
if err != nil {
// handle error
}
fixedReader := io.LimitedReader{R: resp.Body, N: readLimit}
data, _ := ioutil.ReadAll(fixedReader)
resp.Body.Close()
fmt.Println(string(data))
}

Reading non-utf8 encoded data from a network call in golang

I am trying to read bytes from http response body in golang. My problem is that the response body is encoded using ISO-8859-1. I want to read the response body in the same encoding and write the contents to a file in the ISO-8859-1 encoding.
Is there a way using which I can accomplish this? I don't want to convert the data into UTF-8 at all.
Here is a good read about encoding, which you might benefit from.
You are seemingly assuming Go decodes the raw bytes it receives when it performs a request. It does not.
Take this example:
package main
import (
"io"
"log"
"net/http"
"os"
)
func main() {
// We perform a request to a Latin-1 encoded page
resp, err := http.Get("http://andrew.triumf.ca/multilingual/samples/german.meta.html")
if err != nil {
log.Fatalln(err)
}
//
f, err := os.Create("/tmp/latin1")
defer f.Close()
if err != nil {
log.Fatalln(err)
}
io.Copy(f, resp.Body)
}
In the documentation, you can read that resp.Body conforms to the io.ReadCloser interface, which allows you to read the raw bytes and stream them to a file.
Once we run this code, this is the output of file -i /tmp/latin1:
/tmp/latin1: text/html; charset=iso-8859-1
Read and write the response body as a slice of bytes, []byte, an opaque data type.

Unexpected EOF using Go http client

I am learning Go and came across this problem.
I am just downloading web page content using HTTP client:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://mail.ru/", nil)
req.Close = true
response, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
content, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(content)[:100])
}
I get an unexpected EOF error when reading response body. At the same time content variable has full page content.
This error appear only when I downloading https://mail.ru/ content. With other URLs everything works fine - without any errors.
I used curl for downloading this page content - everything works as expected.
I am confused a bit - what's happening here?
Go v1.2, tried on Ubuntu and MacOS X
It looks like the that server (Apache 1.3, wow!) is serving up a truncated gzip response. If you explicitly request the identity encoding (preventing the Go transport from adding gzip itself), you won't get the ErrUnexpectedEOF:
req.Header.Add("Accept-Encoding", "identity")

Inspecting the body of an HTTP request with gocraft Middleware

I've been using the gocraft-web package so far to do some development on an HTTP service. It's really great because you can stick middleware in it to check for stuff like the presence of a Cookie in the header.
At the moment I am wanting to implement request signing. Getting the client to sign the request is easy enough, but I am wanting to check it for all endpoints with a common piece of middleware. Basically the middleware needs to find the key to check against, compute the request HMAC, and check it against the supplied HMAC (presumably in the Authorization Header).
Computing the actual HMAC is really easy in go.
The problem is: reading the message in middleware makes it unavailable to the final endpoint.
The best solution I have come up with (example shown below) is to read everything from the Request in the middleware and stuffing it back into a bytes.Buffer for later reading. Is there a better way to do this? The current implementation seems a bit hackish.
Reading everything into memory sucks, but I can probably just put my service behind a proxy and limit the size of requests anyways. The actual content will always be pretty small(under 5 kilobytes). The extra copy introduced by this approach is likely to be quite slow, but computing the HMAC of a message is not exactly cheap to begin with.
The advantage to this is that it is transparent: it will work with any other go http code that just expects to read from Request.Body without any magic.
I suppose I could be a bit slicker and use a io.TeeReader.
This is my solution so far. If you post to localhost:3300 some JSON it prints the sha512 to the terminal in the server process, but also the response is able to contain a listing of the keys & values in it.
package main
import "fmt"
import "github.com/gocraft/web"
import "net/http"
import "bytes"
import "crypto/sha512"
import "io"
import "encoding/hex"
import "encoding/json"
type Context struct{}
type echoer struct {
*bytes.Buffer
}
func (e echoer) Close() error {
//Just do nothing to make the interface happy
return nil
}
func middlewareThatLooksAtBody(rw web.ResponseWriter, req *web.Request, next web.NextMiddlewareFunc) {
var replacement echoer
replacement.Buffer = &bytes.Buffer{}
hash := sha512.New()
hash.Write([]byte(req.Method))
reader := req.Body
var bytes []byte = make([]byte, 64)
for {
amount, err := reader.Read(bytes)
fmt.Printf("Read %d bytes\n", amount)
if err != nil {
if err == io.EOF {
break
}
panic(err)
}
if amount == 0 {
break
}
hash.Write(bytes)
replacement.Write(bytes)
}
//Is this needed?
reader.Close()
//replacement.Seek(0, 0)
req.Body = replacement
fmt.Printf("%v\n", hex.EncodeToString(hash.Sum(nil)))
next(rw, req)
}
func echoJson(rw web.ResponseWriter, req *web.Request) {
dec := json.NewDecoder(req.Body)
var obj map[string]interface{}
err := dec.Decode(&obj)
if err != nil {
rw.WriteHeader(http.StatusBadRequest)
fmt.Fprintf(rw, "%v\n", err)
return
}
for k, v := range obj {
fmt.Fprintf(rw, "%v = %v\n", k, v)
}
}
func main() {
router := web.New(Context{})
router.Middleware(middlewareThatLooksAtBody)
router.Post("/", echoJson)
http.ListenAndServe("localhost:3300", router)
}
From your description, it looks like you need to read all the bytes from the request body, regardless of what your handlers will do.
If so, then you have at least a couple of options that would avoid the extra copy:
1) Store the read contents inside your gocraft context.
2) Do all body data processing and validation in the middleware and store the results of the processing in the context.
Granted, this means that your handlers now must know that they should look for the contents in the context instead of the req.Body.
I think it's a decent trade-off though, given your requirements.

Go - accepting http post multipart file(s)

I am trying to figure out how to accept/receive a HTTP Post in Go. I just want to be able to receive a file, grab its mime-type and save the file locally.
I've been searching all day but all I can find is how to send a file to some remote location but none of the examples I find cover receiving it.
Any help would be appreciated.
using Justinas' example and mixing with my existing experiment I've gotten this far but m.Post never seems to be called.
package main
import (
"fmt"
"io"
"net/http"
"os"
"github.com/codegangsta/martini"
"github.com/codegangsta/martini-contrib/render"
)
func main() {
m := martini.Classic()
m.Use(render.Renderer(render.Options{
Directory: "templates", // Specify what path to load the templates from.
Layout: "layout", // Specify a layout template. Layouts can call {{ yield }} to render the current template.
Charset: "UTF-8", // Sets encoding for json and html content-types.
}))
m.Get("/", func(r render.Render) {
fmt.Printf("%v\n", "g./")
r.HTML(200, "hello", "world")
})
m.Get("/:who", func(args martini.Params, r render.Render) {
fmt.Printf("%v\n", "g./:who")
r.HTML(200, "hello", args["who"])
})
m.Post("/up", func(w http.ResponseWriter, r *http.Request) {
fmt.Printf("%v\n", "p./up")
file, header, err := r.FormFile("file")
defer file.Close()
if err != nil {
fmt.Fprintln(w, err)
return
}
out, err := os.Create("/tmp/file")
if err != nil {
fmt.Fprintf(w, "Failed to open the file for writing")
return
}
defer out.Close()
_, err = io.Copy(out, file)
if err != nil {
fmt.Fprintln(w, err)
}
// the header contains useful info, like the original file name
fmt.Fprintf(w, "File %s uploaded successfully.", header.Filename)
})
m.Run()
}
Go's net/http server handles this pretty, using mime/multipart package behind the scenes. You only need to call r.FormFile() on your *http.Request to get a multipart.File back.
Here's a complete example. And the result of uploading a file with curl:
justinas#ubuntu /tmp curl -i -F file=#/tmp/stuff.txt http://127.0.0.1:8080/
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Tue, 24 Dec 2013 20:56:07 GMT
Content-Length: 37
Content-Type: text/plain; charset=utf-8
File stuff.txt uploaded successfully.%
justinas#ubuntu /tmp cat file
kittens!

Resources