Go - accepting http post multipart file(s) - http

I am trying to figure out how to accept/receive a HTTP Post in Go. I just want to be able to receive a file, grab its mime-type and save the file locally.
I've been searching all day but all I can find is how to send a file to some remote location but none of the examples I find cover receiving it.
Any help would be appreciated.
using Justinas' example and mixing with my existing experiment I've gotten this far but m.Post never seems to be called.
package main
import (
"fmt"
"io"
"net/http"
"os"
"github.com/codegangsta/martini"
"github.com/codegangsta/martini-contrib/render"
)
func main() {
m := martini.Classic()
m.Use(render.Renderer(render.Options{
Directory: "templates", // Specify what path to load the templates from.
Layout: "layout", // Specify a layout template. Layouts can call {{ yield }} to render the current template.
Charset: "UTF-8", // Sets encoding for json and html content-types.
}))
m.Get("/", func(r render.Render) {
fmt.Printf("%v\n", "g./")
r.HTML(200, "hello", "world")
})
m.Get("/:who", func(args martini.Params, r render.Render) {
fmt.Printf("%v\n", "g./:who")
r.HTML(200, "hello", args["who"])
})
m.Post("/up", func(w http.ResponseWriter, r *http.Request) {
fmt.Printf("%v\n", "p./up")
file, header, err := r.FormFile("file")
defer file.Close()
if err != nil {
fmt.Fprintln(w, err)
return
}
out, err := os.Create("/tmp/file")
if err != nil {
fmt.Fprintf(w, "Failed to open the file for writing")
return
}
defer out.Close()
_, err = io.Copy(out, file)
if err != nil {
fmt.Fprintln(w, err)
}
// the header contains useful info, like the original file name
fmt.Fprintf(w, "File %s uploaded successfully.", header.Filename)
})
m.Run()
}

Go's net/http server handles this pretty, using mime/multipart package behind the scenes. You only need to call r.FormFile() on your *http.Request to get a multipart.File back.
Here's a complete example. And the result of uploading a file with curl:
justinas#ubuntu /tmp curl -i -F file=#/tmp/stuff.txt http://127.0.0.1:8080/
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Date: Tue, 24 Dec 2013 20:56:07 GMT
Content-Length: 37
Content-Type: text/plain; charset=utf-8
File stuff.txt uploaded successfully.%
justinas#ubuntu /tmp cat file
kittens!

Related

Golang http serve file - giving parallel download support - contentLength & accept ranges not working

Very new to GoLang, less than 10 days. I have a http server & i need to http serve files which are inside disk. Here in default this is using "net/http" http.ServeFile(w, r, file). My problem is when i downloading these files, they don't have file pause/resume support but just downloading without showing total size. I tried adding "Content-Length" header & "Accept-Ranges" header. But seems not working.
Http Headers i worrying about are,
Content-Length
Content-Type
Accept-Ranges
Content-Disposition (attachment)
I have path to file, info FileInfo, w http.ResponseWriter, r http.Request before serving function.First I tried adding
w.Header().Set("Accept-Ranges", "bytes")
if w.Header().Get("Content-Encoding") == "" {
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10))
}
to
func (s *Server) serveFiles(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/download/") {
url := strings.TrimPrefix(r.URL.Path, "/download/")
//dldir is absolute
dldir := s.state.Config.DownloadDirectory
file := filepath.Join(dldir, url)
//only allow fetches/deletes inside the dl dir
if !strings.HasPrefix(file, dldir) || dldir == file {
http.Error(w, "Nice try\n"+dldir+"\n"+file, http.StatusBadRequest)
return
}
info, err := os.Stat(file)
if err != nil {
http.Error(w, "File stat error: "+err.Error(), http.StatusBadRequest)
return
}
switch r.Method {
case "GET":
if info.IsDir() {
w.Header().Set("Content-Type", "application/zip")
w.WriteHeader(200)
//write .zip archive directly into response
a := archive.NewZipWriter(w)
a.AddDir(file)
a.Close()
} else {
w.Header().Set("Accept-Ranges", "bytes")
if w.Header().Get("Content-Encoding") == "" {
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10))
}
http.ServeFile(w, r, file)
}
Then i can still see it is downloading without showing total size, doesn't have pause/resume support.
i tried to download files from
sample small file: https://s2.torrentfast.net/download/Dracula.2020.S01E01.HDTV.x264-PHOENiX[TGx]/[TGx]Downloaded%20from%20torrentgalaxy.to%20.txt
sample big fig: https://s2.torrentfast.net/download/Need%20For%20Speed%20Most%20Wanted%20Black%20Edition%20repack%20Mr%20DJ/Setup-1.bin
Http Get request response headers(sample small file) screenshot link
Can help?
w.Header().Set("Accept-Ranges", "bytes") is not required because Range will set http.ServeFile when responding.
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10)) is wrong, may respond to transmission, and http.ServerFile will set this header.
The meaning of Content-Length is to specify the length of the body, and Content-Range will record the section of the transmission range and the total length information. The correct method is to use http.ServeFile to send the file directly. The ServeFile function will automatically handle the situation of Range and Cache.
Just look at the source code of net/http/fs.go.

http.FileServer only serves index.html

My code for a simple file server:
package main
import (
"net/http"
"os"
"github.com/gorilla/handlers"
"github.com/gorilla/mux"
)
func main() {
r := mux.NewRouter()
// default file handler
r.Handle("/", http.FileServer(http.Dir("web")))
// run on port 8080
if err := http.ListenAndServe(":8080", handlers.LoggingHandler(os.Stdout, r)); err != nil {
panic(err)
}
}
My directory structure is:
cmd/server/main.go
web/index.html
web/favicon.ico
web/favicon.png
web/css/main.css
index.html asks for main.css. so when I run go run cmd/server/main.go I get the following output:
127.0.0.1 - - [24/Dec/2019:22:45:26 -0X00] "GET / HTTP/1.1" 304 0
127.0.0.1 - - [24/Dec/2019:22:45:26 -0X00] "GET /css/main.css HTTP/1.1" 404 19
I'm able to see the index.html page, but without the CSS. When I request any other file (e.g. favicon.ico) I also get a 404. Why does my FileServer only serve index.html?
To demonstrate why this does not work consider the following test app:
package main
import (
"fmt"
"net/http"
"github.com/gorilla/mux"
)
type testHandler struct{}
func (h *testHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
fmt.Printf("Got request for %s\n", r.URL)
}
func main() {
r := mux.NewRouter()
hndlr := testHandler{}
r.Handle("/", &hndlr)
// run on port 8080
if err := http.ListenAndServe(":8080", r); err != nil {
panic(err)
}
}
If you run this and access http://127.0.0.1:8080/ in your browser it will log Got request for /. However if you access http://127.0.0.1:8080/foo you will get a 404 error and nothing will be logged. This is because r.Handle("/", &hndlr) will only match on / and not anything below it.
If you change this to r.PathPrefix("/").Handler(&hndlr) then it will work as you expect (the router will pass all paths to the handler). So to fix your example change r.Handle("/", http.FileServer(http.Dir("web"))) to r.PathPrefix("/").Handler( http.FileServer(http.Dir("web"))).
Note: Because this is passing all paths to FileServer there is really no need to use Gorilla Mux; I have left this in on the assumption that you will be using it to add some other routes.

Reading non-utf8 encoded data from a network call in golang

I am trying to read bytes from http response body in golang. My problem is that the response body is encoded using ISO-8859-1. I want to read the response body in the same encoding and write the contents to a file in the ISO-8859-1 encoding.
Is there a way using which I can accomplish this? I don't want to convert the data into UTF-8 at all.
Here is a good read about encoding, which you might benefit from.
You are seemingly assuming Go decodes the raw bytes it receives when it performs a request. It does not.
Take this example:
package main
import (
"io"
"log"
"net/http"
"os"
)
func main() {
// We perform a request to a Latin-1 encoded page
resp, err := http.Get("http://andrew.triumf.ca/multilingual/samples/german.meta.html")
if err != nil {
log.Fatalln(err)
}
//
f, err := os.Create("/tmp/latin1")
defer f.Close()
if err != nil {
log.Fatalln(err)
}
io.Copy(f, resp.Body)
}
In the documentation, you can read that resp.Body conforms to the io.ReadCloser interface, which allows you to read the raw bytes and stream them to a file.
Once we run this code, this is the output of file -i /tmp/latin1:
/tmp/latin1: text/html; charset=iso-8859-1
Read and write the response body as a slice of bytes, []byte, an opaque data type.

Unexpected EOF using Go http client

I am learning Go and came across this problem.
I am just downloading web page content using HTTP client:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
)
func main() {
client := &http.Client{}
req, err := http.NewRequest("GET", "https://mail.ru/", nil)
req.Close = true
response, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer response.Body.Close()
content, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(content)[:100])
}
I get an unexpected EOF error when reading response body. At the same time content variable has full page content.
This error appear only when I downloading https://mail.ru/ content. With other URLs everything works fine - without any errors.
I used curl for downloading this page content - everything works as expected.
I am confused a bit - what's happening here?
Go v1.2, tried on Ubuntu and MacOS X
It looks like the that server (Apache 1.3, wow!) is serving up a truncated gzip response. If you explicitly request the identity encoding (preventing the Go transport from adding gzip itself), you won't get the ErrUnexpectedEOF:
req.Header.Add("Accept-Encoding", "identity")

Reading image from HTTP request's body in Go

I'm playing with Go (first time ever) and I want to build a tool to retrieve images from Internet and cut them (even resize) but I'm stuck on the first step.
package main
import (
"fmt"
"http"
)
var client = http.Client{}
func cutterHandler(res http.ResponseWriter, req *http.Request) {
reqImg, err := client.Get("http://www.google.com/intl/en_com/images/srpr/logo3w.png")
if err != nil {
fmt.Fprintf(res, "Error %d", err)
return
}
buffer := make([]byte, reqImg.ContentLength)
reqImg.Body.Read(buffer)
res.Header().Set("Content-Length", fmt.Sprint(reqImg.ContentLength)) /* value: 7007 */
res.Header().Set("Content-Type", reqImg.Header.Get("Content-Type")) /* value: image/png */
res.Write(buffer)
}
func main() {
http.HandleFunc("/cut", cutterHandler)
http.ListenAndServe(":8080", nil) /* TODO Configurable */
}
I'm able to request an image (let's use Google logo) and to get its kind and size.
Indeed, I'm just re-writing the image (look at this as a toy "proxy"), setting Content-Length and Content-Type and writing the byte slice back but I get it wrong somewhere. See how it looks the final image rendered on Chromium 12.0.742.112 (90304):
Also I checked the downloaded file and it is a 7007 bytes PNG image. It should be working properly if we look at the request:
GET /cut HTTP/1.1
User-Agent: curl/7.22.0 (i486-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.23 libssh2/1.2.8 librtmp/2.3
Host: 127.0.0.1:8080
Accept: /
HTTP/1.1 200 OK
Content-Length: 7007
Content-Type: image/png
Date: Tue, 27 Dec 2011 19:51:53 GMT
[PNG data]
What do you think I'm doing wrong here?
Disclaimer: I'm scratching my own itch, so probably I'm using the wrong tool :) Anyway, I can implement it on Ruby but before I would like to give Go a try.
Update: still scratching itches but... I think this is going to be a good side-of-side project so I'm opening it https://github.com/imdario/go-lazor If it is not useful, at least somebody can find usefulness with the references used to develop it. They were for me.
I think you went too fast to the serve things part.
Focus on the first step, downloading the image.
Here you have a little program that downloads that image to memory.
It works on my 2011-12-22 weekly version, for r60.3 you just need to gofix the imports.
package main
import (
"log"
"io/ioutil"
"net/http"
)
const url = "http://www.google.com/intl/en_com/images/srpr/logo3w.png"
func main() {
// Just a simple GET request to the image URL
// We get back a *Response, and an error
res, err := http.Get(url)
if err != nil {
log.Fatalf("http.Get -> %v", err)
}
// We read all the bytes of the image
// Types: data []byte
data, err = ioutil.ReadAll(res.Body)
if err != nil {
log.Fatalf("ioutil.ReadAll -> %v", err)
}
// You have to manually close the body, check docs
// This is required if you want to use things like
// Keep-Alive and other HTTP sorcery.
res.Body.Close()
// You can now save it to disk or whatever...
ioutil.WriteFile("google_logo.png", data, 0666)
log.Println("I saved your image buddy!")
}
Voilá!
This will get the image to memory inside data.
Once you have that, you can decode it, crop it and serve back to the browser.
Hope this helps.
I tried your code and noticed that the image you were serving was the right size, but the contents of the file past a certain point were all 0x00.
Review the io.Reader documentation. The important thing to remember is that Read reads up to the number of bytes you request. It can read fewer with no error returned. (You should be checking the error too, but that's not an issue here.)
If you want to make sure your buffer is completely full, use io.ReadFull. In this case it's simpler to just copy the entire contents of the Reader with io.Copy.
It's also important to remember to close HTTP request bodies.
I would rewrite the code this way:
package main
import (
"fmt"
"http"
"io"
)
var client = http.Client{}
func cutterHandler(res http.ResponseWriter, req *http.Request) {
reqImg, err := client.Get("http://www.google.com/intl/en_com/images/srpr/logo3w.png")
if err != nil {
fmt.Fprintf(res, "Error %d", err)
return
}
res.Header().Set("Content-Length", fmt.Sprint(reqImg.ContentLength))
res.Header().Set("Content-Type", reqImg.Header.Get("Content-Type"))
if _, err = io.Copy(res, reqImg.Body); err != nil {
// handle error
}
reqImg.Body.Close()
}
func main() {
http.HandleFunc("/cut", cutterHandler)
http.ListenAndServe(":8080", nil) /* TODO Configurable */
}

Resources