Parse Content Disposition header in GO - http

I am trying to retrieve the filename from this http writer for testing purposes.
On the server I have:
func servefile(w http.ResponseWriter, r *http.Request ) {
...
// file.Name() is randomized with os.CreateTemp(dir, temp+"*"+ext) above
w.Header().Set("Content-Disposition", "attachment; filename="+file.Name())
w.Header().Set("Content-Type", "application/octet-stream")
http.ServeFile(w, r, file.Name()+".xyz") // serve file to user to download
...
}
*I put .xyz as a place holder for this demonstration
I am testing this function programatically with Go and I want to access the filename to able to save it in a variable in the client code.
I have looked at this post How can I parse the Content-Disposition header to retrieve the filename property? , but I have not gotten it to work. I have no clue what the filename is on the client side so I don't know how to reference it specifically. I know my code on the server side works, because when I send a request (through the browser) to this endpoint/function, the "Downloads" popup shows the file download progress with the name of the file.
EDIT** This is the client code I am calling it from:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
// Create request to postRecord endpoint
req, err := http.NewRequest("POST", "/PostRecord?CID="+cid, body)
check(err)
req.Header.Add("Content-Type", w.FormDataContentType())
// execute request to PostRecord endpoint. returns an httptest.ResponseRecorder
respRecorder := executeRequest(PostRecord, req)
disposition, params, err := mime.ParseMediaType(`Content-Disposition`)
...
}

Based on #BaytaDarrell's comment. It dawned on me that I could print out the responses. That helped me realize that I was trying to find the content-disposition after the wrong request/response. The linked post still didn't help, but I got my code working like this:
func TestGetFile(t *testing.T) {
...
cid := "some string"
// requestfile() creates, executes, and returns an httptest.ResponseRecorder to the requestFile endpoint
reqfileRespRecorder := requestfile()
disp := reqfileRespRecorder.Header().Get("Content-Disposition")
line := strings.Split(disp, "=")
filename := line[1]
fmt.Println("filename: ", filename)
// createTmpFile creates a new file out of the contents recieved in requestfile()
filePath := "/tmp/temp.xyz"
file := createTmpFile(reqfileRespRecorder , filePath)
// CreateWriter() - writes file contents to body of multipart.Writer
w, body := createWriter(file)
...
}
Their comment realized I should re-look at the httptest package documentation. Here I found the Header() function and that I can use it to look at the header with Get().
This line reqfileRespRecorder.Header().Get("Content-Disposition") returns attachment; filename=temp37bf73gd.xyz and to store the filename in a variable I split on =.

Related

Unable to extract value from r.PostFormValue in Go?

I'm trying to extract a value from an HTTP POST request body (in my simple Go HTTP server) using the net/http PostFormValue and my output is an empty string when I'm looking for the any key in general, but in my case trying to fetch the hub.secret for use in a HMAC check. I use Postman to send the request to my localhost:8080 instance using the Gorilla/mux router, with header Content-Type: application/x-www-form-urlencoded set.
My handler looks like so:
func rootPostHandler(w http.ResponseWriter, r *http.Request) {
var expectedMac []byte
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
log.Println("r.Body is:", string(body)) // debug: print the request POST body
message := body // debug: set message just for extra clarity
errParse := r.ParseForm()
if errParse != nil {
// handle err
}
secret := []byte(r.PostFormValue("hub.secret"))
log.Println("secret is: ", string(secret))
mac := hmac.New(sha256.New, secret)
mac.Write(message)
expectedMac = mac.Sum(nil)
fmt.Println("Is HMAC equal? ", hmac.Equal(message, expectedMac))
w.Header().Add("X-Hub-Signature", "sha256="+string(message))
}
The r.Body:
hub.callback=http%253A%252F%252Fweb-sub-client%253A8080%252FbRxvcmOcNk&hub.mode=subscribe&hub.secret=xTgSGLOtPNrBLLgYcKnL&hub.topic=%252Fa%252Ftopic
And the output for print secret etc is empty string, meaning it can't find hub.secret, right? What am I missing here?
The application reads the request body to EOF on this line:
body, err := ioutil.ReadAll(r.Body)
ParseForm returns an empty form because the body is at EOF at this line:
errParse := r.ParseForm()
The request body is read from the network connection. The request body cannot be read a second time.
Remove the call to ioutil.ReadAll or create a new body reader using the data returned from ioutil.ReadAll:
r.Body = io.NopCloser(bytes.NewReader(body))

Golang http serve file - giving parallel download support - contentLength & accept ranges not working

Very new to GoLang, less than 10 days. I have a http server & i need to http serve files which are inside disk. Here in default this is using "net/http" http.ServeFile(w, r, file). My problem is when i downloading these files, they don't have file pause/resume support but just downloading without showing total size. I tried adding "Content-Length" header & "Accept-Ranges" header. But seems not working.
Http Headers i worrying about are,
Content-Length
Content-Type
Accept-Ranges
Content-Disposition (attachment)
I have path to file, info FileInfo, w http.ResponseWriter, r http.Request before serving function.First I tried adding
w.Header().Set("Accept-Ranges", "bytes")
if w.Header().Get("Content-Encoding") == "" {
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10))
}
to
func (s *Server) serveFiles(w http.ResponseWriter, r *http.Request) {
if strings.HasPrefix(r.URL.Path, "/download/") {
url := strings.TrimPrefix(r.URL.Path, "/download/")
//dldir is absolute
dldir := s.state.Config.DownloadDirectory
file := filepath.Join(dldir, url)
//only allow fetches/deletes inside the dl dir
if !strings.HasPrefix(file, dldir) || dldir == file {
http.Error(w, "Nice try\n"+dldir+"\n"+file, http.StatusBadRequest)
return
}
info, err := os.Stat(file)
if err != nil {
http.Error(w, "File stat error: "+err.Error(), http.StatusBadRequest)
return
}
switch r.Method {
case "GET":
if info.IsDir() {
w.Header().Set("Content-Type", "application/zip")
w.WriteHeader(200)
//write .zip archive directly into response
a := archive.NewZipWriter(w)
a.AddDir(file)
a.Close()
} else {
w.Header().Set("Accept-Ranges", "bytes")
if w.Header().Get("Content-Encoding") == "" {
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10))
}
http.ServeFile(w, r, file)
}
Then i can still see it is downloading without showing total size, doesn't have pause/resume support.
i tried to download files from
sample small file: https://s2.torrentfast.net/download/Dracula.2020.S01E01.HDTV.x264-PHOENiX[TGx]/[TGx]Downloaded%20from%20torrentgalaxy.to%20.txt
sample big fig: https://s2.torrentfast.net/download/Need%20For%20Speed%20Most%20Wanted%20Black%20Edition%20repack%20Mr%20DJ/Setup-1.bin
Http Get request response headers(sample small file) screenshot link
Can help?
w.Header().Set("Accept-Ranges", "bytes") is not required because Range will set http.ServeFile when responding.
w.Header().Set("Content-Length", strconv.FormatInt(info.Size(), 10)) is wrong, may respond to transmission, and http.ServerFile will set this header.
The meaning of Content-Length is to specify the length of the body, and Content-Range will record the section of the transmission range and the total length information. The correct method is to use http.ServeFile to send the file directly. The ServeFile function will automatically handle the situation of Range and Cache.
Just look at the source code of net/http/fs.go.

Is it possible to run http.ListenAndServe() AND ReadFromUDP() concurrently?

I am trying to write a simple web app that will listen for UDP packets.
But I can either only listen for UDP packets, or run the web app...
I am not familiar with GoLang, but here's the code I'm using to...
listen for UDP:
ServerConn, _ := net.ListenUDP("udp", &net.UDPAddr{IP:[]byte{#,#,#,#},Port:####,Zone:""})
defer ServerConn.Close()
buf := make([]byte, 1024)
for {
n, addr, _ := ServerConn.ReadFromUDP(buf)
fmt.Println("Received ", string(buf[0:n]), " from ", addr)
}
Server logic:
package main
We import 4 important libraries
1. “net/http” to access the core go http functionality
2. “fmt” for formatting our text
3. “html/template” a library that allows us to interact with our html file.
4. "time" - a library for working with date and time.
import (
"net/http"
"fmt"
"time"
"html/template"
)
//Create a struct that holds information to be displayed in our HTML file
type Welcome struct {
Name string
Time string
}
//Go application entrypoint
func main() {
//Instantiate a Welcome struct object and pass in some random information.
//We shall get the name of the user as a query parameter from the URL
welcome := Welcome{"Anonymous", time.Now().Format(time.Stamp)}
//We tell Go exactly where we can find our html file. We ask Go to parse the html file (Notice
// the relative path). We wrap it in a call to template.Must() which handles any errors and halts if there are fatal errors
templates := template.Must(template.ParseFiles("templates/welcome-template.html"))
//Our HTML comes with CSS that go needs to provide when we run the app. Here we tell go to create
// a handle that looks in the static directory, go then uses the "/static/" as a url that our
//html can refer to when looking for our css and other files.
http.Handle("/static/", //final url can be anything
http.StripPrefix("/static/",
http.FileServer(http.Dir("static")))) //Go looks in the relative "static" directory first using http.FileServer(), then matches it to a
//url of our choice as shown in http.Handle("/static/"). This url is what we need when referencing our css files
//once the server begins. Our html code would therefore be <link rel="stylesheet" href="/static/stylesheet/...">
//It is important to note the url in http.Handle can be whatever we like, so long as we are consistent.
//This method takes in the URL path "/" and a function that takes in a response writer, and a http request.
http.HandleFunc("/" , func(w http.ResponseWriter, r *http.Request) {
//Takes the name from the URL query e.g ?name=Martin, will set welcome.Name = Martin.
if name := r.FormValue("name"); name != "" {
welcome.Name = name;
}
//If errors show an internal server error message
//I also pass the welcome struct to the welcome-template.html file.
if err := templates.ExecuteTemplate(w, "welcome-template.html", welcome); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
})
//Start the web server, set the port to listen to 8080. Without a path it assumes localhost
//Print any errors from starting the webserver using fmt
fmt.Println("Listening");
fmt.Println(http.ListenAndServe(":8080", nil));
}
taken from(https://medium.com/google-cloud/building-a-go-web-app-from-scratch-to-deploying-on-google-cloud-part-1-building-a-simple-go-aee452a2e654)
I tried putting both of these extracts in 1 file, as well as running 2 files at the same time using
go run *.go
Any help would be appreciated!
You're going to need to start looking into goroutines - since you're asking to do two things concurrently. I suggest doing some reading into channels, goroutines, and concurrency in general :)

No content in http.Request.Body when read using Golang

Note: All error handling code is omitted for simplicity. I handled them locally, and no errors were produced.
In Golang, I tried to use the code below to read from a http.Request.Body from a POST request:
func readBody(req *http.Request) string {
bytes, _ := httputils.DumpRequestOut(req, true)
return string(bytes)
}
It shows a non-zero Content-Length, but no content returned:
ContentLength=413 with Body length 0
. I tried with the code below, also no luck:
func readBody(req *http.Request) string {
bytes, _ := ioutil.ReadAll(req.Body)
return string(bytes)
}
It returns an empty string. After googling, I found a blog about this problem: Golang: Read from an io.ReadWriter without losing its content. I tried to follow the pattern, still no luck:
func readBody(req *http.Request) string {
bodyBytes, _ := ioutil.ReadAll(req.Body)
// Restore the io.ReadCloser to its original state
req.Body = ioutil.NopCloser(bytes.NewBuffer(bodyBytes))
// Use the content
return string(bodyBytes)
}
Any suggestions? Thanks in advance :)
If the request does not include an appropriate "Content-Type" header you may see a 0 length body when you try to read it.
This is happening because you call httputils.DumpRequestOut(req, true) after the http client has performed the request and the body is drained.
Make sure you do the following step in order:
httputils.DumpRequestOut(req, true)
resp, err :=http.DefaultClient.Do(req)
httputil.DumpResponse(res, true)

Golang writing to http response breaks input reading?

I'm attempting to write a small webapp in Go where the user uploads a gzipped file in a multipart form. The app unzips and parses the file and writes some output to the response. However, I keep running into an error where the input stream looks corrupted when I begin writing to the response. Not writing to the response fixes the problem, as does reading from a non-gzipped input stream. Here's an example http handler:
func(w http.ResponseWriter, req *http.Request) {
//Get an input stream from the multipart reader
//and read it using a scanner
multiReader, _ := req.MultipartReader()
part, _ := multiReader.NextPart()
gzipReader, _ := gzip.NewReader(part)
scanner := bufio.NewScanner(gzipReader)
//Strings read from the input stream go to this channel
inputChan := make(chan string, 1000)
//Signal completion on this channel
donechan := make(chan bool, 1)
//This goroutine just reads text from the input scanner
//and sends it into the channel
go func() {
for scanner.Scan() {
inputChan <- scanner.Text()
}
close(inputChan)
}()
//Read lines from input channel. They all either start with #
//or have ten tab-separated columns
go func() {
for line := range inputChan {
toks := strings.Split(line, "\t")
if len(toks) != 10 && line[0] != '#' {
panic("Dang.")
}
}
donechan <- true
}()
//periodically write some random text to the response
go func() {
for {
time.Sleep(10*time.Millisecond)
w.Write([]byte("write\n some \n output\n"))
}
}()
//wait until we're done to return
<-donechan
}
Weirdly, this code panics every time because it always encounters a line with fewer than 10 tokens, although at different spots every time. Commenting out the line that writes to the response fixes the issue, as does reading from a non-gzipped input stream. Am I missing something obvious? Why would writing to the response break if reading from a gzip file, but not a plain text formatted file? Why would it break at all?
The HTTP protocol is not full-duplex: it is request-response based. You should only send output once you're done with reading the input.
In your code you use a for with range on a channel. This will try to read the channel until it is closed, but you never close the inputChan.
If you never close inputChan, the following line is never reached:
donechan <- true
And therefore receiving from donechan blocks:
<-donechan
You have to close the inputChan when EOF is reached:
go func() {
for scanner.Scan() {
inputChan <- scanner.Text()
}
close(inputChan) // THIS IS NEEDED
}()

Resources