I'm writing an application for Google AppEngine using Go, and need to tar together a bunch of files to serve to the user when they navigate to a particular URL. At the moment the files are static, and so I could solve this problem by tarring them before upload to the server. In the future I would like to dynamically alter them before tarring, and so would like to learn how to tar & serve the static files on request.
In my init() function I have the following line:
http.HandleFunc("/download.tar", tarit)
The function tarit is the one I am having a problem with, and it currently looks like the following:
func tarit(w http.ResponseWriter, r *http.Request) {
tarball := tar.NewWriter(w)
defer tarball.Close()
info, err := os.Stat("/files")
if err != nil {
return
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base("/files")
}
filepath.Walk("/files", func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, "/files"))
}
if err := tarball.WriteHeader(header); err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(tarball, file)
return err
})
}
The files I am trying to add to the tarball are located in /files, and I've added this folder as a static_dir in the app.yaml.
When navigating to the appropriate URL, the browser downloads a tar file that is only 1 KB in size, and appears to be empty.
I would very much appreciate if someone could point out where I am going wrong, or what I am misunderstanding. I'd also be very happy to provide any other details that you would like.
Thanks!
Specify directories using paths relative to the directory containing app.yaml. The path in the posted code is the absolute path "/files". Perhaps you should change it to "files".
Log the errors returned from os.Stat and filepath.Walk. The errors will probably lead you to the problem.
Related
I want to make it so when someone visits a page on my Go HTTP server, they won't see the .html extension.
E.g. when they visit https://example.org/test they will see the content of https://example.org/test.html.
My code:
package main
import (
"net/http"
)
func main() {
fs := http.FileServer(http.Dir("public/"))
http.Handle("/", http.StripPrefix("/", fs))
http.ListenAndServe(":8000", nil)
}
One option is to implement http.FileSystem using http.Dir. The advantage of this approach is that it takes advantage of the carefully written code in http.FileServer.
It will look something like this:
type HTMLDir struct {
d http.Dir
}
func main() {
fs := http.FileServer(HTMLDir{http.Dir("public/")})
http.Handle("/", http.StripPrefix("/", fs))
http.ListenAndServe(":8000", nil)
}
The implementation of the Open method depends on the application requirements.
If you always want to tack on the .html extension, then use this code:
func (d HTMLDir) Open(name string) (http.File, error) {
return d.d.Open(name + ".html")
}
If you want to fallback to the .html extension, then use this code:
func (d HTMLDir) Open(name string) (http.File, error) {
// Try name as supplied
f, err := d.d.Open(name)
if os.IsNotExist(err) {
// Not found, try with .html
if f, err := d.d.Open(name + ".html"); err == nil {
return f, nil
}
}
return f, err
}
Flip the previous one around to start with the .html extension and fallback to the name as supplied:
func (d HTMLDir) Open(name string) (http.File, error) {
// Try name with added extension
f, err := d.d.Open(name + ".html")
if os.IsNotExist(err) {
// Not found, try again with name as supplied.
if f, err := d.d.Open(name); err == nil {
return f, nil
}
}
return f, err
}
So basically you want the http.FileServer functionality, but you don't want clients having to enter the trailing .html extension.
Another simple solution is to add it yourself, at the server side. This is how it can be done:
fs := http.FileServer(http.Dir("public"))
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
r.URL.Path += ".html"
fs.ServeHTTP(w, r)
})
panic(http.ListenAndServe(":8000", nil))
And that's all.
If you want this file server to also serve other files (such as images and CSS files), only append the .html extension if it has no extension:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if ext := path.Ext(r.URL.Path); ext == "" {
r.URL.Path += ".html"
}
fs.ServeHTTP(w, r)
})
It's possible, but it cannot be done by serving the files using http.FileServer().
Instead, create a custom handler for the / route. Inside the handler, serve the requested file directly using http.ServeFile().
viewPath := "public/"
http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// hack, if requested url is / then point towards /index
if r.URL.Path == "/" {
r.URL.Path = "/index"
}
requestedPath := strings.TrimLeft(filepath.Clean(r.URL.Path), "/")
filename := fmt.Sprintf("%s/%s.html", viewPath, requestedPath)
http.ServeFile(w, r, filename)
}))
http.ListenAndServe(":8000", nil)
The .html suffix is added to every request path, so it'll point correctly towards the html files.
path / -> ./public/index.html
path /index -> ./public/index.html
path /some/folder/about -> ./public/some/folder/about.html
...
I want to listen to real time changes in firestore and I am also only allowed to use Go. Since firestore SDK for Go doesn't have any option to listen for real time changes, I decided to use the firestore v1beta1 sdk.
I have written the following code to do that
func TestRPCHandler(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
c, err := firestore.NewClient(context.Background())
databaseName := "projects/[project_name]/databases/(default)"
if err != nil {
panic(err)
}
stream, err := client.Listen(context.Background())
if err != nil {
panic(err)
}
request := &firestorepb.ListenRequest{
Database: databaseName,
TargetChange: &firestorepb.ListenRequest_AddTarget{
AddTarget: &firestorepb.Target{
TargetType: &firestorepb.Target_Documents{
Documents: &firestorepb.Target_DocumentsTarget{
Documents: []string{"projects/[project_name]/databases/(default)/[collection_name]"} ,
},
},
},
},
}
if err := stream.Send(request); err != nil {
panic(err)
}
if err := stream.CloseSend(); err != nil {
panic(err)
}
for {
resp, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
}
}
When I am doing this, the code does not detect any changes I bring about manually in the database. stream.Recv() just returns EOF and exits immediately. I even tried manually waiting by adding time.Sleep() but that does not help either.
You don't need the beta SDK or hacks to make this happen, I found the solution, it's pretty easy actually.
The https://firebase.google.com/docs/firestore/query-data/listen documentation does not contain an example for Go.
The source code of the firestore client API for Go has an unexported watchStream which we cannot directly use: https://github.com/googleapis/google-cloud-go/blob/master/firestore/watch.go#L130
Deep search of the repository shows that this is actually used on the DocumentSnapshotIterator and QuerySnapshotIterator at: https://github.com/googleapis/google-cloud-go/blob/master/firestore/docref.go#L644 and: https://github.com/googleapis/google-cloud-go/blob/master/firestore/query.go#L716.
The Collection contains a Snapshots method which returns the snapshot iterator that we want, after that all is easy, we just make an infivitive loop through its Next method.
Example:
cols, err := client.Collections(context.Background()).GetAll()
for _, col := range cols {
iter := col.Snapshots(context.Background())
defer iter.Stop()
for {
doc, err := iter.Next()
if err != nil {
if err == iterator.Done {
break
}
return err
}
for _, change := range doc.Changes {
// access the change.Doc returns the Document,
// which contains Data() and DataTo(&p) methods.
switch change.Kind {
case firestore.DocumentAdded:
// on added it returns the existing ones.
isNew := change.Doc.CreateTime.After(l.startTime)
// [...]
case firestore.DocumentModified:
// [...]
case firestore.DocumentRemoved:
// [...]
}
}
}
}
Yours, Gerasimos Maropoulos aka #kataras
Firebase's Get realtime updates with Cloud Firestore documentation currently indicates that Go is not yet supported.
// Not yet supported in Go client library
I have tried setting up a download server and a download client for individual files. How can I modify them to serve/download all the files from a directory?
Following are my server and client codes:
//server.go
func main() {
http.HandleFunc("/dlpath", handle)
err := http.ListenAndServe(":10001", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
func handle(writer http.ResponseWriter, r *http.Request) {
filename := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve\\send.txt"
http.ServeFile(writer, r, filename)
}
//client.go
func main() {
downloadFile("res_out.txt", "http://localhost:10001/dlpath")
}
func downloadFile(dirname string, url string) error {
// Create the file
out, err := os.OpenFile(dirname, os.O_WRONLY | os.O_CREATE | os.O_APPEND, 0666)
if err != nil {
fmt.Println(err)
}
defer out.Close()
// get data
request, err := http.NewRequest("GET", url, nil)
if err != nil {
fmt.Println(err)
}
client := http.Client{}
resp, err := client.Do(request)
if err != nil {
return err
}
defer resp.Body.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
return err
}
return nil
}
I tried serving the directory in the handle function like so:
dirname := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve"
http.FileServer(http.Dir(dirname))
and tried to print out the response on the client side, but I got nothing. How can I serve all the files from the /serve directory, and download them in the client?
EDIT:
Following are the contents of the serve directory:
serve
---sample.txt
---send.txt
---dir2
------abc.txt
How can I download all these files on the client side as separate files, with the directory structure intact?
Update: When I call the http.Handle function (as mentioned in the answer) directly in the main function, I am able to serve all the files, and the file within the inner directory too.
However, when I call the same within the handle function, it doesn't serve anything. I am guessing this has something to do with the path?
The problem can be in the file path you are requesting. It is prefixed with /dlpath/. You must strip this prefix and pass the rest of the text as a path. See: https://godoc.org/net/http#FileServer
Could you try this code snippet:
package main
import (
"log"
"net/http"
)
func main() {
dirName := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve"
http.Handle("/dlpath/", http.StripPrefix("/dlpath", http.FileServer(http.Dir(dirName))))
err := http.ListenAndServe(":8001", nil)
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
Hope this helps.
P.S.
This is from the case when you are serving the directory in the handler function.
The accepted answer and some amount of googling helped me write FileServer with a different handler (http.HandleFunc). Using HandleFunc allowed me to set up other things like authentication etc.,
The key was to call ServeHTTP()
So, the handler function can be modified in the following manner to serve all the files in a directory:
func handle(writer http.ResponseWriter, r *http.Request) {
{
dirName := "C:\\Users\\aarvi\\GolandProjects\\src\\Practice\\download\\serve"
http.StripPrefix("/dlpath", http.FileServer(http.Dir(dirName))).ServeHTTP(writer, r)
}
}
I'm working on an api endpoint in go that will accept an upload and then immediately forward to another API. I don't want to write the file to disk anywhere, but I'm not sure storing the file temporarily in memory the way I have is correct either. All the examples that I can find deal with saving the file to disk. I've posted what I'm doing below. The response I get back from the second API is that I failed to post a file, but I can see that it is receiving the "userID" field. Can someone please point out what I'm doing wrong as well as possibly advise if this is the best way to go about this?
Route Handler
func (r *Routes) forwardFile(w http.ResponseWriter, req *http.Request){
parameters := mux.Vars(req)
userID := parameters["userID"]
const maxFileSize = 1 * 1024 * 1024 // 1MB
// pull in the uploaded file into memory
req.ParseMultipartForm(maxFileSize)
file, fileHeader, err := req.FormFile("fileUpload")
if err != nil {
encodeResponse(w, req, response{obj: nil, err: err})
return
}
defer file.Close()
success, err := service.DoForwardFile(userID, file, fileHeader)
encodeResponse(w, req, response{obj: success, err: err})
}
Service Handler
func (b *base) DoForwardFile(userID int, file multipart.File, fileHeader *multipart.FileHeader) (FileForwardedResponse, error) {
// start building our request to forward the file
var resp *http.Response
defer func() {
if resp != nil {
resp.Body.Close()
}
reportStat.Complete(0)
}()
// build a form body
body := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(body)
// add form fields
bodyWriter.WriteField("userID", userID)
// add a form file to the body
fileWriter, err := bodyWriter.CreateFormFile("fileUpload", fileHeader.Filename)
if err != nil {
return FileForwardedResponse{}, err
}
// copy the file into the fileWriter
_, err = io.Copy(fileWriter, file)
if err != nil {
return FileForwardedResponse{}, err
}
// Close the body writer
bodyWriter.Close()
// build request url
apiURL := fmt.Sprintf("%s/v2/users/%d/files", config.APIURL, userID)
// send request
client := &http.Client{Timeout: time.Second * 10}
req, err := http.NewRequest("POST", apiURL, body)
resp, err = client.Do(req)
...
}
You're not setting the Content-Type for the request. Even if the header gets set automatically to multipart/form-data, it's missing the data boundary.
req, err := http.NewRequest("POST", uri, body)
if err != nil {
return FileForwardedResponse{}, err
}
req.Header.Set("Content-Type", bodyWriter.FormDataContentType())
...
I'm trying to build a sample web application demonstrating rest techniques using go at the back-end, serving json based requests and javascript, jquery in the front-end (I'm not using html/template package).
FileServer "returns a handler that serves HTTP requests with the contents of the file system rooted at root."
supose that I'm publishing my static folder that contains index.html and scripts folder holding some javascript files.
How can I prevent the client from viewing my js files (publishing just the index.html at /) ?
You can easily restrict the FileServer, which is a HttpHandler by wrapping another HttpHandler around that. For example, take this wrapper which ONLY allows *.js files to be served:
func GlobFilterHandler(h http.Handler, pattern string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
fileName := filepath.Base(path)
if ok, err := filepath.Match(pattern, fileName); !ok || err != nil {
if err != nil {
log.Println("Error in pattern match:", err)
}
http.NotFound(w, r)
return
}
h.ServeHTTP(w, r)
})
}
func main() {
fileHandler := http.FileServer(http.Dir("/tmp/dtest"))
wrappedHandler := GlobFilterHandler(fileHandler, "*.js")
}
You can find a blog post here which describes the basic idea pretty good.
Another option you have is to extend on http.Dir and make your own http.FileSystem implementation which does exactly what you want:
type GlobDir struct {
Dir http.Dir
Pattern string
}
func (d GlobDir) Open(name string) (http.File, error) {
baseName := filepath.Base(name)
if ok, err := filepath.Match(d.Pattern, baseName); !ok || err != nil {
if err != nil {
return nil, err
}
return nil, fmt.Errorf("%s not match GlobDir pattern.", baseName)
}
return d.Dir.Open(name)
}
func main() {
fileHandler := http.FileServer(GlobDir{
Dir: http.Dir("/tmp/dtest"),
Pattern: "*.js",
})
http.ListenAndServe(":8080", fileHandler)
}
The second solution implements the http.FileSystem interface which is accepted by http.FileServer.
It checks whether the input file name matches the supplied pattern and then hands control down to the original http.Dir. This is probably the way you want to go here.