Golang File/Directory Walker adding files multiple times - recursion

So I am using the filepath.Walk function to monitor for changes in files recursively. fsnotify can't do recursive out of the box. I set up a Goroutine to monitor for changes, and then I add the paths to the watcher in the Walk() function.
func (w Watch) walkDirectories(fp string) {
error := filepath.Walk(fp, func(path string, info os.FileInfo, err error) error {
// skip files
if info == nil {
log.Fatalf("wrong watcher package: %s", path)
}
if !info.IsDir() {
return nil
}
if len(path) > 1 && strings.HasPrefix(filepath.Base(path), ".") {
return filepath.SkipDir
}
log.Println("filepath: ", filepath)
w.W.Add(path)
return err
})
log.Println("error: ", error)
}
I have a custom Struct that holds a Watcher, so that I can easily add paths for it to watch. You can see it being used here: w.W.Add(path). It all works great, except that files in the top level directory seem to be added twice to the watcher, or at least my hypothesis is "as many times as their are directory levels below the top level". My directory structure is such that:
.
├── README.md
├── languages.go
├── languages.json
├── librarymonitor.go
├── telemetryClient
└── testfiles
├── test.go
├── test.c
├── test.java
If I change a file in the testfiles directory, I get one "notification" from the watcher. If I change a file in the root, I get two. Can anyone shed light on this?
Thanks

Check your main code, this works fine, (try The Go Playground):
package main
import (
"fmt"
"os"
"path/filepath"
"reflect"
"time"
)
func main() {
rootDir := ".."
pattern := "*"
dirs, err := GetDirectories(rootDir, pattern)
if err != nil {
panic(err)
}
ticker := time.NewTicker(1 * time.Second)
for i := 1; i < 10; i++ {
<-ticker.C
dirs2, err := GetDirectories(rootDir, pattern)
//fmt.Println(dirs2)
if err != nil {
panic(err)
}
if !reflect.DeepEqual(dirs, dirs2) {
fmt.Println("Dir Changed: ", len(dirs), len(dirs2))
dirs = dirs2
}
}
ticker.Stop()
fmt.Println("Done")
}
// Returns the names of the subdirectories (including their paths)
// that match the specified search pattern in the specified directory.
func GetDirectories(root, pattern string) ([]string, error) {
dirs := make([]string, 0, 144)
return dirs, filepath.Walk(root, func(path string, fi os.FileInfo, err error) error {
if !fi.IsDir() {
return nil
}
matched, err := filepath.Match(pattern, fi.Name())
if err != nil {
return err
}
if !matched {
return nil
}
dirs = append(dirs, path)
return nil
})
}
Sample output ( with one new dir):
Dir Changed: 16 17
Done

Related

Remove the .html extension from every file in a simple HTTP server

I want to make it so when someone visits a page on my Go HTTP server, they won't see the .html extension.
E.g. when they visit https://example.org/test they will see the content of https://example.org/test.html.
My code:
package main
import (
"net/http"
)
func main() {
fs := http.FileServer(http.Dir("public/"))
http.Handle("/", http.StripPrefix("/", fs))
http.ListenAndServe(":8000", nil)
}
One option is to implement http.FileSystem using http.Dir. The advantage of this approach is that it takes advantage of the carefully written code in http.FileServer.
It will look something like this:
type HTMLDir struct {
d http.Dir
}
func main() {
fs := http.FileServer(HTMLDir{http.Dir("public/")})
http.Handle("/", http.StripPrefix("/", fs))
http.ListenAndServe(":8000", nil)
}
The implementation of the Open method depends on the application requirements.
If you always want to tack on the .html extension, then use this code:
func (d HTMLDir) Open(name string) (http.File, error) {
return d.d.Open(name + ".html")
}
If you want to fallback to the .html extension, then use this code:
func (d HTMLDir) Open(name string) (http.File, error) {
// Try name as supplied
f, err := d.d.Open(name)
if os.IsNotExist(err) {
// Not found, try with .html
if f, err := d.d.Open(name + ".html"); err == nil {
return f, nil
}
}
return f, err
}
Flip the previous one around to start with the .html extension and fallback to the name as supplied:
func (d HTMLDir) Open(name string) (http.File, error) {
// Try name with added extension
f, err := d.d.Open(name + ".html")
if os.IsNotExist(err) {
// Not found, try again with name as supplied.
if f, err := d.d.Open(name); err == nil {
return f, nil
}
}
return f, err
}
So basically you want the http.FileServer functionality, but you don't want clients having to enter the trailing .html extension.
Another simple solution is to add it yourself, at the server side. This is how it can be done:
fs := http.FileServer(http.Dir("public"))
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
r.URL.Path += ".html"
fs.ServeHTTP(w, r)
})
panic(http.ListenAndServe(":8000", nil))
And that's all.
If you want this file server to also serve other files (such as images and CSS files), only append the .html extension if it has no extension:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if ext := path.Ext(r.URL.Path); ext == "" {
r.URL.Path += ".html"
}
fs.ServeHTTP(w, r)
})
It's possible, but it cannot be done by serving the files using http.FileServer().
Instead, create a custom handler for the / route. Inside the handler, serve the requested file directly using http.ServeFile().
viewPath := "public/"
http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// hack, if requested url is / then point towards /index
if r.URL.Path == "/" {
r.URL.Path = "/index"
}
requestedPath := strings.TrimLeft(filepath.Clean(r.URL.Path), "/")
filename := fmt.Sprintf("%s/%s.html", viewPath, requestedPath)
http.ServeFile(w, r, filename)
}))
http.ListenAndServe(":8000", nil)
The .html suffix is added to every request path, so it'll point correctly towards the html files.
path / -> ./public/index.html
path /index -> ./public/index.html
path /some/folder/about -> ./public/some/folder/about.html
...

Good way to disable directory listing with http.FileServer in Go

If you use the http.FileServer in Go like:
func main() {
port := flag.String("p", "8100", "port to serve on")
directory := flag.String("d", ".", "the directory of static file to host")
flag.Parse()
http.Handle("/", http.FileServer(http.Dir(*directory)))
log.Printf("Serving %s on HTTP port: %s\n", *directory, *port)
log.Fatal(http.ListenAndServe(":"+*port, nil))
}
Then accessing a directory will give you a listing of files. Often this is disabled for web services and instead responds with 404 and I would like this behaviour too.
http.FileServer has no options for this AFAIK and I have seen a proposed way to solve this here https://groups.google.com/forum/#!topic/golang-nuts/bStLPdIVM6w what they do is wrapping the http.FileSystem type and implementing an own Open method. However this doesn't give a 404 when the path is a directory, it just gives a blank page, and it's unclear how to modify it to accomodate this. This is what they do:
type justFilesFilesystem struct {
fs http.FileSystem
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil {
return nil, err
}
return neuteredReaddirFile{f}, nil
}
type neuteredReaddirFile struct {
http.File
}
func (f neuteredReaddirFile) Readdir(count int) ([]os.FileInfo, error) {
return nil, nil
}
func main() {
fs := justFilesFilesystem{http.Dir("/tmp/")}
http.ListenAndServe(":8080", http.FileServer(fs))
}
Note: if you make Readdir return nil, os.ErrNotExist then you get a 500 response with "Error reading directory" - not 404.
Any ideas on how to neatly present a 404 and still preserving the feature of automatically finding an index.html if present?
This behavior can be changed if you substitute not a Readdir method, but the Stat.
Please take a look at working code below. It supports serving of index.html files if they are inside of requested directory and returns 404 in case there is no index.html and it is a directory.
package main
import (
"io"
"net/http"
"os"
)
type justFilesFilesystem struct {
fs http.FileSystem
// readDirBatchSize - configuration parameter for `Readdir` func
readDirBatchSize int
}
func (fs justFilesFilesystem) Open(name string) (http.File, error) {
f, err := fs.fs.Open(name)
if err != nil {
return nil, err
}
return neuteredStatFile{File: f, readDirBatchSize: fs.readDirBatchSize}, nil
}
type neuteredStatFile struct {
http.File
readDirBatchSize int
}
func (e neuteredStatFile) Stat() (os.FileInfo, error) {
s, err := e.File.Stat()
if err != nil {
return nil, err
}
if s.IsDir() {
LOOP:
for {
fl, err := e.File.Readdir(e.readDirBatchSize)
switch err {
case io.EOF:
break LOOP
case nil:
for _, f := range fl {
if f.Name() == "index.html" {
return s, err
}
}
default:
return nil, err
}
}
return nil, os.ErrNotExist
}
return s, err
}
func main() {
fs := justFilesFilesystem{fs: http.Dir("/tmp/"), readDirBatchSize: 2}
fss := http.FileServer(fs)
http.ListenAndServe(":8080", fss)
}

Creating a tar file to be served via Google App Engine

I'm writing an application for Google AppEngine using Go, and need to tar together a bunch of files to serve to the user when they navigate to a particular URL. At the moment the files are static, and so I could solve this problem by tarring them before upload to the server. In the future I would like to dynamically alter them before tarring, and so would like to learn how to tar & serve the static files on request.
In my init() function I have the following line:
http.HandleFunc("/download.tar", tarit)
The function tarit is the one I am having a problem with, and it currently looks like the following:
func tarit(w http.ResponseWriter, r *http.Request) {
tarball := tar.NewWriter(w)
defer tarball.Close()
info, err := os.Stat("/files")
if err != nil {
return
}
var baseDir string
if info.IsDir() {
baseDir = filepath.Base("/files")
}
filepath.Walk("/files", func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
if baseDir != "" {
header.Name = filepath.Join(baseDir, strings.TrimPrefix(path, "/files"))
}
if err := tarball.WriteHeader(header); err != nil {
return err
}
if info.IsDir() {
return nil
}
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
_, err = io.Copy(tarball, file)
return err
})
}
The files I am trying to add to the tarball are located in /files, and I've added this folder as a static_dir in the app.yaml.
When navigating to the appropriate URL, the browser downloads a tar file that is only 1 KB in size, and appears to be empty.
I would very much appreciate if someone could point out where I am going wrong, or what I am misunderstanding. I'd also be very happy to provide any other details that you would like.
Thanks!
Specify directories using paths relative to the directory containing app.yaml. The path in the posted code is the absolute path "/files". Perhaps you should change it to "files".
Log the errors returned from os.Stat and filepath.Walk. The errors will probably lead you to the problem.

How to remove the index.html path from url in golang

How do I remove the index.html from my URL bar e.g. localhost:8000/index.html
package main
import (
"net/http"
"io/ioutil"
)
func main() {
http.Handle("/", new(MyHandler))
http.ListenAndServe(":8000", nil)
}
type MyHandler struct {
http.Handler
}
func (this *MyHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) {
path := "public" + req.URL.Path
data, err := ioutil.ReadFile(string(path))
if err == nil {
w.Write(data)
} else {
w.WriteHeader(404)
w.Write([]byte("404 - " + http.StatusText(404)))
}
}
Add a condition to serve index.html if the URL path is empty:
path := "public"
if req.URL.Path == "/" {
path += "/index.html"
} else {
path += req.URL.Path
}
Also, it would be a good idea to use net/http.ServeFile over manually writing the data to the output stream (see net/http#ServeContent's documentation to learn why this is a good idea).
It's also worth noting that a built-in handler for serving files exists.

Using FileServer to serve my single html page

I'm trying to build a sample web application demonstrating rest techniques using go at the back-end, serving json based requests and javascript, jquery in the front-end (I'm not using html/template package).
FileServer "returns a handler that serves HTTP requests with the contents of the file system rooted at root."
supose that I'm publishing my static folder that contains index.html and scripts folder holding some javascript files.
How can I prevent the client from viewing my js files (publishing just the index.html at /) ?
You can easily restrict the FileServer, which is a HttpHandler by wrapping another HttpHandler around that. For example, take this wrapper which ONLY allows *.js files to be served:
func GlobFilterHandler(h http.Handler, pattern string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
path := r.URL.Path
fileName := filepath.Base(path)
if ok, err := filepath.Match(pattern, fileName); !ok || err != nil {
if err != nil {
log.Println("Error in pattern match:", err)
}
http.NotFound(w, r)
return
}
h.ServeHTTP(w, r)
})
}
func main() {
fileHandler := http.FileServer(http.Dir("/tmp/dtest"))
wrappedHandler := GlobFilterHandler(fileHandler, "*.js")
}
You can find a blog post here which describes the basic idea pretty good.
Another option you have is to extend on http.Dir and make your own http.FileSystem implementation which does exactly what you want:
type GlobDir struct {
Dir http.Dir
Pattern string
}
func (d GlobDir) Open(name string) (http.File, error) {
baseName := filepath.Base(name)
if ok, err := filepath.Match(d.Pattern, baseName); !ok || err != nil {
if err != nil {
return nil, err
}
return nil, fmt.Errorf("%s not match GlobDir pattern.", baseName)
}
return d.Dir.Open(name)
}
func main() {
fileHandler := http.FileServer(GlobDir{
Dir: http.Dir("/tmp/dtest"),
Pattern: "*.js",
})
http.ListenAndServe(":8080", fileHandler)
}
The second solution implements the http.FileSystem interface which is accepted by http.FileServer.
It checks whether the input file name matches the supplied pattern and then hands control down to the original http.Dir. This is probably the way you want to go here.

Resources