It is so slow when using template package to generate a dynamic web page to client.
Testing code as below, golang 1.4.1
http.Handle("/js/", (http.FileServer(http.Dir(webpath))))
http.Handle("/css/", (http.FileServer(http.Dir(webpath))))
http.Handle("/img/", (http.FileServer(http.Dir(webpath))))
http.HandleFunc("/test", TestHandler)
func TestHandler(w http.ResponseWriter, r *http.Request) {
Log.Info("Entering TestHandler ...")
r.ParseForm()
filename := NiConfig.webpath + "/test.html"
t, err := template.ParseFiles(filename)
if err != nil {
Log.Error("template.ParseFiles err = %v", err)
}
t.Execute(w, nil)
}
According to the log, I found that it took about 3 seconds in t.Execute(w, nil), I do not know why it uses so much time. I also tried Apache server to test test.html, it responded very fast.
You should not parse templates every time you serve a request!
There is a significant time delay to read a file, parse its content and build the template. Also since templates do not change (varying parts should be parameters!) you only have to read and parse a template once.
Also parsing and creating the template each time when serving a request generates lots of values in memory which are then thrown away (because they are not reused) giving additional work for the garbage collector.
Parse the templates when your application starts, store it in a variable, and you only have to execute the template when a request comes in. For example:
var t *template.Template
func init() {
filename := NiConfig.webpath + "/test.html"
t = template.Must(template.ParseFiles(filename))
http.HandleFunc("/test", TestHandler)
}
func TestHandler(w http.ResponseWriter, r *http.Request) {
Log.Info("Entering TestHandler ...")
// Template is ready, just Execute it
if err := t.Execute(w, nil); err != nil {
log.Printf("Failed to execute template: %v", err)
}
}
Related
Is there a way to remove the whole static directory from the server once its content was served for one time? (with served I mean being displayed on the browser once).
func main() {
fs := http.FileServer(http.Dir(tempDir))
http.Handle("/", fs)
http.HandleFunc("/app/wo", workOrderApp)
log.Fatal(http.ListenAndServe(":"+os.Args[1], nil))
}
func workOrderApp(w http.ResponseWriter, r *http.Request) {
workOrderAppProcess(w)
time.Sleep(time.Duration(4 * time.Second)) //some time to let render the html
os.RemoveAll(tempDir)
}
The sleep os.RemoveAll was a hit and miss. Had to adjust the sleep time to a few seconds otherwise the file sometimes was served and sometimes not, I believe because bandwith or network related stuff. But it also had the side effect of delaying the whole rendering of the page.
In this example I remove all the directory, which is what I want.
func workOrderAppProcess(aid, date, language, token string, w http.ResponseWriter) {
zipDir := os.Args[2]
if _, err := os.Stat(tempDir); os.IsNotExist(err) {
log.Printf("Creating directory: %v", tempDir)
err := os.MkdirAll(tempDir, 0777)
if err != nil {
log.Print(err.Error())
}
}
log.Printf("Extracting file: %v to: %v", date+".zip", tempDir)
zipPath, _ := filepath.Abs(zipDir + "/" + date + ".zip")
app.ExtractZip(zipPath, tempDir)
batch := app.ReturnBatchNumber(tempDir + date)
typesData := app.ReturnWorkTypeData(app.ParseXML(tempDir + date + "/" + batch + "_type_list.xml"))
record := app.FindAppointmentRecord(aid, app.ParseXML(tempDir+date+"/"+batch+"_appt.xml"))
signatureFileURL := app.ReturnSignatureFileURL(tempDir+date, aid, date)
app.RenderTemplate(record, typesData, "template/wo.html", language, "/"+signatureFileURL, w)
}
Your code is causing the HTTP handler to wait 4 seconds, delete the files, then finalize the HTTP response. Just remove the sleep.
func workOrderApp(w http.ResponseWriter, r *http.Request) {
workOrderAppProcess(w)
os.RemoveAll(tempDir)
}
This is more efficient, more directly reflects your intention, and doesen't leave the HTTP connection open needlessly for an extra 4 seconds.
If you have other logic in your handler not shown, and you want to ensure that the delete happens in all cases, a defer can be useful:
func workOrderApp(w http.ResponseWriter, r *http.Request) {
workOrderAppProcess(w)
defer os.RemoveAll(tempDir)
/* Other logic that may do things */
}
After discussion in chat, it's apparent that workOrderAppProcess is rendering your HTML, and that os.RemoveAll is removing the images needed by that HTML. To solve this, you need to delay the removal, but after serving the HTML. This can be done with a simple goroutine:
func workOrderApp(w http.ResponseWriter, r *http.Request) {
workOrderAppProcess(w)
go func() {
time.Sleep(60 * time.Second)
os.RemoveAll(tempDir)
}()
}
My goal is to scrape a website that requires me to log in first using HTTP requests in Golang. I actually succeeded by finding out I can send a post request to the website writing form-data into the body of the request. When I test this through an API development software I use called Postman, the response is instantaneous with no delays. However, when performing the request with an HTTP client in Go, there is a consistent 60 second delay every single time. I end up getting a logged in page, but for my program I need the response to be nearly instantaneous.
As you can see in my code, I've tried adding a bunch of headers to the request like "Connection", "Content-Type", "User-Agent" since I thought maaaaaybe the website can tell I'm requesting from a program and is forcing me to wait 60 seconds for a response. Adding these headers to make my request more legitimate(?) doesn't work at all.
Is the delay coming from Go's HTTP client being slow or is there something wrong with how I'm forming my HTTP POST request? Also, was I on to something with my headers and HTTP client is rewriting them when they send out?
Here's my simple program...
package main
import (
"bytes"
"fmt"
"mime/multipart"
"net/http"
"net/http/cookiejar"
"os"
)
func main() {
url := "https://easypronunciation.com/en/log-in"
method := "POST"
payload := &bytes.Buffer{}
writer := multipart.NewWriter(payload)
_ = writer.WriteField("email", "foo#bar.com")
_ = writer.WriteField("password", "*********")
_ = writer.WriteField("persistent_login", "on")
_ = writer.WriteField("submit", "")
err := writer.Close()
if err != nil {
fmt.Println(err)
}
cookieJar, _ := cookiejar.New(nil)
client := &http.Client{
Jar: cookieJar,
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
fmt.Println(err)
}
req.Header.Set("Content-Type", writer.FormDataContentType())
req.Header.Set("Connection", "Keep-Alive")
req.Header.Set("Accept-Language", "en-US")
req.Header.Set("User-Agent", "Mozilla/5.0")
res, err := client.Do(req)
if err != nil {
fmt.Println(err)
}
defer res.Body.Close()
f, err := os.Create("response.html")
defer f.Close()
res.Write(f)
}
I doubt, this is the go client library too. I would suggest printing out the latencies for different components and see if/where the 60 second delay is. I would also replace and try different URLs instead
I'm trying to download a file from the web. It should be a simple processes. One that I've alredy done before. But, this particular link (a 135 kB zip file) gives me an error message: Get "http://www1.caixa.gov.br/loterias/_arquivos/loterias/D_megase.zip": stopped after 10 redirect. If I copy the link into the browser the file is downloaded without any issues, but when using the code below, the error pops up.
package main
import (
"io"
"net/http"
"os"
)
func main() {
link := "http://www1.caixa.gov.br/loterias/_arquivos/loterias/D_megase.zip"
resp, err := http.Get(link)
if err != nil {
panic(err)
}
defer resp.Body.Close()
// Create the file
out, err := os.Create("ms.zip")
if err != nil {
panic(err)
}
defer out.Close()
// Write the body to file
_, err = io.Copy(out, resp.Body)
if err != nil {
panic(err)
}
}
Any ideas on why does this happens and how to get around it?
Thanks for the attention.
After investigating this url I see that it sets cookie
Set-Cookie: security=true; path=/
You can set cookie manually, or implement CookieJar
c := http.Client{}
req, err := http.NewRequest("GET", link, nil)
if err != nil {
panic(err)
}
req.AddCookie(&http.Cookie{Name: "security", Value: "true", Path: "/"})
resp, err := c.Do(req)
if err != nil {
panic(err)
}
Your code is totally fine, but you'll often find this issue is more related to the source you're trying to download a file from, itself, rather than Go.
You would have had the same issue with other tools/languages, because the host you are trying to reach, keeps redirecting you because of an invalid 'User-Agent' header property. This is often the case when you want to allow your files to be downloadable only from 'browsers', rather than crawls, automated scripts etc.
With Go, you can add the header property with req.Header.Set("User-Agent", "<some-user-agent-value>"), before sending the request. You'd create an instance of request set the header, and execute it with a http.Client{} and client.Do(req).
Eg:
link := "http://www1.caixa.gov.br/loterias/_arquivos/loterias/D_megase.zip"
req, err := http.NewRequest("GET", link, nil)
if err != nil {
panic(err)
}
req.Header.Set("User-Agent", "Mozilla/4.0") // Doesn't even have to be a full
// proper user agent string
client := &http.Client{}
resp, err := client.Do(req)
You can read more in the Go's http pkg docs, it states that:
"For control over HTTP client headers, redirect policy, and other
settings, create a Client..."
Here's also the http.reqeust and http.client docs.
More about this ingeneral you can find in e.g. Mozilla's HTTP docs, as well as many other great docs and resources out there.
Btw. the zip archive you're trying to download seems like invalid. :-)
I am trying to implement http server that:
Calculate farther redirect using some logic
Redirect user
Log user data
The goal is to achieve maximum throughput (at least 15k rps). In order to do this, I want to save log asynchronously. I'm using kafka as logging system and separate logging block of code into separate goroutine. Overall example of current implementation:
package main
import (
"github.com/confluentinc/confluent-kafka-go/kafka"
"net/http"
"time"
"encoding/json"
)
type log struct {
RuntimeParam string `json:"runtime_param"`
AsyncParam string `json:"async_param"`
RemoteAddress string `json:"remote_address"`
}
var (
producer, _ = kafka.NewProducer(&kafka.ConfigMap{
"bootstrap.servers": "localhost:9092,localhost:9093",
"queue.buffering.max.ms": 1 * 1000,
"go.delivery.reports": false,
"client.id": 1,
})
topicName = "log"
)
func main() {
siteMux := http.NewServeMux()
siteMux.HandleFunc("/", httpHandler)
srv := &http.Server{
Addr: ":8080",
Handler: siteMux,
ReadTimeout: 2 * time.Second,
WriteTimeout: 5 * time.Second,
IdleTimeout: 10 * time.Second,
}
if err := srv.ListenAndServe(); err != nil {
panic(err)
}
}
func httpHandler(w http.ResponseWriter, r *http.Request) {
handlerLog := new(log)
handlerLog.RuntimeParam = "runtimeDataString"
http.Redirect(w, r, "http://google.com", 301)
go func(goroutineLog *log, request *http.Request) {
goroutineLog.AsyncParam = "asyncDataString"
goroutineLog.RemoteAddress = r.RemoteAddr
jsonLog, err := json.Marshal(goroutineLog)
if err == nil {
producer.ProduceChannel() <- &kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topicName, Partition: kafka.PartitionAny},
Value: jsonLog,
}
}
}(handlerLog, r)
}
The questions are:
Is it correct/efficient to use separate goroutine to implement async logging or should I use a different approach? (workers and channels for example)
Maybe there is a way to further improve performance of server, that I'm missing?
Yes, this is correct and efficient use of a goroutine (as Flimzy pointed in the comments). I couldn't agree more, this is a good approach.
The problem is that the handler may finish executing before the goroutine started processing everything and the request (which is a pointer) may be gone or you may have some races down the middleware stack. I read your comments, that it isn't your case, but in general, you shouldn't pass a request to a goroutine. As I can see from your code, you're really using only RemoteAddr from the request and why not to redirect straight away and put logging in the defer statement? So, I'd rewrite your handler a bit:
func httpHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "http://google.com", 301)
defer func(runtimeDataString, RemoteAddr string) {
handlerLog := new(log)
handlerLog.RuntimeParam = runtimeDataString
handlerLog.AsyncParam = "asyncDataString"
handlerLog.RemoteAddress = RemoteAddr
jsonLog, err := json.Marshal(handlerLog)
if err == nil {
producer.ProduceChannel() <- &kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topicName, Partition: kafka.PartitionAny},
Value: jsonLog,
}
}
}("runtimeDataString", r.RemoteAddr)
}
The goroutines unlikely improve performance of your server as you just send the response earlier and those kafka connections could pile up in the background and slow down the whole server. If you find this as the bottleneck, you may consider saving logs locally and sending them to kafka in another process (or pool of workers) outside of your server. This may spread the workload over time (like sending fewer logs when you have more requests and vice versa).
I'm new to Golang and trying to get a basic http app running using the httprouter API. I've hit a wall with reading posted form data, despite following the advice given in another StackOverflow question.
Here's my code (minus irrelevancies):
import (
"fmt"
"net/http"
"github.com/julienschmidt/httprouter"
)
func main() {
r := httprouter.New()
r.POST("/sub", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
r.Header.Set("content-type", "text/html")
err := r.ParseForm()
if err != nil {
fmt.Fprintf(w, "<h1>Error: %s</h1>\n", err)
}
fmt.Fprintf(w, "<h1>Submitted message!</h1>\n<p>-%s-</p>\n", r.PostFormValue("msg"))
})
http.ListenAndServe("localhost:3000", r)
}
In the output, where I should see -hello-, I just see --. When I inspect the http request in Firefox, in the Form Data panel, I see msg:"hello", so why is r.PostFormValue("msg") returning a blank string?
Thanks to Volker for pointing out an error. When I commented out the line r.Header.Set("content-type", "text/html"), the problem was resolved. Perhaps that was the issue, or perhaps there was some issue with the IDE (LiteIDE) caching an old version of the code. In any case, I can now read the posted value.