I'm writing a simple webserver in golang that gets/creates/edits/deletes a simple text file. I've written the function handlers and I'd like to test them by sending a request to the appropriate url and checking to see what happens. My code is as below:
func createHandler(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
name := vars["name"]
body, _ := ioutil.ReadAll(r.Body)
fmt.Fprint(w, name)
ioutil.WriteFile(name, []byte(body), 0644)
}
func main() {
r := mux.NewRouter()
r.HandleFunc("/textFiles/{name}", createHandler).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", r))
var url = "http://localhost:8080/textFiles/testFile.txt"
var text = []byte(`{"title":"this is an example."}`)
req, err := http.NewRequest("POST", url, bytes.NewBuffer(text))
if err != nil {
panic(err)
}
client := &http.Client{}
client.Do(req)
}
Once this code is run, however, no new file is created. I've been googling but I can't find anything on this type of problem, where I'm sending a request to the server that I'm building within the same file. Help appreciated.
The client code is not executed. The callhttp.ListenAndServe(":8080", r) runs the server. The function only returns when there was an error running the server. If the function does return, then log.Fatal will exit the process.
One fix is to run the server in a goroutine. This will allow main goroutine to continue executing to the client code.
go func() {
log.Fatal(http.ListenAndServe(":8080", r))
}()
This may not fix the problem because there's no guarantee that server will run before the client makes the request. Fix this issue by creating the listening socket in the main function and running the server in a goroutine.
ln, err := net.Listen("tcp", ":8080")
if err != nil {
log.Fatal(err)
}
go func() {
log.Fatal(http.Serve(ln, r))
}()
... client code as before
If the goal if this code is testing, then use httptest.Server. The example in the documentation show show to use the test server.
Related
I am trying to convert below nodejs code to Go. I have to establish keep alive http request to PouchDB server's _changes?feed=continuous. However, I'm not able to achieve it in Go.
var http = require('http')
var agent = new http.Agent({
keepAlive: true
});
var options = {
host: 'localhost',
port: '3030',
method: 'GET',
path: '/downloads/_changes?feed=continuous&include_docs=true',
agent
};
var req = http.request(options, function(response) {
response.on('data', function(data) {
let val = data.toString()
if(val == '\n')
console.log('newline')
else {
console.log(JSON.parse(val))
//to close the connection
//agent.destroy()
}
});
response.on('end', function() {
// Data received completely.
console.log('end');
});
response.on('error', function(err) {
console.log(err)
})
});
req.end();
Below is the Go code
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
defer resp.Body.Close()
result, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
}
fmt.Println(result)
I am getting status 200 Ok, but no data gets printed, its stuck. On the other hand if I use longpoll option ie. http://localhost:3030/downloads/_changes?feed=longpoll then I am receiving data.
Your code is working "as expected" and what you wrote in Go is not equivalent to code shown in Node.js. Go code blocks on ioutil.ReadAll(resp.Body) because connection is kept open by CouchDB server. Once server closes the connection your client code will print out result as ioutil.ReadAll() will be able to read all data down to EOF.
From CouchDB documentation about continuous feed:
A continuous feed stays open and connected to the database until explicitly closed and changes are sent to the client as they happen, i.e. in near real-time. As with the longpoll feed type you can set both the timeout and heartbeat intervals to ensure that the connection is kept open for new changes and updates.
You can try experiment and add &timeout=1 to URL which will force CouchDB to close connection after 1s. Your Go code then should print the whole response.
Node.js code works differently, event data handler is called every time server sends some data. If you want to achieve same and process partial updates as they come (before connection is closed) you cannot use ioutil.ReadAll() as that waits for EOF (and thus blocks in your case) but something like resp.Body.Read() to process partial buffers. Here is very simplified snippet of code that demonstrates that and should give you basic idea:
package main
import (
"fmt"
"net/http"
"net/url"
"strings"
)
func main() {
client := &http.Client{}
data := url.Values{}
req, err := http.NewRequest("GET", "http://localhost:3030/downloads/_changes?feed=continuous&include_docs=true", strings.NewReader(data.Encode()))
req.Header.Set("Connection", "keep-alive")
resp, err := client.Do(req)
defer resp.Body.Close()
fmt.Println(resp.Status)
if err != nil {
fmt.Println(err)
}
buf := make([]byte, 1024)
for {
l, err := resp.Body.Read(buf)
if l == 0 && err != nil {
break // this is super simplified
}
// here you can send off data to e.g. channel or start
// handler goroutine...
fmt.Printf("%s", buf[:l])
}
fmt.Println()
}
In real world application you probably want to make sure your buf holds something that looks like a valid message and then pass it to channel or handler goroutine for further processing.
Finally, I was able to resolve the issue. The issue was related to DisableCompression flag. https://github.com/golang/go/issues/16488 this issue gave me some hint.
By setting DisableCompression: true fixed the issue.
client := &http.Client{Transport: &http.Transport{
DisableCompression: true,
}}
I am assuming client := &http.Client{} sends DisableCompression : false by default and pouchdb server is sending compressed json, Hence received data was compressed and resp.Body.Read was not able to read.
In Go, how can I start the browser AFTER the server started listening?
Preferably the simplest way possible.
My code so far, super dumbed down to the point:
package main
import (
// Standard library packages
"fmt"
"net/http"
"github.com/skratchdot/open-golang/open"
// Third party packages
"github.com/julienschmidt/httprouter"
)
// go get github.com/toqueteos/webbrowser
func main() {
// Instantiate a new router
r := httprouter.New()
// Add a handler on /test
r.GET("/test", func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
// Simply write some test data for now
fmt.Fprint(w, "Welcome!\n")
})
//open.Run("https://google.com/")
// open.Start("https://google.com")
// http://127.0.0.1:3000/test
// Fire up the server
http.ListenAndServe("localhost:3000", r)
fmt.Println("ListenAndServe is blocking")
open.RunWith("http://localhost:3000/test", "firefox")
fmt.Println("Done")
}
Open the listener, start the browser and then enter the server loop:
l, err := net.Listen("tcp", "localhost:3000")
if err != nil {
log.Fatal(err)
}
// The browser can connect now because the listening socket is open.
err := open.Start("http://localhost:3000/test")
if err != nil {
log.Println(err)
}
// Start the blocking server loop.
log.Fatal(http.Serve(l, r))
There's no need to poll as shown in another answer. The browser will connect if the listening socket is open before the browser is started.
ListenAndServe is a convenience function that opens a socket and calls Serve. The code in this answer splits out these steps so the browser can be opened after listening starts but before the blocking call to Serve.
If there is no error, http.ListenAndServe() will never return. So you shouldn't add code after that except code that handles failure.
You have to start a new goroutine, so ListenAndServe() is called in one goroutine, and code checking if it is up should run on the other goroutine.
And you can check if your server is up by making a simple HTTP GET call to it, for example using http.Get().
The following example delays startup for 7 seconds on purpose. The new goroutine starts an endless for loop that checks if server is up, sleeping 1 second between attempts.
Example:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hi!"))
})
go func() {
for {
time.Sleep(time.Second)
log.Println("Checking if started...")
resp, err := http.Get("http://localhost:8081")
if err != nil {
log.Println("Failed:", err)
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Println("Not OK:", resp.StatusCode)
continue
}
// Reached this point: server is up and running!
break
}
log.Println("SERVER UP AND RUNNING!")
}()
log.Println("Starting server...")
time.Sleep(time.Second * 7)
log.Fatal(http.ListenAndServe(":8081", nil))
Example output:
2015/09/23 13:53:03 Starting server...
2015/09/23 13:53:04 Checking if started...
2015/09/23 13:53:06 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:07 Checking if started...
2015/09/23 13:53:09 Failed: Get http://localhost:8081: dial tcp [::1]:8081: connectex: No connection could be made because the target machine actively refused it.
2015/09/23 13:53:10 Checking if started...
2015/09/23 13:53:10 SERVER UP AND RUNNING!
The API is not absolutely terrible, but let's just say "It takes some getting used to". Here is how you use custom attributes on the Server struct:
s := &http.Server{
Addr: cnf.API_SERVER_ADDRESS,
Handler: h,
ReadTimeout: 0, // 1 * time.Minute,
WriteTimeout: 30 * time.Minute,
MaxHeaderBytes: 1 << 20,
}
go func() {
l, err := net.Listen("tcp", cnf.API_SERVER_ADDRESS)
if err != nil {
log.Fatal(err)
}
fmt.Println(`{"server_state":"listening"}`)
log.Fatal(s.Serve(l));
}()
because if you instead use:
http.Serve(l, handler)
then you can't define custom properties on the server
I am playing with the net package, and i want to make a simple proxy.
First i make a listener on localhost, then i dial the remote address
remote, err := net.Dial("tcp", "google.com:80")
if err != nil {
log.Fatal(err)
}
defer remote.Close()
fmt.Fprint(remote, "GET / HTTP/1.0\r\n\r\n")
How can i pipe the response to the browser? Or do i need to work with the default webserver and copy the response body? I really want to try it with net package or something
thx
To copy the connection from the remote is used 2 goroutines with io.Copy
func copyContent(from, to net.Conn, done chan bool) {
_, err := io.Copy(from, to)
if err != nil {
done <- true
}
done <- true
}
// in the main func
done := make(chan bool, 2)
go copyContent(conn, remote, done)
go copyContent(remote, conn, done)
<-done
<-done
I'm writing a simple proxy in go that relays an HTTP request to one or more backend servers. Right now I'm still using only one backend server, but performance is not good and I'm sure I am doing something wrong. Probably related to how I send the HTTP request to another server: if I comment the call to send() then the server goes blazing fast, yielding more than 14 krps. While with the call to send() performance drops to less than 1 krps and drops even lower with time. This on a MacBook Pro.
The code is based on trivial code samples; I have created the client and reused it following the recommendation in the docs. Tests are done with Apache ab:
$ ab -n 10000 -c 10 -k "http://127.0.0.1:8080/test"
The backend server running on port 55455 does not change between experiments; Apache or nginx can be used. My custom web server yields more than 7 krps when measured directly, without proxy:
$ ab -n 10000 -c 10 -k "http://127.0.0.1:55455/test"
I would expect the proxied version to behave just as well as the non-proxied version, and sustain the performance over time.
The complete sample code follows.
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
tr := &http.Transport{
DisableCompression: true,
DisableKeepAlives: false,
}
client := &http.Client{Transport: tr}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
send(r, client)
fmt.Fprintf(w, "OK")
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
func send(r *http.Request, client *http.Client) int {
req, err := http.NewRequest("GET", "http://localhost:55455" + r.URL.Path, nil)
if err != nil {
log.Fatal(err)
return 0
}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
return 0
}
if resp == nil {
return 0
}
return 1
}
Eventually the code should send the request to multiple servers and process their answers, returning an int with the result. But I'm stuck at this step of just making the call.
What am I doing horribly wrong?
As the comment suggests, you should be returning (and dealing with) type error instead of ints, and to reiterate, don't use AB. The biggest thing that stands out to me is
You should set the MaxIdleConnsPerHost in your Transport. This represents how many connections will persist (keep-alive) even if they have nothing to do at the moment.
You have to read and close the body of the response in order for it to be returned to the pool.
...
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
defer resp.Body.Close()
...
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/