Why can't I run both HTTP and HTTPS from the same golang program?
Here is the code where the two servers are initiated.. The server which is initiated first will run - the second won't.. If they are switched arround the other will run and the other won't..
No errors are returned when running the program, but the requests http://www.localhost or https://secure.localhost times out
// Start HTTP
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
Here is the complete code
package main
import (
"net/http"
"fmt"
"log"
"os"
"io"
"runtime"
// go get github.com/gorilla/mux
"github.com/gorilla/mux"
)
const (
HOST = "localhost"
)
func Handler_404(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "Oops, something went wrong!")
}
func Handler_www(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "Hello world :)")
}
func Handler_api(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "This is the API")
}
func Handler_secure(w http.ResponseWriter, r *http.Request){
fmt.Fprint(w, "This is Secure")
}
func redirect(r *mux.Router, from string, to string){
r.Host(from).Subrouter().HandleFunc("/", func (w http.ResponseWriter, r *http.Request){
http.Redirect(w, r, to, 301)
})
}
func main(){
port := 9000
ssl_port := 443
runtime.GOMAXPROCS(runtime.NumCPU())
http_r := mux.NewRouter()
https_r := mux.NewRouter()
// HTTP 404
http_r.NotFoundHandler = http.HandlerFunc(Handler_404)
// Redirect "http://HOST" => "http://www.HOST"
redirect(http_r, HOST, fmt.Sprintf("http://www.%s:%d", HOST, port))
// Redirect "http://secure.HOST" => "https://secure.HOST"
redirect(http_r, "secure."+HOST, fmt.Sprintf("https://secure.%s", HOST))
www := http_r.Host("www."+HOST).Subrouter()
www.HandleFunc("/", Handler_www)
api := http_r.Host("api."+HOST).Subrouter()
api.HandleFunc("/", Handler_api)
secure := https_r.Host("secure."+HOST).Subrouter()
secure.HandleFunc("/", Handler_secure)
// Start HTTP
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
}
ListenAndServe and ListenAndServeTLS open the listening socket and then loop forever serving client connections. These functions only return on an error.
The main goroutine never gets to the starting the TLS server because the main goroutine is busy waiting for HTTP connections in ListenAndServe.
To fix the problem, start the HTTP server in a new goroutine:
// Start HTTP
go func() {
err_http := http.ListenAndServe(fmt.Sprintf(":%d", port), http_r)
if err_http != nil {
log.Fatal("Web server (HTTP): ", err_http)
}
}()
// Start HTTPS
err_https := http.ListenAndServeTLS(fmt.Sprintf(":%d", ssl_port), "D:/Go/src/www/ssl/public.crt", "D:/Go/src/www/ssl/private.key", https_r)
if err_https != nil {
log.Fatal("Web server (HTTPS): ", err_https)
}
As previously said, both ListenAndServe and ListenAndServeTLS are blocking. That being said, I would agree that examples above are in fact resolving your issue as the point is to be in goroutine BUT same examples are not quite following go idioms.
You should be using error channels here as you want to capture ALL errors that are sent to you instead of having just one error returned back. Here's fully working sample that starts HTTP as HTTPS servers and return errors as channel that's later on used just to display errors.
package main
import (
"log"
"net/http"
)
func Run(addr string, sslAddr string, ssl map[string]string) chan error {
errs := make(chan error)
// Starting HTTP server
go func() {
log.Printf("Staring HTTP service on %s ...", addr)
if err := http.ListenAndServe(addr, nil); err != nil {
errs <- err
}
}()
// Starting HTTPS server
go func() {
log.Printf("Staring HTTPS service on %s ...", addr)
if err := http.ListenAndServeTLS(sslAddr, ssl["cert"], ssl["key"], nil); err != nil {
errs <- err
}
}()
return errs
}
func sampleHandler(w http.ResponseWriter, req *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.Write([]byte("This is an example server.\n"))
}
func main() {
http.HandleFunc("/", sampleHandler)
errs := Run(":8080", ":10443", map[string]string{
"cert": "/path/to/cert.pem",
"key": "/path/to/key.pem",
})
// This will run forever until channel receives error
select {
case err := <-errs:
log.Printf("Could not start serving service due to (error: %s)", err)
}
}
Hope this helps! :)
func serveHTTP(mux *http.ServeMux, errs chan<- error) {
errs <- http.ListenAndServe(":80", mux)
}
func serveHTTPS(mux *http.ServeMux, errs chan<- error) {
errs <- http.ListenAndServeTLS(":443", "fullchain.pem", "privkey.pem", mux)
}
func main() {
mux := http.NewServeMux()
// setup routes for mux // define your endpoints
errs := make(chan error, 1) // a channel for errors
go serveHTTP(mux, errs) // start the http server in a thread
go serveHTTPS(mux, errs) // start the https server in a thread
log.Fatal(<-errs) // block until one of the servers writes an error
}
The ListenAndServe (and ListenAndServeTLS) functions do not return to their caller (unless an error is encountered). You can test this by trying to print something in between the two calls.
Related
Problem Statement
I have a client (which dials to the server) and server (that listens for incoming requests) written in golang and with the RPC calls defined. I am trying to initiate an HTTP request on the server side which would in turn execute the RPC call for streaming and send a JSON response back to the user
Challenge
I was able to handle both grpc and HTTP requests on different ports but having issues with passing parameters from the HTTP request onto the RPC call on the server side
Server Code
log.Println("Listening for connections from client ........")
lis, err := net.Listen("tcp", ":9000")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := testApi.Server{}
grpcServer := grpc.NewServer()
testApi.RegisterTestApiServiceServer(grpcServer, &s)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %s", err)
}
func main() {
go runGrpc()
log.Printf("*------ Waiting for requests from users ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":8080", router))
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
test_id, _ := strconv.Atoi(vars["test_id"])
log.Println("Test id request from user : ", test_id)
func (s * Server) ConnectAndStream(channelStream TestApiService_ConnectAndStreamServer) error {
// Question: This Id has to come from http request above- test_id
var id int32 = 1234566
// id := a.ConnectAndExchange
log.Println("Id from sam user ", id)
// var id int32 = 1234566
for i := 1; i <= 2; i++ {
id += 1
log.Println("Speed Server is sending data : ", id)
channelStream.Send(&Input{Id: id})
}
for i := 1; i <= 2; i++ {
log.Println("now time to receive")
client_response, err := channelStream.Recv()
log.Println("Response from samd client : ", client_response.Id)
if err != nil {
log.Println("Error while receiving from samd : ", err)
}
}
return nil
}
I am stuck with being able to pass the test_id from the curl request to the RPC call as above. Any input is greatly appreciated
Note
Client - Dials in and connects to the server and starts receiving and sending data (bi-directional streaming)
Both the Http and GRPC client are part of the same server application. So why call the RPC method from the Http handler? The Http handler should have access to the same backend functionality.
Your question is slightly unclear but if you are trying to have your client establish a GRPC connection to the server via the HTTP handler this will not work. The GRPC connection established in this situation is between the server and its self.
Edit - thanks for the clarification. Now I understand better the flow that you are trying to achieve. Your http handler method can make the outgoing grpc call to the server and return the response back via the http.ResponseWriter
For simplicity I have used the hello world example on https://github.com/grpc/grpc-go/tree/master/examples/helloworld
Running the code sample below and hitting http://localhost:1000/exchangeId/Test will show the output
Starting
*------ Waiting for http requests from users on port 1000 ------*
server listening at 127.0.0.1:1001
Test id request from user : Test
Server Received: Test
Greeting: Hello Test
Code sample:
import (
"context"
"log"
"net"
"net/http"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
pb "google.golang.org/grpc/examples/helloworld/helloworld"
"github.com/gorilla/mux"
)
var (
grpcserver = "localhost:1001"
)
func main() {
log.Print("Starting")
go StartGrpcServer()
log.Printf("*------ Waiting for http requests from users on port 1000 ------*")
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/exchangeId/{test_id}", ConnectAndExchange).Methods("GET")
log.Fatal(http.ListenAndServe(":1000", router))
}
type server struct {
pb.UnimplementedGreeterServer
}
// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Server Received: %v", in.GetName())
return &pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}
func StartGrpcServer() {
lis, err := net.Listen("tcp", grpcserver)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
func ConnectAndExchange(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
test_id := vars["test_id"]
log.Println("Test id request from user : ", test_id)
// Set up a connection to the server.
conn, err := grpc.Dial(grpcserver, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)
// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
resp, err := c.SayHello(ctx, &pb.HelloRequest{Name: test_id})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", resp.GetMessage())
w.Write([]byte(resp.GetMessage()))
}
I might be just clueless on this but my basic localhost server doesn't have HTTP2 Enabled for some odd reason, I normally proxy behind Caddy, but as I don't want to use my domain for this side project, I created a basic server in Go, and ran it, it works okay, but the headers show HTTP/1.1 instead of 2.0, what's wrong?
package main
import (
"fmt"
"net/http"
"html/template"
"os"
)
func IfError(err error, Quit bool) {
if err != nil {
fmt.Println(err.Error())
if(Quit) {
os.Exit(1);
}
}
}
func ServeHome(w http.ResponseWriter, r *http.Request) {
t, err := template.ParseFiles("html/home")
IfError(err, false)
err = t.Execute(w, nil)
IfError(err, false)
}
func RedirectRoot(fs http.Handler, home http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/" {
home.ServeHTTP(w, r)
} else {
fs.ServeHTTP(w, r)
}
})
}
func main() {
proto := ":8081"
ServeFiles := http.FileServer(http.Dir("static/"))
http.Handle("/", RedirectRoot(ServeFiles, http.HandlerFunc(ServeHome)))
fmt.Printf("Listening on ... %s", proto)
IfError(http.ListenAndServe(proto, nil), true)
}
Very basic stuff, but doesn't work even thought the documentation says it works by default. Also, my go version is 1.8.3
Yes, its enabled by default when you use with SSL certs.
Doc Reference: Starting with Go 1.6, the http package has transparent
support for the HTTP/2 protocol when using HTTPS.
err := http.ListenAndServeTLS(":8081", "server.crt", "server.key", handler)
if err != nil && err != http.ErrServerClosed {
log.Fatal("ListenAndServe: ", err)
}
Then, access it via
https://localhost:8081/
Is there any way to print something when the http server starts? For instance "Server is started at port 8080"
In Node (using Express), it would be like:
app.listen(8080, function() { console.log('Server started at port 8080') });
This is my code:
func main() {
http.HandleFunc("/", MyHandler)
http.ListenAndServe(":8080", nil)
}
Thanks.
Use Go's log package:
package main
import (
"net/http"
"log"
)
func main() {
addr := ":8080"
http.HandleFunc("/", MyHandler)
log.Println("listen on", addr)
log.Fatal( http.ListenAndServe(addr, nil) )
}
http.ListenAndServe opens the server port, and blocks forever waiting for clients. If it fails to open the port, the log.Fatal call will report the problem and exit the program.
You can't print a log message after ListenAndServe since it blocks and never returns, so basically you have two main options:
Print "Starting server on port...." and that's it - BUT if ListenAndServe could not start it returns an error, so unless there's some error or panic printed because of that, you can assume the server started.
Call ListenAndServe in a separate goroutine, and make sure there was no error returned and print "Server started..." etc.
I personally prefer the first approach.
To run ListenAndServe in a goroutine as mentioned by Not_a_Golfer, you can use an unbuffered, blocking channel to run it in a goroutine and also keep the server alive.
The following example creates a channel called done where <-done will keep the server alive as it waits for the goroutine to finish, which it won't in this case. Typically, the goroutine will tell the main function it is finished by executing done <- true.
package main
import (
"log"
"net/http"
)
func MyHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello World"))
}
func main() {
port := "8080"
http.HandleFunc("/", MyHandler)
done := make(chan bool)
go http.ListenAndServe(":"+port, nil)
log.Printf("Server started at port %v", port)
<-done
}
Here's a larger example that has the server verify it is operational, using Listen and Serve separately. The nice thing about doing it this way is you can capture an incorrect port easily.
package main
import (
"log"
"net"
"net/http"
"os"
)
func MyHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello World"))
}
func main() {
port := "8080"
http.HandleFunc("/", MyHandler)
listener, err := net.Listen("tcp", ":"+port)
if err != nil {
log.Fatal(err)
}
done := make(chan bool)
go http.Serve(listener, nil)
// Log server started
log.Printf("Server started at port %v", port)
// Attempt to connect
log.Printf("Fetching...")
res, err := http.Get("http://" + listener.Addr().String())
log.Printf("Received: %v, %v", res, err)
if err != nil {
log.Fatal(err)
}
res.Write(os.Stdout)
<-done
}
I'm currently writing some software in Go that interacts with a REST API. The REST API endpoint I'm trying to query returns an HTTP 302 redirect along with an HTTP Location header, pointing to a resource URI.
I'm trying to use my Go script to grab the HTTP Location header for later processing.
Here's what I'm currently doing to achieve this functionality:
package main
import (
"errors"
"fmt"
"io/ioutil"
"net/http"
)
var BASE_URL = "https://api.example.com/v1"
var STORMPATH_API_KEY_ID = "xxx"
var STORMPATH_API_KEY_SECRET = "xxx"
func noRedirect(req *http.Request, via []*http.Request) error {
return errors.New("Don't redirect!")
}
func main() {
client := &http.Client{
CheckRedirect: noRedirect
}
req, err := http.NewRequest("GET", BASE_URL+"/tenants/current", nil)
req.SetBasicAuth(EXAMPLE_API_KEY_ID, EXAMPLE_API_KEY_SECRET)
resp, err := client.Do(req)
// If we get here, it means one of two things: either this http request
// actually failed, or we got an http redirect response, and should process it.
if err != nil {
if resp.StatusCode == 302 {
fmt.Println("got redirect")
} else {
panic("HTTP request failed.")
}
}
defer resp.Body.Close()
}
This feels like a bit of a hack to me. By overriding the http.Client's CheckRedirect function, I'm essentially forced to treat HTTP redirects like errors (which they aren't).
I've seen several other places suggesting to use an HTTP transport instead of an HTTP client -- but I'm not sure how to make this work since I need the HTTP Client as I need to use HTTP Basic Auth to communicate with this REST API.
Can any of you tell me a way to make HTTP requests with Basic Authentication -- while not following redirects -- that doesn't involve throwing errors and error handling?
There's a much simpler solution right now:
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
This way, the http package automatically knows: "Ah, I shouldn't follow any redirects", but does not throw any error. From the comment in the source code:
As a special case, if CheckRedirect returns ErrUseLastResponse,
then the most recent response is returned with its body
unclosed, along with a nil error.
Another option, using the client itself, without the RoundTrip:
// create a custom error to know if a redirect happened
var RedirectAttemptedError = errors.New("redirect")
client := &http.Client{}
// return the error, so client won't attempt redirects
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
return RedirectAttemptedError
}
// Work with the client...
resp, err := client.Head(urlToAccess)
// test if we got the custom error
if urlError, ok := err.(*url.Error); ok && urlError.Err == RedirectAttemptedError{
err = nil
}
UPDATE: this solution is for go < 1.7
It is possible, but the solution inverts the problem a little. Here's a sample written up as a golang test.
package redirects
import (
"github.com/codegangsta/martini-contrib/auth"
"github.com/go-martini/martini"
"net/http"
"net/http/httptest"
"testing"
)
func TestBasicAuthRedirect(t *testing.T) {
// Start a test server
server := setupBasicAuthServer()
defer server.Close()
// Set up the HTTP request
req, err := http.NewRequest("GET", server.URL+"/redirect", nil)
req.SetBasicAuth("username", "password")
if err != nil {
t.Fatal(err)
}
transport := http.Transport{}
resp, err := transport.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
// Check if you received the status codes you expect. There may
// status codes other than 200 which are acceptable.
if resp.StatusCode != 200 && resp.StatusCode != 302 {
t.Fatal("Failed with status", resp.Status)
}
t.Log(resp.Header.Get("Location"))
}
// Create an HTTP server that protects a URL using Basic Auth
func setupBasicAuthServer() *httptest.Server {
m := martini.Classic()
m.Use(auth.Basic("username", "password"))
m.Get("/ping", func() string { return "pong" })
m.Get("/redirect", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ping", 302)
})
server := httptest.NewServer(m)
return server
}
You should be able to put the above code into it's own package called "redirects" and run it after fetching the required dependencies using
mkdir redirects
cd redirects
# Add the above code to a file with an _test.go suffix
go get github.com/codegangsta/martini-contrib/auth
go get github.com/go-martini/martini
go test -v
Hope this helps!
To make request with Basic Auth that does not follow redirect use RoundTrip function that accepts *Request
This code
package main
import (
"fmt"
"io/ioutil"
"net/http"
"os"
)
func main() {
var DefaultTransport http.RoundTripper = &http.Transport{}
req, _ := http.NewRequest("GET", "http://httpbin.org/headers", nil)
req.SetBasicAuth("user", "password")
resp, _ := DefaultTransport.RoundTrip(req)
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("%s", err)
os.Exit(1)
}
fmt.Printf("%s\n", string(contents))
}
outputs
{
"headers": {
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNzd29yZA==",
"Connection": "close",
"Host": "httpbin.org",
"User-Agent": "Go 1.1 package http",
"X-Request-Id": "45b512f1-22e9-4e49-8acb-2f017e0a4e35"
}
}
As an addition of top rated answer,
You can control the particle size
func myCheckRedirect(req *http.Request, via []*http.Request, times int) error {
err := fmt.Errorf("redirect policy: stopped after %d times", times)
if len(via) >= times {
return err
}
return nil
}
...
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return myCheckRedirect(req, via, 1)
},
}
ref: https://golangbyexample.com/http-no-redirect-client-golang/
i use
resp, err := http.Get("http://example.com/")
get a http.Response, and i want to exactly write to a http handler, but only http.ResponseWriter, so i hijack it.
...
webConn, webBuf, err := hj.Hijack()
if err != nil {
// handle error
}
defer webConn.Close()
// Write resp
resp.Write(webBuf)
...
Write raw request
But When i hijack, http connection can't reuse (keep-alive), so it slow.
How to solve?
Thanks! Sorry for my pool English.
update 12/9
keep-alive, It keep two tcp connection, and can reuse.
but when i hijack, and conn.Close(), It can't reuse old connection, so it create a new tcp connection when i each refresh.
Do not use hijack, Because once hijack, the HTTP server library will not do anything else with the connection, So can't reuse.
I change way, copy Header and Body, look like reverse proxy (http://golang.org/src/pkg/net/http/httputil/reverseproxy.go), Is works.
Example:
func copyHeader(dst, src http.Header) {
for k, w := range src {
for _, v := range w {
dst.Add(k, v)
}
}
}
func copyResponse(r *http.Response, w http.ResponseWriter) {
copyHeader(w.Header(), r.Header)
w.WriteHeader(r.StatusCode)
io.Copy(w, r.Body)
}
func handler(w http.ResponseWriter, r *http.Response) {
resp, err := http.Get("http://www.example.com")
if err != nil {
// handle error
}
copyResponse(resp, w)
}
It seem that once the connection is closed the keep-alive connection closes as well.
One possible solution would be to prevent the connection from closing until desired, but I'm not sure if that good advise.
Maybe the correct solution involves creating a instance of net.TCPConn, copying the connection over it, then calling .SetKeepAlive(true).
Before running the below example, launch another terminal with netstat -antc | grep 9090.
Routes in example:
localhost:9090/ok is a basic (non-hijacked) connection
localhost:9090 is a hijacked connection, lasting for 10 seconds.
Example
package main
import (
"fmt"
"net/http"
"sync"
"time"
)
func checkError(e error) {
if e != nil {
panic(e)
}
}
var ka_seconds = 10
var conn_id = 0
func main() {
http.HandleFunc("/ok", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "ok")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
conn_id++
fmt.Printf("Connection %v: Keep-alive is enabled %v seconds\n", conn_id, ka_seconds)
hj, ok := w.(http.Hijacker)
if !ok {
http.Error(w, "webserver doesn't support hijacking", http.StatusInternalServerError)
return
}
conn, bufrw, err := hj.Hijack()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Don't forget to close the connection:
time.AfterFunc(time.Second* time.Duration(ka_seconds), func() {
conn.Close()
fmt.Printf("Connection %v: Keep-alive is disabled.\n", conn_id)
})
resp, err := http.Get("http://www.example.com")
checkError(err)
resp.Write(bufrw)
bufrw.Flush()
})
fmt.Println("Listing to localhost:9090")
http.ListenAndServe(":9090", nil)
}
Related issue: http://code.google.com/p/go/issues/detail?id=5645