I was trying to make a simple webapp in Go/Golang. My main fuction code is :
func main() {
fileServer := http.FileServer(http.Dir("./static"))
http.Handle("/", fileServer)
http.HandleFunc("/form", formHandler)
http.HandleFunc("/hello", helloHandler)
fmt.Printf("Starting server at port 8080\n")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatal(err)
}
}
When I run the go run main.go I get the following error:
listen tcp: address ;8080: missing port in address
exit status 1
I tried using the lsof -i :8080 command but got no output. Chaging the port to :80 from :8080 had no effect. nc -l :8080 didn't work too.
How do I resolve this issue?
PS: I'm using Fedora OS
Try this code instead:
func main() {
fileServer := http.FileServer(http.Dir("./static"))
httpf := http.NewServeMux()
httpf.Handle("/", fileServer)
httpf.HandleFunc("/form", settings.Grp_Insert)
httpf.HandleFunc("/hello", settings.Grp_Insert)
srv := &http.Server{
ReadTimeout: 3 * time.Second,
WriteTimeout: 6 * time.Second,
Addr: "localhost:8080",
IdleTimeout: 15 * time.Second,
ReadHeaderTimeout: 3 * time.Second,
Handler: httpf,
}
fmt.Printf("Starting server at port 8080\n")
if err := srv.ListenAndServe(); err != nil {
log.Fatal(err)
}
}
Related
I have a dockerfile which runs tor -
FROM alpine:edge
RUN apk update && apk add tor
EXPOSE 9050
USER tor
CMD ["/usr/bin/tor"]
and ran it using command - docker run --name tor -p 11000:9050 tor
and checked connection using - telnet 127.0.0.1 11000 and it showed connected
Now I want to use tor as proxy while any request from go program. I tried -
package main
import (
"fmt"
"net/http"
"net/url"
"time"
)
func main() {
proxyUrl, err := url.Parse("socks5://127.0.0.1:11000")
if err != nil {
// TODO handle me
panic(err)
}
cl := http.Client{
Transport: &http.Transport{
Proxy: http.ProxyURL(proxyUrl),
},
Timeout: 18000 * time.Millisecond,
}
resp, err := cl.Get("http://google.com")
if err != nil {
// TODO handle me
panic(err)
}
// TODO work with the response
fmt.Println(resp)
}
But running this program threw error -
panic: Get http://google.com: socks connect tcp 127.0.0.1:11000->google.com:80: read tcp 127.0.0.1:59630->127.0.0.1:11000: read: connection reset by peer
goroutine 1 [running]: <stacktrace>
exit status 2
I tried other approaches also, notably mentioned here and here but kept getting same error - read: connection reset by peer
Please help which part is incorrect here.
Thanks.
--------------------another approach that I tried ----------------
As mentioned in one of the links, I tried this code also -
const (
PROXY_ADDR = "127.0.0.1:11000"
URL = "http://facebookcorewwwi.onion"
)
func main() {
// create a socks5 dialer
dialer, err := proxy.SOCKS5("tcp", PROXY_ADDR, nil, proxy.Direct)
if err != nil {
fmt.Fprintln(os.Stderr, "can't connect to the proxy:", err)
os.Exit(1)
}
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
// do anything with ctx
return dialer.Dial(network, address)
}
// setup a http client
httpTransport := &http.Transport{
DialContext: dialContext,
}
httpClient := &http.Client{Transport: httpTransport}
// create a request
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
fmt.Fprintln(os.Stderr, "can't create request:", err)
os.Exit(2)
}
resp, err := httpClient.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot make get request: ", err)
os.Exit(2)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Fprintln(os.Stderr, "cannot read response body: ", err)
os.Exit(2)
}
fmt.Println("received response -> ", body)
}
but received error -
cannot make get request: Get http://facebookcorewwwi.onion: socks connect tcp 127.0.0.1:11000->facebookcorewwwi.onion:80: read tcp 127.0.0.1:59826->127.0.0.1:11000: read: connection reset by peer
exit status 2
Any help is appreciable.
After making sure tor is working properly on port 9050.
Try the following curl command to ensure tor is working properly.
curl --socks5 localhost:9050 --socks5-hostname localhost:9050 -s https://wtfismyip.com/json
Can you try this
package main
import (
"context"
"fmt"
"io/ioutil"
"net"
"net/http"
"golang.org/x/net/proxy"
)
func main() {
proxyUrl := "127.0.0.1:9050"
dialer, err := proxy.SOCKS5("tcp", proxyUrl, nil, proxy.Direct)
dialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
return dialer.Dial(network, address)
}
transport := &http.Transport{DialContext: dialContext,
DisableKeepAlives: true}
cl := &http.Client{Transport: transport}
resp, err := cl.Get("https://wtfismyip.com/json")
if err != nil {
// TODO handle me
panic(err)
}
body, err := ioutil.ReadAll(resp.Body)
// TODO work with the response
if err != nil {
fmt.Println("body read failed")
}
fmt.Println(string(body))
}
As suggested in the answer/comment above, main problem was socks5 connection to dockerized tor container.
Found the solution here, i just had to modify dockerfile like -
FROM alpine:edge
RUN apk update && apk add tor
RUN cp /etc/tor/torrc.sample /etc/tor/torrc && echo "SocksPort 0.0.0.0:9050" > /etc/tor/torrc
EXPOSE 9050
CMD ["/usr/bin/tor"]
Other code pieces are working fine.
I'm attempting to write a reverse proxy in golang that reuses the proxy but I'm getting a nil pointer for a reason that I can't see.
type Configuration struct {
Proxies map[string]Proxy `mapstructure:"proxies"`
Port int `mapstructure:"port"`
}
type Proxy struct {
Upstream string `mapstructure:"upstream"`
SkipVerify bool `mapstructure:"tls-skip-verify"`
HostProxy *httputil.ReverseProxy
}
Those are the structs that I'm loading. I then iterate through the map to create the ReverseProxy object int the proxy struct and start the service.
for host, proxy := range config.Proxies {
log.Printf("Loading proxies for: %s", host)
log.Printf("Upstream: %s", proxy.Upstream)
log.Printf("SkipVerify: %t", proxy.SkipVerify)
remoteURL, err := url.Parse(proxy.Upstream)
if err != nil {
log.Fatalf("Unable to parse target: %v", err)
}
log.Infof("Remote : %v", remoteURL)
proxy.HostProxy = httputil.NewSingleHostReverseProxy(remoteURL)
proxy.HostProxy.Transport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 15 * time.Second,
}).Dial,
TLSHandshakeTimeout: 10 * time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: proxy.SkipVerify},
}
log.Print(proxy.HostProxy.Transport)
}
log.Info("Starting Up on Port", config.Port)
http.Handle("/", &config)
log.Fatal(http.ListenAndServe(":"+strconv.Itoa(config.Port), &config))
When it gets called I handle it like this:
func (config *Configuration) ServeHTTP(w http.ResponseWriter, r *http.Request) {
host := r.Host
r.Host = r.URL.Host
log.Info(config)
if proxy, ok := config.Proxies[host]; ok {
log.Infof("Processing request for %s", host)
log.Printf("Upstream: %s", proxy.Upstream)
log.Printf("SkipVerify: %t", proxy.SkipVerify)
log.Print(proxy.HostProxy.Transport)
proxy.HostProxy.ServeHTTP(w, r)
} else {
w.Write([]byte("403: Host forbidden " + host))
}
}
But it throws exceptions that the proxy.HostProxy object is nil. All the other data persists and logs correctly. Such as the proxy.Upstream and the proxy.SkipVerify.
The http.Request struct includes the remote IP and port of the request's sender:
// RemoteAddr allows HTTP servers and other software to record
// the network address that sent the request, usually for
// logging. This field is not filled in by ReadRequest and
// has no defined format. The HTTP server in this package
// sets RemoteAddr to an "IP:port" address before invoking a
// handler.
// This field is ignored by the HTTP client.
**RemoteAddr string**
The http.Response object has no such field.
I would like to know the IP address that responded to the request I sent, even when I sent it to a DNS address.
I thought that net.LookupHost() might be helpful, but 1) it can return multiple IPs for a single host name, and 2) it ignores the hosts file unless cgo is available, which it is not in my case.
Is it possible to retrieve the remote IP address for an http.Response?
Use the net/http/httptrace package and use the GotConnInfo hook to capture the net.Conn and its corresponding Conn.RemoteAddr().
This will give you the address the Transport actually dialled, as opposed to what was resolved in DNSDoneInfo:
package main
import (
"log"
"net/http"
"net/http/httptrace"
)
func main() {
req, err := http.NewRequest("GET", "https://example.com/", nil)
if err != nil {
log.Fatal(err)
}
trace := &httptrace.ClientTrace{
GotConn: func(connInfo httptrace.GotConnInfo) {
log.Printf("resolved to: %s", connInfo.Conn.RemoteAddr())
},
}
req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
client := &http.Client{}
_, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
}
Outputs:
~ go run ip.go
2017/02/18 19:38:11 resolved to: 104.16.xx.xxx:443
Another solution I came up with was the hook the DialContext function in the http client transport. This is a specific solution that lets you modify the http.Client instead of the request which may be useful.
We first create a function that returns a hooked dial context
func remoteAddressDialHook(remoteAddressPtr *net.Addr) func(ctx context.Context, network string, address string) (net.Conn, error) {
hookedDialContext := func(ctx context.Context, network, address string) (net.Conn, error) {
originalDialer := &net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
conn, err := originalDialer.DialContext(ctx, network, address)
if err != nil {
return nil, err
}
// conn was successfully created
*remoteAddressPtr = conn.RemoteAddr()
return conn, err
}
return hookedDialContext
}
We can then use this function to create a DialContext that writes to an outparameter
var remoteAddr net.Addr
customTransport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: remoteAddressDialHook(&remoteAddr),
ForceAttemptHTTP2: true,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
customHttpClient := http.Client{
Transport: customTransport,
}
// do what you normally would with a http client, it will then set the remoteAddr to be the remote address
fmt.Println(remoteAddr.String())
I have a client machine with multiple NICs, how do I bind an http.Client in Go to a certain NIC or to a certain SRC IP Address?
Say you have some very basic http client code that looks like:
package main
import (
"net/http"
)
func main() {
webclient := &http.Client{}
req, _ := http.NewRequest("GET", "http://www.google.com", nil)
httpResponse, _ := webclient.Do(req)
defer httpResponse.Body.Close()
}
Is there a way to bind to a certain NIC or IP?
Similar to this question, you need to set the http.Client.Transport field. Setting it to an instance of net.Transport allows you to specify which net.Dialer you want to use. net.Dialer then allows you to specify the local address to make connections from.
Example:
localAddr, err := net.ResolveIPAddr("ip", "<my local address>")
if err != nil {
panic(err)
}
// You also need to do this to make it work and not give you a
// "mismatched local address type ip"
// This will make the ResolveIPAddr a TCPAddr without needing to
// say what SRC port number to use.
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
webclient := &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
DualStack: true,
}).DialContext,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
},
}
Here is a fully working example that incorporates the answer from Tim. I also broke out all of the nested pieces to make it easier to read and learn from.
package main
import (
"fmt"
"io/ioutil"
"net"
"net/http"
"time"
)
func main() {
localAddr, err := net.ResolveIPAddr("ip", "10.128.64.219")
if err != nil {
panic(err)
}
localTCPAddr := net.TCPAddr{
IP: localAddr.IP,
}
d := net.Dialer{
LocalAddr: &localTCPAddr,
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}
tr := &http.Transport{
Proxy: http.ProxyFromEnvironment,
Dial: d.Dial,
TLSHandshakeTimeout: 10 * time.Second,
}
webclient := &http.Client{Transport: tr}
// Use NewRequest so we can change the UserAgent string in the header
req, err := http.NewRequest("GET", "http://www.google.com:80", nil)
if err != nil {
panic(err)
}
res, err := webclient.Do(req)
if err != nil {
panic(err)
}
fmt.Println("DEBUG", res)
defer res.Body.Close()
content, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err)
}
fmt.Printf("%s", string(content))
}
I have a Go program that generates a lot of HTTP requests from multiple goroutines. after running for a while, the program spits out an error: connect: cannot assign requested address.
When checking with netstat, I get a high number (28229) of connections in TIME_WAIT.
The high number of TIME_WAIT sockets happens when I the number of goroutines is 3 and is severe enough to cause a crash when it is 5.
I run Ubuntu 14.4 under docker and go version 1.7
This is the Go program.
package main
import (
"io/ioutil"
"log"
"net/http"
"sync"
)
var wg sync.WaitGroup
var url="http://172.17.0.9:3000/";
const num_coroutines=5;
const num_request_per_coroutine=100000
func get_page(){
response, err := http.Get(url)
if err != nil {
log.Fatal(err)
} else {
defer response.Body.Close()
_, err =ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal(err)
}
}
}
func get_pages(){
defer wg.Done()
for i := 0; i < num_request_per_coroutine; i++{
get_page();
}
}
func main() {
for i:=0;i<num_coroutines;i++{
wg.Add(1)
go get_pages()
}
wg.Wait()
}
This is the server program:
package main
import (
"fmt"
"net/http"
"log"
)
var count int;
func sayhelloName(w http.ResponseWriter, r *http.Request) {
count++;
fmt.Fprintf(w,"Hello World, count is %d",count) // send data to client side
}
func main() {
http.HandleFunc("/", sayhelloName) // set router
err := http.ListenAndServe(":3000", nil) // set listen port
if err != nil {
log.Fatal("ListenAndServe: ", err)
}
}
The default http.Transport is opening and closing connections too quickly. Since all connections are to the same host:port combination, you need to increase MaxIdleConnsPerHost to match your value for num_coroutines. Otherwise, the transport will frequently close the extra connections, only to have them reopened immediately.
You can set this globally on the default transport:
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = numCoroutines
Or when creating your own transport
t := &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
MaxIdleConnsPerHost: numCoroutines,
MaxIdleConns: 100,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
}
Similar question: Go http.Get, concurrency, and "Connection reset by peer"