golang: core net/http package import errors - http

Question

I've cloned go source code using git clone https://go.googlesource.com/go into my ~/godev/ directory (outside of GOPATH as the docs advise).
My $GOPATH is ~/gocode
I installed go 1.8.1 using the official osx installer.
If I cd into ~/godev/go/src/net/http and run go test, I get these errors:
h2_bundle.go:46:2: cannot find package "golang_org/x/net/http2/hpack" in any of:
/usr/local/go/src/golang_org/x/net/http2/hpack (from $GOROOT)
~/gocode/src/golang_org/x/net/http2/hpack (from $GOPATH)
h2_bundle.go:47:2: cannot find package "golang_org/x/net/idna" in any of:
/usr/local/go/src/golang_org/x/net/idna (from $GOROOT)
~/gocode/src/golang_org/x/net/idna (from $GOPATH)
h2_bundle.go:48:2: cannot find package "golang_org/x/net/lex/httplex" in any of:
/usr/local/go/src/golang_org/x/net/lex/httplex (from $GOROOT)
~/gocode/src/golang_org/x/net/lex/httplex (from $GOPATH)
transport.go:32:2: cannot find package "golang_org/x/net/proxy" in any of:
/usr/local/go/src/golang_org/x/net/proxy (from $GOROOT)
~/gocode/src/golang_org/x/net/proxy (from $GOPATH)
transfer.go:14:2: use of internal package not allowed
After I follow the directions here by doing cd $GOPATH/src
followed by cp -R /usr/local/go/src/vendor/golang_org ., I still get these errors:
h2_bundle.go:47:2: code in directory ~/gocode/src/golang_org/x/net/idna expects import "golang.org/x/net/idna"
transport.go:32:2: cannot find package "golang_org/x/net/proxy" in any of:
/usr/local/go/src/golang_org/x/net/proxy (from $GOROOT)
~/gocode/src/golang_org/x/net/proxy (from $GOPATH)
transfer.go:14:2: use of internal package not allowed
It is true that there is no proxy package under ~/gocode/src/golang_org/x/net, but I still don't know how to fix that and the other 2 errors.
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="~/gocode"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/71/k_tftg2d1qd7gf5ww0n_wl_r0000gn/T/go-build541211050=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
If I run the all.bash script, it will run all unit tests, but that is time consuming. Is there a way to run just net/http tests without getting these errors?

Answer 1

The problem here is that you're running go test, where the go command is your 1.8.1 installation. For the tests to work properly you should run with the Go toolchain built from your development directory.
Make sure you've built the Go toolchain, cd ~/godev/src; ./make.bash (./all.bash will work too, but then you'll have to wait for tests to run instead of just building the toolchain).
Run the tests with the newly compiled toolchain, cd ~/godev/src/net/http; ~/godev/bin/go test.
I suggest adding an alias to your profile, such as alias godev=~/godev/bin/go, then you can run godev test.
Also make sure that you are not setting the GOROOT environment variable as it will cause the go command to use the specified path as the GOROOT regardless of which toolchain you're running with, which is not what you want.
Update
As requested in the comments, here's as brief an explaination as I can come up with:
The errors that mention "cannot find package" are looking for some packages that are vendored in ~/godev/src/vendor/golang_net/.... However, the vendoring support added in 1.5/1.6 only works when the package is inside the GOPATH or GOROOT. Your godev installation is not (and should not be) inside GOPATH and GOROOT is pointing to your 1.8.1 install.
transfer.go:14:2: use of internal package not allowed is because transfer.go imports net/http/internal. Since this is not a relative path it'll be found in $GOROOT/src/net/http/internal, instead of ~/godev/src/net/http/internal and internal packages cannot be imported if the importing package does not share a common root with the internal directory.
It boils down to GOROOT pointing to your 1.8.1 installation. You might wonder if you could just set GOROOT to point at your godev directory, but this is not going to work correctly either. I'm not as certain of the mechanics here, but I think the problems come down to mismatches between what the 1.8.1 compiler expects are what is in ~/godev/src/runtime.
When the toolchain is compiled the location of GOROOT is compiled in, so when ~/godev/bin/go is run, it uses ~/godev as it's GOROOT.