I'm newbie on Golang and have a simple question about building a web server.
Saying that my web server has users so the users can change their names and their passwords. Here is how I design the URLs:
/users/Test GET
/users/Test/rename POST newname=Test2
/users/Test/newpassword POST newpassword=PWD
The first line is to show the information of the user named Test. The second and the third is to rename and to reset password.
So I'm thinking that I need to use some regular expression to match the HTTP requests, things like http.HandleFunc("/users/{\w}+", controller.UsersHandler).
However, it doesn't seem that Golang supports such a thing. So does it mean that I have to change my design? For example, to show the information of the user Test, I have to do /users GET name=Test?
You may want to run pattern matching on r.URL.Path, using the regex package (in your case you may need it on POST) This post shows some pattern matching samples. As #Eugene suggests there are routers/http utility packages also which can help.
Here's something which can give you some ideas, in case you don't want to use other packages:
In main:
http.HandleFunc("/", multiplexer)
...
func multiplexer(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case "GET":
getHandler(w, r)
case "POST":
postHandler(w, r)
}
}
func getHandler(w http.ResponseWriter, r *http.Request) {
//Match r.URL.path here as required using switch/use regex on it
}
func postHandler(w http.ResponseWriter, r *http.Request) {
//Regex as needed on r.URL.Path
//and then get the values POSTed
name := r.FormValue("newname")
}
Unfortunately standard http router not to savvy. There are 2 ways:
Manually check methods, urls and extract usernames.
Use routers from other packages like
https://github.com/gorilla/mux
gorilla mix, echo gin-gonic etc
Related
Basically, I'm developing an HTTP endpoint to get the metrics from prometheus package.
Following the instructions in this link [https://stackoverflow.com/a/65609042/17150602] I created a handler to be able to call promhttp.Handler() like so:
g.GET("/metrics", prometheusHandler())
func prometheusHandler() gin.HandlerFunc {
h := promhttp.Handler()
return func(c *gin.Context) {
h.ServeHTTP(c.Writer, c.Request)
}
}
The thing is, when I call localhost:1080/metrics the output shows like this (btw, I'm using Postman):
Postman request to get metrics with wrong output
But if, for instance, I change the port and use http instead of gin package like so:
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(promAddr, nil)
The output shows OK as you can see here:
Postman reuest to get metrics with correct output
What and why is this happening?
Thanks
You probably forgot to remove a compression middleware like gzip.
With HTTP/1.1 Ranged Requests there is a non-zero chance of corruption. Various transparent proxies usually used by ISPs can mangle the requests and thereafter return junk.
Also, not all servers support ranged requests and content length headers.
Is there a KISS way to concurrently download a file using GO without using these tricks?
Is there a KISS way to concurrently download a file using GO without using these tricks?
No, there isn't. But this is not related to Go, this is a plain fact of how HTTP works. If you do not want to use range headers you cannot download just "the first" and "the second" half of a request.
Not only is there no "KISS way", there is absolutely no way (except by using the appropriate tools like range request).
One possible method (depending on exactly how you interpret "concurrently") would be something like the code below. This will NOT allow you to set time-outs or a method through which you can monitor the progress of the download, but it will allow you to start a download, then read it at leisure later.
type download struct {
err error
r io.ReadCloser
}
// Download a file in a goroutine, return a channel through which we can
// read a download struct. The struct will either contain an error or
// an io.ReadCloser through which we can read the file.
func download(url string) chan download {
rv := make(chan download)
go func() {
resp, err := http.Get(url)
if err != nil {
rv <- download{err: err}
return
}
switch {
resp.StatusCode == 200:
// This can be made better, but is probably good
// enough for an illustration
rv <- download{r: resp.Body}
default:
rv <- download{err: fmt.Errorf("HTTP GET gave status %s", resp.Status)
}
}()
return rv
}
According to the openCPU documentation, there are some default HTTP status codes and return types for a few situations. For example, when R raises an error, openCPU returns code 400 with a response type of text/plain.
While I believe it should be possible to control those things, is it possible to customize any of those things directly from R? For example, what if I wanted to return a JSON for a specific error in my R function, with a status code 503?
You can change their R package behavior by forking opencpu or via a local copy i.e. not sure if package allows this like a functionality but the responses are configured in res.R
For e.g. this method in the link above uses 400 for error.
error <- function(msg, status=400){
setbody(msg);
finish(status);
}
I will update the answer if I can confirm this is available without changing package code.
UPDATE 17-04-2021
You can write your serving html i.e. index.html which uses opencpu.js to call the corresponding R functions from your app, the return type can be requested to be json in the opencpu.js call. And in the R function , you can tryCatch() errors to send appropriate error code as a json argument.
For e.g. in the stock example app you can see the file stock.js which calls the functions from R folder i.e.
//this function gets a list of stocks to populate the tree panel
function loadtree(){
var req = ocpu.rpc("listbyindustry", {}, function(data){
Ext.getCmp("tree-panel").getStore().setProxy({
type : "memory",
data : data,
reader : {
type: "json"
}
});
Ext.getCmp("tree-panel").getStore().load();
}).fail(function(){
alert("Failed to load stocks: " + req.responseText);
});
}
The corresponding R code being called is in listbyindustry.R, inside which you can tryCatch() and send custom json.
I'm trying to use Riak's mapreduce via http. his is what i'm sending:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"value(RiakObject, _KeyData, _Arg) -> Key = riak_object:key(RiakObject), Count = riak_kv_crdt:value(RiakObject, <<\"riak_kv_pncounter\">>), [ {Key, Count} ]."
}
}
]}
Riak fails with "[worker_startup_failed]", which isn't very informative. Could anyone please help me get this to actually execute the function?
WARNING
Allowing arbitrary Erlang functions via map-reduce is a security risk. Any valid Erlang can be executed, including sending your entire data set offsite or formatting the hard drive.
You have been warned.
However, if you implicitly trust any client that may connect to your cluster, you can allow Erlang source to be passed in a map-reduce request by setting {allow_strfun, true} in the riak_kv section of app.config, (or in the advanced.config if you are using riak.conf).
Once you have allowed passing an Erlang function in a map-reduce phase, you need to pass in a function of the form fun(RiakObject,KeyData,Arg) -> [result] end. Note that this must be an anonymous fun, so fun is a keyword, not a name, and it must end with end.
Your function should handle the case where {error,notfound} is passed as the first argument instead of an object. Simply adding a catch-all clause to the function could accomplish that.
Perhaps something like:
{
"inputs":{
"bucket":"test",
"key_filters":[["matches", ".*"]]
},
"query":[
{
"map":{
"language":"erlang",
"source":"fun(RiakObject, _KeyData, _Arg) ->
Key = riak_object:key(RiakObject),
Count = riak_kv_crdt:value(
RiakObject,
<<\"riak_kv_pncounter\">>),
[ {Key, Count} ];
(_,_,_) -> [{error,0}]
end."
}
}
]}
Allowing the source to be passed in the request is very useful while developing and debugging. For production, you really should put the functions in a dedicated pre-compiled module that you copy to the code path of each node so that the phase spec can specify the module and function by name instead of providing arbitrary code.
{"map":{
"language":"erlang",
"module":"yourprecompiledmodule",
"function":"functionname"}}
You need to enable allow_strfun on all nodes in your cluster. To do so in Riak 2, you will need to use the advanced.config file to add this to the riak_kv configuration:
[
{riak_kv, [
{allow_strfun, true}
]}
].
The other option is to create your own Erlang module by using the compiler shipped with Riak and placing the *.beam file in a well-known location for Riak to find. The basho-patches directory is one such place.
Please see the documentation as well:
advanced.config
Installing custom Erlang code
HTTP MapReduce
Using MapReduce
Advanced MapReduce
MapReduce / curl example
I have the most basic net/http program that I'm using to learn the namespace in Go:
package main
import (
"fmt"
"log"
"net/http"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Println(r.URL)
go HandleIndex(w, r)
})
fmt.Println("Starting Server...")
log.Fatal(http.ListenAndServe(":5678", nil))
}
func HandleIndex(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(200)
w.Write([]byte("Hello, World!"))
}
When I run the program and connect to localhost:5678 in Chrome, I get this in the console:
Starting Server...
/
2015/01/15 13:41:29 http: multiple response.WriteHeader calls
/favicon.ico
2015/01/15 13:41:29 http: multiple response.WriteHeader calls
But I don't see how that's possible. I print the URL, start up a new goroutine, write the header once, and give it a static body of Hello, World! It seems like one of two things is happening. Either something behind the scenes is writing another header or somehow HandleIndex is called twice for the same request. What can I do to stop writing multiple headers?
EDIT: It seems to have something to do with the go HandleIndex(w, r) line because if I remove go and just make it a function call instead of a goroutine, I don't get any issues and the browser gets it's data. With it being a goroutine, I get the multiple WriteHeader error and the browser doesn't show "Hello World." Why is making this a goroutine breaking it?
Take a look at the anonymous function you register as the handler of incoming requests:
func(w http.ResponseWriter, r *http.Request) {
fmt.Println(r.URL)
go HandleIndex(w, r)
}
It prints the URL (to the standard output) then calls HandleIndex() in a new goroutine and continues execution.
If you have a handler function where you do not set the response status before the first call to Write, Go will automatically set the response status to 200 (HTTP OK). If the handler function does not write anything to the response (and does not set the response status and completes normally), that is also treated as a successful handling of the request and the response status 200 will be sent back. Your anonymous function does not set it, it does not even write anything to the response. So Go will do just that: set the response status to 200 HTTP OK.
Note that handling each request runs in its own goroutine.
So if you call HandleIndex in a new goroutine, your original anonymous function will continue: it will end and so the response header will be set - meanwhile (concurrently) your started new goroutine will also set the response header - hence the "multiple response.WriteHeader calls" error.
If you remove the "go", your HandleIndex function will set the response header in the same goroutine before your handler function returns, and the "net/http" will know about this and will not try to set the response header again, so the error you experienced will not happen.
You already received a correct answer which addresses your problem, I will give some information about the general case (such error appears often).
From the documentation, you see that WriteHeader sends an http status code and you can't send more than 1 status code. If you Write anything this is equivalent to sending 200 status code and then writing things.
So the message that you see appears if you either user w.WriteHeader more than once explicitly or uses w.Write before w.WriteHeader.
From the documentation:
// WriteHeader sends an HTTP response header with status code.
// If WriteHeader is not called explicitly, the first call to Write
// will trigger an implicit WriteHeader(http.StatusOK).
What is happening in your case is that you are launching go HandleIndex from the handler.
The first handler finishes. The standard WriteHeader writes to the ResponseWriter. Then the go routine HandleIndex is launched and it also tries to write a header and write.
Just remove the go from HandleIndex and it will work.
the root cause is that you called WriteHeader more than once. from the source codes
func (w *response) WriteHeader(code int) {
if w.conn.hijacked() {
w.conn.server.logf("http: response.WriteHeader on hijacked connection")
return
}
if w.wroteHeader {
w.conn.server.logf("http: multiple response.WriteHeader calls")
return
}
w.wroteHeader = true
w.status = code
if w.calledHeader && w.cw.header == nil {
w.cw.header = w.handlerHeader.clone()
}
if cl := w.handlerHeader.get("Content-Length"); cl != "" {
v, err := strconv.ParseInt(cl, 10, 64)
if err == nil && v >= 0 {
w.contentLength = v
} else {
w.conn.server.logf("http: invalid Content-Length of %q", cl)
w.handlerHeader.Del("Content-Length")
}
}
}
so when you wrote once, the variable wroteHeader would be true, then you wrote header again, it wouldn't be effective and gave a warning "http: multiple respnse.WriteHeader calls".
actually the function Write also calls WriteHeader, so putting the function WriteHeader after the function Write also causes that error, and the later WriteHeader doesn't work.
from your case, go handleindex runs in another thread and the original already returns, if you do nothing, it will call WriteHeader to set 200. when running handleindex, it calls another WriteHeader, at that time wroteHeader is true, then the message "http: multiple response.WriteHeader calls" is output.
Yes, use HandleIndex(w, r) instead of go HandleIndex(w, r) will fix your issue, I think you have already figured that out.
The reason is simple, when handling multiple requests at the same time, the http server will start multiple goroutines, and your handler function will be called separately in each of the goroutines without blocking others.
You don't need to start your own goroutine in the handler, unless you practically need it, but that will be another topic.
Because modern browsers send an extra request for /favicon.ico which is also handled in your / request handler.
If you ping your server with curl for example, you'll see only one request being sent:
curl localhost:5678
To be sure you can add an EndPoint in your http.HandleFunc
http.HandleFunc("/Home", func(w http.ResponseWriter, r *http.Request)