I'm reading a JSON file that contains Unix Epoch dates, but they are strings in the JSON. In Go, can I convert a string in the form "1490846400" into a Go time.Time?
There is no such function in time package, but it's easy to write:
func stringToTime(s string) (time.Time, error) {
sec, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return time.Time{}, err
}
return time.Unix(sec, 0), nil
}
Playground: https://play.golang.org/p/2h0Vd7plgk.
There's nothing wrong, or incorrect about the answer provided by #Ainar-G, but likely a better way to do this is with a custom JSON unmarshaler:
type EpochTime time.Time
func (et *EpochTime) UnmarshalJSON(data []byte) error {
t := strings.Trim(string(data), `"`) // Remove quote marks from around the JSON string
sec, err := strconv.ParseInt(t, 10, 64)
if err != nil {
return err
}
epochTime := time.Unix(sec,0)
*et = EpochTime(epochTime)
return nil
}
Then in your struct, replace time.Time with EpochTime:
type SomeDocument struct {
Timestamp EpochTime `json:"time"`
// other fields
}
Related
I'm trying to append an id (and other info) to the url, so I can access it later, but I can't find the right method after some research.
I've tried to use Get() method, query(), Add(), but I couldn't redirect the URL.
var email_ployer string
func RegisterNewPloyer(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/ployer/register" {
http.Error(w, "404 not found.", http.StatusNotFound)
return
}
db := connect.ConnectDB()
defer db.Close()
switch r.Method {
case "POST":
email_ployer = r.FormValue("email")
senha := r.FormValue("senha")
senha, _ = HashPassword(senha)
tx, _ := db.Begin()
stmt, _ := tx.Prepare("INSERT INTO ployers(email_ployer, senha_ployer) VALUES(?,?)")
_, erro := stmt.Exec(email_ployer, senha)
if erro != nil {
tx.Rollback()
log.Fatal(erro)
}
tx.Commit()
}
Redirect(w, r)
}
func Redirect(w http.ResponseWriter, r *http.Request) {
db2 := connect.ConnectDB()
defer db2.Close()
var id string
tx, _ := db2.Begin()
rows, _ := tx.Query("SELECT id FROM ployers WHERE email_ployer = '?'", email_ployer)
for rows.Next() {
if err := rows.Scan(&id); err != nil {
log.Fatal(err)
}
if err := rows.Err(); err != nil {
log.Fatal(err)
}
}
http.Redirect(w, r, x, http.StatusSeeOther)
}
func main() {
http.HandleFunc("/ployer/seja-um-de-nos", LoadPloyerContent)
http.HandleFunc("/ployer/register", register.RegisterNewPloyer)
http.HandleFunc("/ployer/complete/", LoadPloyerContent)
http.HandleFunc("/ployer/register-received", LoadPloyerContent)
log.Fatal(http.ListenAndServe(":3306", nil))
}
In my system, I want the user to register his E-mail and password, create an new user in the DB and redirect the URL to something like localhost:3306/ployer/complete/id
Just use res.LastInsertId() to get the id and pass it to your redirect function, and build the url:
func RegisterNewPloyer(w http.ResponseWriter, r *http.Request) {
//...
//...
res, erro := stmt.Exec(email_ployer, senha)
if erro != nil {
tx.Rollback()
log.Fatal(erro)
}
tx.Commit()
id, erro := res.LastInsertId()
if erro != nil {
log.Fatal(erro)
}
Redirect(w, r, id)
}
func Redirect(w http.ResponseWriter, r *http.Request, id int64) {
uri := fmt.Sprintf("http://localhost:3306/ployer/complete/%d", id)
http.Redirect(w, r, uri, http.StatusSeeOther)
}
You should use url.Values to construct a map of query string values. Later on, you can use Encode method to generate encoded query string. Assign this value to Url.RawQuery to get the final output.
e.g.:
base, _ := url.Parse("https://www.foo.com/")
// Create an instance of Values to hold your query string parameters
values := url.Values{}
values.Add("abc", "def")
// Set the encoded output to RawQuery attribute of Url
base.RawQuery = values.Encode()
println(base.String())
If you are starting with a URL that already has query string parameters and you want add/modify items in it, use Query() method to obtain a reference to parsed map.
e.g.:
base, _ := url.Parse("https://www.foo.com/?a=b")
values := base.Query()
values.Set("a", "c")
values.Set("x", "y")
// Set the encoded output to RawQuery attribute of Url
base.RawQuery = values.Encode()
println(base.String())
I couldn't find anything helpful online on this one.
I am writing an REST API, and I want to log the size of the body of the request in bytes for metrics. Go net/http API does not provide that directly. http.Request does have Content-Length field, but that field can be empty or the client might send false data.
Is there a way to get that in the middlware level? The bruteforce method would be to read the full body and check the size. But if I do that in the middleware, the handler will not have access to the body because it would have been read and closed.
Why do you want a middle in here?
The simple way is b, err = io.Copy(anyWriterOrMultiwriter, r.Body)
b is total content length of request when err == nil
Use request body as you want. Also b, err = io.Copy(ioutil.Discard, r.Body)
You could write a custom ReadCloser that proxies an existing one and counts bytes as it goes. Something like:
type LengthReader struct {
Source io.ReadCloser
Length int
}
func (r *LengthReader) Read(b []byte) (int, error) {
n, err := r.Source.Read(b)
r.Length += n
return n, err
}
func (r *LengthReader) Close() error {
var buf [32]byte
var n int
var err error
for err == nil {
n, err = r.Source.Read(buf[:])
r.Length += n
}
closeerr := r.Source.Close()
if err != nil && err != io.EOF {
return err
}
return closeerr
}
This will count bytes as you read them from the stream, and when closed it will consume and count all remaining unread bytes first. After you're finished with the stream, you can then access the length.
Option 1
Use TeeReader and this is scalable. It splits reader into two and one of them calculates the size using allocated memory. Also, in the first case
maxmem := 4096
var buf bytes.Buffer
// comment this line out if you want to disable gathering metrics
resp.Body = io.TeeReader(resp.Body, &buf)
readsize := func(r io.Reader) int {
bytes := make([]byte, maxmem)
var size int
for {
read, err := r.Read(bytes)
if err == io.EOF {
break
}
size += read
}
return size
}
log.Printf("Size is %d", readsize(&buf))
Option 2 unscalable way (original answer)
You can just read the body, calculate the size, then unmarshal into struct, so that it becomes:
b, _ := ioutil.ReadAll(r.Body)
size := len(b) // can be nil so check err in your app
if err := json.Unmarshal(b, &input); err != nil {
s.BadReq(w, errors.New("error reading body"))
return
}
Im trying to build a small website, I use the html/template to create dynamic pages. One thing on the pages is a list of URL's inside those urls sometimes I need character encoding. for special characters like ô (%C3%B4).
When i try to parse the variables into a page using html/template i get the following as a result: %!c(MISSING)3%!b(MISSING)4. I have no clue what is wrong here
type Search_list struct {
Search_name string
Search_url string
Search_price float64
}
func generateSearchPage(language int, q string) (string, error) {
/* ommited, fetshing data from elasticsrearch*/
sl := []Search_list{}
var urle *url.URL
//looping through ES results and putting them in a custom List
for _, res := range data.Hits.Hits {
//
//Encode Url
var err error
urle, err = url.Parse(res.Source.URL)
if err != nil {
continue
// TODO: add log
}
//I've tried already the following:
fmt.Println(res.Source.URL) //ô
fmt.Println(url.QueryUnescape(res.Source.URL)) //ô
fmt.Println(urle.String()) //%C3%B4
u, _ := url.QueryUnescape(res.Source.URL)
sl = append(sl, Search_list{res.Source.Name, u, res.Source.Price})
}
var buffer bytes.Buffer
t := template.New("Index template")
t, err = t.Parse(page_layout[language][PageTypeSearch])
if err != nil {
panic(err)
}
err = t.Execute(&buffer, Search_data{
Title: translations[language]["homepage"],
Page_title: WebSiteName,
Listed_items: sl,
})
if err != nil {
panic(err)
}
return buffer.String(), nil // %!c(MISSING)3%!b(MISSING)4
}
# Moshe Revah
thanks for the help, in the meantime I found the error
Later in the code I send my generated page to the http client with
fmt.Fprintf(w, page) // Here was the error b/c of the % symbols
I just changed it to
fmt.Fprint(w, page)
and it works perfect
I'm trying to make a simple package to send SSH commands to a server.
I have the following code:
type Connection *ssh.Client
func Connect(addr, user, password string) (conn Connection, err error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
return
}
func (conn Connection) SendCommand() ([]byte, error) {
session, err := (*ssh.Client)(conn).NewSession()
// ...
}
My problem is on the two lines func (conn Connection) SendCommand() ([]byte, error) and session, err := (*ssh.Client)(conn).NewSession().
I can't figure out how to use the methods available for *ssh.Client from my overlaying Connection type.
I understand that I need to do some conversion, and using ssh.Client(*conn).NewSession() would work, but it copies the values of the *ssh.Client which doesn't seem to be the right method.
What should do to access the methods available for a *ssh.Client when working with my custom type Connection *ssh.Client type?
You can't declare a new type with a pointer TypeSpec. Also declaring a new type is used specifically to remove the entire method set, so you won't have any of the original methods from the *ssh.Client.
What you want is to use composition by embedding the *ssh.Client in your own struct type:
type Connection struct {
*ssh.Client
}
func Connect(addr, user, password string) (*Connection, error) {
sshConfig := &ssh.ClientConfig{
User: user,
Auth: []ssh.AuthMethod{
ssh.Password(password),
},
HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
}
conn, err = ssh.Dial("tcp", addr, sshConfig)
if err != nil {
return nil, err
}
return &Connection{conn}, nil
}
func (conn *Connection) SendCommand() ([]byte, error) {
session, err := conn.NewSession()
// ...
}
This is the best I can come up with:
type Connection ssh.Client
func (conn *Connection) SendCommand() ([]byte, error) {
(*ssh.Client)(conn).NewSession()
Note that I've changed the type to not be a pointer type (but then I've made a pointer receiver for SendCommand). I'm not sure there's any way to create a function with a pointer type as a receiver.
Another option is to use type aliasing to achieve the desired behavior. I was trying to do something "clever" for readability:
type foo struct {
i int
}
type foo_ptr = *foo
type foo_ptr_slice = []foo_ptr
type foo_ptr_map = map[string]foo_ptr
type foo_ptr_slice_map = map[string]foo_ptr_slice
func (r foo_ptr) dump() {
fmt.Printf("%d\n", r.i)
}
func main() {
// need a map of slice of pointers
var m foo_ptr_map
m = make(foo_ptr_map, 0)
m["test"] = &foo{i: 1}
var m2 foo_ptr_slice_map
m2 = make(foo_ptr_slice_map, 0)
m2["test"] = make(foo_ptr_slice, 0, 10)
m2["test"] = append(m2["test"], &foo{i: 2})
fmt.Printf("%d\n", m["test"].i)
fmt.Printf("%d\n", m2["test"][0].i)
m["test"].dump()
}
I acknowledge that type aliasing is used for large-scale refactoring but this seems like a very good use for readability sake.
I'm trying to send a int64 over a TCP in golang, however, my receiver prints gets a different number then what I've sent out. What is the proper way to accomplish this?
//Buffer on both client and server
buffer := make([]byte, 1024)
//Sender
fileInfo, error := os.Stat(fileName)
if error != nil {
fmt.Println("Error opening file")
}
var fSize int = int(fileInfo.Size())
connection.Write([]byte(string(fSize)))
//Receiver
connection.Read(buffer)
fileSize := new(big.Int).SetBytes(bytes.Trim(buffer, "\x00")).Int64()
if err != nil {
fmt.Println("not a valid filesize")
fileSize = 0
}
Using binary.Write / binary.Read:
//sender
err := binary.Write(connection, binary.LittleEndian, fileInfo.Size())
if err != nil {
fmt.Println("err:", err)
}
//receiver
var size int64
err := binary.Read(connection, binary.LittleEndian, &size)
if err != nil {
fmt.Println("err:", err)
}
[]byte(string(fSize)) doesn't do what you think it does, it treats the number as unicode character, it doesn't return the string representation of it.
If you want the string representation of a number, use strconv.Itoa, if you want the binary represention then use:
num := make([]byte, 8) // or 4 for int32 or 2 for int16
binary.LittleEndian.PutUint64(num, 1<<64-1)
Use binary.BigEndian or binary.LittleEndian to encode the integer:
var size int64
// Send
var buf [8]byte
binary.BigEndian.PutUint64(buf[:], uint64(size))
_, err := w.Write(buf[:])
// Receive
var buf [8]byte
_, err := io.ReadFull(r, buf[:])
if err != nil {
// handle error
}
size = int64(binary.BigEndian.Uint64(buf[:])
You can also use the binary.Read and binary.Write. Your application code will be a little shorter at the cost of type switches and other goo inside these functions.
A couple of points about the code in the question. The conversion
string(fSize)
returns the UTF-8 representation of the rune fSize. It does not return a decimal encoding or binary encoding the value. Use the strconv packate to convert a numeric value to a decimal representation. Use the above mentioned binary package to convert to binary representation.
The sequence
connection.Read(buffer)
buffer = bytes.Trim(buffer, "\x00")
trims away real data if the data happens to include a 0 byte at the ends. Read returns the number of bytes read. Use that length to slice the buffer:
n, err := connection.Read(buffer)
buffer = buffer[:n]
You can't use string() to cast from an int, you need to use the strconv package.
connection.Write([]byte(strconv.FormatInt(fileInfo.Size(), 10))