I'm running into the most bizarre issue regarding datestring parsing in golang using the time package.
The Error:
parsing time "07-20-2018" as "2006-01-02": cannot parse "0-2018" as "2006"
The Code Block:
log.Println(datestring) //07-20-2018
date, err := time.Parse("2006-01-02", datestring)
log.Println(err) //parsing time "07-20-2018" as "2006-01-02": cannot parse "0-2018" as "2006"
log.Println(date) //parsing time "07-20-2018" as "2006-01-02": cannot parse "0-2018" as "2006"
I'm completely at a loss to what this issue is referring to, the string is parsed from the URI in golang with gorilla mux.
datestring, _ := vars["date"] //some/path/{date}, date is 07-20-2018
Any ideas?
It is obvious. You are trying to parse mm-dd-yyyy as yyyy-mm-dd.
A simple fix:
package main
import (
"fmt"
"time"
)
func main() {
datestring := "07-20-2018"
fmt.Println(datestring)
date, err := time.Parse("01-02-2006", datestring)
fmt.Println(date, err)
}
Playground: https://play.golang.org/p/gK7cMAkrP7l
Output:
07-20-2018
2018-07-20 00:00:00 +0000 UTC <nil>
See Go Package time.
Related
I've been struggling the last couple of days with a problem which I am not sure if its Golang's limitation or I am doing sth completely wrong.
Experiment:
I am trying to benchmark the time needed for making a simple HTTP GET request to an external server. (I am hosting on EKS a web app)
I have made a Python implementation using simple requests module:
import requests
import time
start = time.time()
r = requests.get('https://httpbin.org/net')
end = time.time()
print(f'Time taken: {(end - start)*1000}ms')
Output:
Time taken: 338.42525ms
On average the time is needed for this URL is ~320ms.
Now the Golang equivalent:
import (
"fmt"
"net/http"
"os"
"time"
"github.com/go-resty/resty/v2"
)
func main() {
startTime := time.Now()
tr := &http.Transport{
MaxIdleConns: 100,
IdleConnTimeout: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
MaxConnsPerHost: 1000,
//DisableKeepAlives: true,
MaxIdleConnsPerHost: 1000,
}
resp, _ := resty.New().SetTransport(tr).R().EnableTrace().SetHeader("Accept", "application/json").Get(os.Args[1])
endTime := time.Since(startTime)
fmt.Println("Total Time", endTime)
fmt.Println("Request Fire Time", resp.Request.Time)
fmt.Println("Response Received Time (body not read yet)", resp.ReceivedAt())
fmt.Println("Diff request fire & receive", resp.Time())
Output:
Total Time 310.143542ms
Request Fire Time 2022-08-01 11:43:16.750401 +0300 EEST m=+0.000985376
Response Received Time (body not read yet) 2022-08-01 11:43:17.060387 +0300 EEST m=+0.310971126
Diff request fire & receive 309.724667ms
Note I am using go-resty (but the behaviour is similar to native net/http package.
As you see, for this URL (which is public and not HTTPS), both worlds compete the same.
Things are starting to change when I am using my custom web app which is hosted on EKS (I am using VPN to access it from my local machine).
Same code with only URL change using HTTPS (so TLSHandshake is involved) reports back the following numbers:
Python program:
Time taken: 578.6697864532471ms
Golang program
Total Time 761.919125ms
Request Fire Time 2022-08-01 11:49:10.911217 +0300 EEST m=+0.000936917
Response Received Time (body not read yet) 2022-08-01 11:49:11.673035 +0300 EEST m=+0.762752167
Diff request fire & receive 205.553333ms
So, why is there such a huge change between them? (~250ms)
In general I am using python asyncio to spin up concurrently such requests and at the same time go routines to achieve it in Golang. So my Golang program takes around 1minute and20s to complete a bunch of 100 URLs, while Python takes only ~8s.
What I don't understand also is the statistics from go-resty:
While the time measuring using simple time.Now() reports that from the time I sent the request till I received it takes ~800ms, the Time() method of the resty client says that the actual response took ~240ms which according to the documentation doesn't make sense.
So how are these numbers sum up to match each other?
Total time is ~250ms but the time measured in my code using time.Now() and time.Since(start) says ~800ms.
I have just put this code to measure it:
start := time.Now()
http_client.Get(url)
end := time.Since(start)
The http_client.Get(url) simply is doing the following:
tr := &http.Transport{
MaxIdleConns: 100,
IdleConnTimeout: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
MaxConnsPerHost: 1000,
//DisableKeepAlives: true,
MaxIdleConnsPerHost: 1000,
}
resp, err := NewClient().SetTransport(tr).R().EnableTrace().SetHeader("Accept", "application/json").Get(url)
fmt.Println("Response Info:")
fmt.Println(" Error :", err)
fmt.Println(" Request time:", resp.Request.Time)
fmt.Println(" Status Code:", resp.StatusCode())
fmt.Println(" Status :", resp.Status())
fmt.Println(" Proto :", resp.Proto())
fmt.Println(" Time :", resp.Time())
fmt.Println(" Received At:", resp.ReceivedAt())
//fmt.Println(" Body :\n", resp)
fmt.Println()
// Explore trace info
fmt.Println("Request Trace Info:")
ti := resp.Request.TraceInfo()
fmt.Println(" DNSLookup :", ti.DNSLookup)
fmt.Println(" ConnTime :", ti.ConnTime)
fmt.Println(" TCPConnTime :", ti.TCPConnTime)
fmt.Println(" TLSHandshake :", ti.TLSHandshake)
fmt.Println(" ServerTime :", ti.ServerTime)
fmt.Println(" ResponseTime :", ti.ResponseTime)
fmt.Println(" TotalTime :", ti.TotalTime)
fmt.Println(" IsConnReused :", ti.IsConnReused)
fmt.Println(" IsConnWasIdle :", ti.IsConnWasIdle)
fmt.Println(" ConnIdleTime :", ti.ConnIdleTime)
fmt.Println(" RequestAttempt:", ti.RequestAttempt)
fmt.Println(" RemoteAddr :", ti.RemoteAddr.String())
Response Info:
Error : <nil>
Request time: 2022-08-01 11:08:01.772263 +0300 EEST m=+0.019878460
Status Code: 200
Status : 200 OK
Proto : HTTP/2.0
Time : 252.098375ms
Received At: 2022-08-01 11:08:02.594281 +0300 EEST m=+0.841889668
Request Trace Info:
DNSLookup : 2.92825ms
ConnTime : 458ns
TCPConnTime : 174.664125ms
TLSHandshake : 203.444625ms
ServerTime : 251.888542ms
ResponseTime : 209.375µs
TotalTime : 252.098375ms
IsConnReused : true
IsConnWasIdle : true
ConnIdleTime : 76.292µs
RequestAttempt: 1
RemoteAddr : 10.0.1.178:443
{"level":"info","ts":1659341282.597458,"msg":"Elapsed time to get network response: 826.428334ms"}
this is my function:
func downloadDoc(c *gin.Context) {
var fileToSearch service.ApDocumentsMedia
if err := c.BindJSON(&fileToSearch); err != nil {
c.AbortWithStatusJSON(http.StatusUnprocessableEntity, "is not binding!")
return
}
var file service.ApDocumentsMedia
if err := db.Where("uuid = ?", fileToSearch.Uuid).First(&file); err.RowsAffected <= 0 {
c.IndentedJSON(http.StatusNotFound, "Document not founded!")
return
}
c.Header("Content-Description", "File Transfer")
c.Header("Content-Transfer-Encoding", "binary")
c.Header("Content-Disposition", "attachment; filename="+strconv.Quote(file.Path))
c.Header("Content-Type", "application/octet-stream; charset=utf-8")
c.File(file.Path)
c.IndentedJSON(http.StatusOK, "File inviato")
}
but then when I go to call the function (using the frontend) it gives me this error:
http: panic serving 127.0.0.1:50138: http: wrote more than the declared Content-Length
goroutine 24 [running]:
net/http.(*conn).serve.func1()
/home/stage01/sdk/go1.18.3/src/net/http/server.go:1825 +0xbf
panic({0xac00e0, 0xc000020ab0})
/home/stage01/sdk/go1.18.3/src/runtime/panic.go:844 +0x258
github.com/gin-gonic/gin.(*Context).Render(0xc0000b2900, 0xc8, {0xc8d318, 0xc000245950})
/home/stage01/go/pkg/mod/github.com/gin-gonic/gin#v1.8.1/context.go:911 +0x112
github.com/gin-gonic/gin.(*Context).IndentedJSON(...)
/home/stage01/go/pkg/mod/github.com/gin-gonic/gin#v1.8.1/context.go:928
main.downloadDoc(0xc0000b2900)
/home/stage01/Scrivania/minia_git/docmanagement-alexperrucci/api/main.go:74 +0x36d
github.com/gin-gonic/gin.(*Context).Next(...)
/home/stage01/go/pkg/mod/github.com/gin-gonic/gin#v1.8.1/context.go:173
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000449520, 0xc0000b2900)
/home/stage01/go/pkg/mod/github.com/gin-gonic/gin#v1.8.1/gin.go:616 +0x671
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000449520, {0xc8f430?, 0xc0004ee620}, 0xc0000b2700)
/home/stage01/go/pkg/mod/github.com/gin-gonic/gin#v1.8.1/gin.go:572 +0x1dd
net/http.serverHandler.ServeHTTP({0xc8c968?}, {0xc8f430, 0xc0004ee620}, 0xc0000b2700)
/home/stage01/sdk/go1.18.3/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc00054b360, {0xc8ff38, 0xc00028f290})
/home/stage01/sdk/go1.18.3/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
/home/stage01/sdk/go1.18.3/src/net/http/server.go:3071 +0x4db
This problem is given to me when once I click on the button that makes the post call (with the uuid of the file) it gives me this error and I don't understand why, would anyone know how to help me?
i have created a struct and it contains two time.Time formatted fields, named with json tags: start_time and end_time.
type MyStruct struct {
StartTime time.Time `json:"start_time"`
EndTime time.Time `json:"end_time"`
}
when i'm trying to send a PUT request over HTTP using gin framework to update those values, the time format which i'm sending, changes in sent struct.
what i'm sending:
curl -X PUT -H 'Content-Type: application/json'
http://my_address -d '{
"start_time": "2021-04-27T22:24:31Z",
"end_time": "2021-11-01T22:24:31Z"
}'
what it receives:
start_time="2021-04-27 22:24:31 +0000 UTC",
end_time="2021-11-01 22:24:31 +0000 UTC",
in the other hand,
i'm saving the struct in a couchbase and as returning value of query, i'm sending back the document(my struct):
my query:
Update BucketName as e
set start_time="2021-04-27 22:24:31 +0000 UTC",
end_time="2021-11-01 22:24:31 +0000 UTC" where ( my document equality condition)
returning e
and it executes with no errors.
when i'm trying to read the returned struct,
my code to reading it:
var s domain.MyStructSample //
err = result.One(&s)
if err != nil {
if err == gocb.ErrNoResult {
return nil, errors.New("there is no result")
}
logger.ZSLogger.Errorf("error on update one item from my struct with error :%s", err)
return nil, err
}
gocb generates errors on those time items and here is the error:
"message":"error on update one item from my struct with error :parsing time \"\"2021-11-01 22:24:31 +0000 UTC\"\" as \"\"2006-01-02T15:04:05Z07:00\"\": cannot parse \" 22:24:31 +0000 UTC\"\" as \"T\""}
by the way, as i said, update is done with no errors ( query executes with no errors).
so what should i do with it ?
How did you generate this query:
Update BucketName as e
set start_time="2021-04-27 22:24:31 +0000 UTC",
end_time="2021-11-01 22:24:31 +0000 UTC" where ( my document equality condition)
returning e
As the error says, time data stored in couchbase should be in format RFC3339 (2006-01-02T15:04:05Z07:00) instead of the default 2006-01-02 15:04:05 -0700 MST, so maybe you should insert data with query:
Update BucketName as e
set start_time="2021-04-27T22:24:31Z07:00",
end_time="2021-11-01T22:24:31Z07:00" where ( my document equality condition)
returning e
If you have problem formatting time, read the doc https://golang.cafe/blog/golang-time-format-example.html
And, as #MrFuppes commented, if you need to customize JSON output format, read this How to format timestamp in outgoing JSON
I'm trying to get a list of system interfaces on my machine, I'm using the net package, which is defined here.
I have this tiny little snippet of code, which crashes when trying to print out the error, and I can't figure out why the error is sigsev'ing on me. It's supposed to return an error, or nil right?
I've ran this code with sudo, root and under a regular user account thinking it might be a permissions thing for the network interface, but it persists amongst all user levels.
package main
import (
"net"
"fmt"
)
func main() {
var err error
var interfaces []net.Interface
var ifString []string
interfaces, err = net.Interfaces()
fmt.Printf("%v",interfaces)
if err != nil {
for _, v := range interfaces {
ifString = append(ifString, v.Name)
}
} else {
fmt.Printf(err.Error())
ifString = append(ifString, "unable to get system interfaces")
}
}
Program output is as follows when running go build and executing it:
[{1 65536 lo up|loopback} {2 1500 eno1 b8:cc:3c:8e:d4:d3 up|broadcast|multicast} {9 1500 tun0 up|pointtopoint|multicast}]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4a5269]
goroutine 1 [running]:
main.main()
/home/andrew/interface/borked.go:20 +0x269
You are receiving a nil pointer dereference because that's exactly what you're doing.
Given the following if statement in your code:
if err != nil {
for _, v := range interfaces {
ifString = append(ifString, v.Name)
}
} else {
fmt.Printf(err.Error())
...
}
The else is reached when err is nil. Yet you are attempting to access a field in it: err.Error(), dereferencing a nil pointer.
You need to invert your if statement, it's the wrong way around.
I am struggling to know how to access the response to an Indy POST request. I post the data either as JSON or paramstring. My code when using JSON is as follows.
params := TStringList.Create;
try
params.Text :=
'{'
+ format ('"client_secret":"%s",', [FilesFrm.ClientSecret])
+ format ('"client_id":"%s",', [FilesFrm.ClientId])
+ '"grant_type":"authorization_code",'
+ '"redirect_uri":"http://localhost:8080",'
+ format ('"code":"%s"', [fCode])
+ '}';
idLogFile1.Active := true;
// Make sure it uses HTTP 1.1, not 1.0
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoKeepOrigProtocol];
IdHTTP1.Request.ContentType := 'application/json';
IdHttp1.Request.Accept := 'application/vnd.hmrc.1.0+json';
try
result := IdHTTP1.Post (
'https://test-api.service.hmrc.gov.uk/oauth/token',
params);
except
on E: Exception do
memo1.lines.add (E.ClassName + ': ' + E.message);
end;
memo1.Lines.add (result);
memo1.Lines.add (idHTTP1.ResponseText);
finally
params.free;
end;
The result of printing out result and RepsonseText is just
EIdHTTPProtocolException: HTTP/1.1 400 Bad Request
HTTP/1.1 400 Bad Request
However, because I have a TidLogFile component attached to the TidHTTP, I can see what actually arrives, which is the following.
Recv 2/1/2019 7:56:07 AM: HTTP/1.1 400 Bad Request<EOL>
Content-Type: application/json<EOL>
Content-Length: 72<EOL>
Cache-Control: no-cache,no-store,etc, etc...
; Secure; HTTPOnly<EOL><EOL>
{"error":"invalid_request","error_description":"grant_type is required"}
Aside from the fact that grant_type appears to be in the original request data, I would like to be able to access the JSON response at the end, since "grant_type_is_required" is much more helpful than "Bad request", but I cannot find where it is.
I have subsequently found Response.ContentLength, which contains the value 72, and Response.ContentStream, which should in theory contain the 72 bytes of data, but produces access violations when I try to extract the data.
len := idHTTP1.Response.ContentLength;
memo1.Lines.Add(format ('Length = %d', [len]));
if assigned (idHTTP1.Response.ContentStream) then
begin
//idHTTP1.Response.ContentStream.Position := 0;
result := ReadStringFromStream (idHTTP1.Response.ContentStream, len);
end;
memo1.lines.add (result);
In addition to mjn42's answer, which is technically correct, TIdHTTP also has optional hoNoProtocolErrorException and hoWantProtocolErrorContent flags in its HTTPOptions property, which you can enable to avoid the EIdHTTPProtocolException being raised and to populate your result variable with error data, respectively:
params := TStringStream.Create(
'{'
+ format ('"client_secret":"%s",', [FilesFrm.ClientSecret])
+ format ('"client_id":"%s",', [FilesFrm.ClientId])
+ '"grant_type":"authorization_code",'
+ '"redirect_uri":"http://localhost:8080",'
+ format ('"code":"%s"', [fCode])
+ '}',
TEncoding.UTF8);
try
IdLogFile1.Active := true;
// Make sure it uses HTTP 1.1, not 1.0,
// and disable EIdHTTPProtocolException on errors
IdHTTP1.ProtocolVersion := pv1_1;
IdHTTP1.HTTPOptions := IdHTTP1.HTTPOptions + [hoKeepOrigProtocol, hoNoProtocolErrorException, hoWantProtocolErrorContent];
IdHTTP1.Request.ContentType := 'application/json';
IdHTTP1.Request.Accept := 'application/vnd.hmrc.1.0+json';
try
result := IdHTTP1.Post('https://test-api.service.hmrc.gov.uk/oauth/token', params);
except
on E: Exception do begin
Memo1.Lines.Add(E.ClassName + ': ' + E.message);
raise;
end;
end;
Memo1.Lines.Add(result);
finally
params.Free;
end;
Here is an example which shows how the HTTP body can be accessed.
The body can be accessed if you catch the EIdHTTPProtocolException exception.
on E: EIdHTTPProtocolException do
begin
WriteLn(E.Message);
WriteLn(E.ErrorMessage);
end;
Full example code:
program JSONPostExample;
{$APPTYPE CONSOLE}
uses
IdHTTP, IdGlobal, SysUtils, Classes;
var
HTTP: TIdHTTP;
RequestBody: TStream;
ResponseBody: string;
begin
HTTP := TIdHTTP.Create;
try
try
RequestBody := TStringStream.Create('{"日本語":42}',
TEncoding.UTF8);
try
HTTP.Request.Accept := 'application/json';
HTTP.Request.ContentType := 'application/json';
ResponseBody := HTTP.Post('https://httpbin.org/post',
RequestBody);
WriteLn(ResponseBody);
WriteLn(HTTP.ResponseText);
finally
RequestBody.Free;
end;
except
on E: EIdHTTPProtocolException do
begin
WriteLn(E.Message);
WriteLn(E.ErrorMessage);
end;
on E: Exception do
begin
WriteLn(E.Message);
end;
end;
finally
HTTP.Free;
end;
ReadLn;
ReportMemoryLeaksOnShutdown := True;
end.
Note that you must not use a TStringList for the POST body. That version of TIdHTTP.Post() formats the data according to the application/x-www-form-urlencoded media type, which is not appropriate for JSON and will corrupt it.