How to listen to firestore through RPC - firebase

I want to listen to real time changes in firestore and I am also only allowed to use Go. Since firestore SDK for Go doesn't have any option to listen for real time changes, I decided to use the firestore v1beta1 sdk.
I have written the following code to do that
func TestRPCHandler(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
c, err := firestore.NewClient(context.Background())
databaseName := "projects/[project_name]/databases/(default)"
if err != nil {
panic(err)
}
stream, err := client.Listen(context.Background())
if err != nil {
panic(err)
}
request := &firestorepb.ListenRequest{
Database: databaseName,
TargetChange: &firestorepb.ListenRequest_AddTarget{
AddTarget: &firestorepb.Target{
TargetType: &firestorepb.Target_Documents{
Documents: &firestorepb.Target_DocumentsTarget{
Documents: []string{"projects/[project_name]/databases/(default)/[collection_name]"} ,
},
},
},
},
}
if err := stream.Send(request); err != nil {
panic(err)
}
if err := stream.CloseSend(); err != nil {
panic(err)
}
for {
resp, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
}
}
When I am doing this, the code does not detect any changes I bring about manually in the database. stream.Recv() just returns EOF and exits immediately. I even tried manually waiting by adding time.Sleep() but that does not help either.

You don't need the beta SDK or hacks to make this happen, I found the solution, it's pretty easy actually.
The https://firebase.google.com/docs/firestore/query-data/listen documentation does not contain an example for Go.
The source code of the firestore client API for Go has an unexported watchStream which we cannot directly use: https://github.com/googleapis/google-cloud-go/blob/master/firestore/watch.go#L130
Deep search of the repository shows that this is actually used on the DocumentSnapshotIterator and QuerySnapshotIterator at: https://github.com/googleapis/google-cloud-go/blob/master/firestore/docref.go#L644 and: https://github.com/googleapis/google-cloud-go/blob/master/firestore/query.go#L716.
The Collection contains a Snapshots method which returns the snapshot iterator that we want, after that all is easy, we just make an infivitive loop through its Next method.
Example:
cols, err := client.Collections(context.Background()).GetAll()
for _, col := range cols {
iter := col.Snapshots(context.Background())
defer iter.Stop()
for {
doc, err := iter.Next()
if err != nil {
if err == iterator.Done {
break
}
return err
}
for _, change := range doc.Changes {
// access the change.Doc returns the Document,
// which contains Data() and DataTo(&p) methods.
switch change.Kind {
case firestore.DocumentAdded:
// on added it returns the existing ones.
isNew := change.Doc.CreateTime.After(l.startTime)
// [...]
case firestore.DocumentModified:
// [...]
case firestore.DocumentRemoved:
// [...]
}
}
}
}
Yours, Gerasimos Maropoulos aka #kataras

Firebase's Get realtime updates with Cloud Firestore documentation currently indicates that Go is not yet supported.
// Not yet supported in Go client library

Related

How should I handle the errors when I try to write in DB

Not sure how should I deal with errors when I try to write in DB in this particular case:
So I use this func to insert in DB
func SaveToDB(dateid string, content string) {
db, err := sql.Open("mysql", dbLink)
if err != nil {
log.Fatal(err)
}
queryString := fmt.Sprintf("INSERT INTO balances (dateid, content) VALUES('%v','%v');", dateid, content)
rows, err := db.Query(queryString)
if err != nil {
log.Fatal(err)
}
defer rows.Close()
}
But I would like to don't stop the server when error is duplicate so I tried this version:
func SaveToDB(dateid string, content string) {
db, err := sql.Open("mysql", dbLink)
if err != nil {
log.Fatal(err)
}
queryString := fmt.Sprintf("INSERT INTO balances (dateid, content) VALUES('%v','%v');", dateid, content)
rows, err1 := db.Query(queryString)
if err1 != nil {
thisErr, err2 := regexp.MatchString("Error 1062: Duplicate entry", err.Error())
if err2 != nil {
log.Fatal("ERROR: error occured while trying to perform regex on SaveToDB", err2)
}
if thisErr == true {
log.Println("ERROR: Not able to save in DB due to ducplicate: ", err1)
}else{log.Fatal("ERROR: error occured when trying to save to DB: ", err1)}
}
defer rows.Close()
}
But in this situation I receive panic. So how I can stop this function from executing before it reaches "defer rows.Close()"? I guess that is the reason for panic...
Don't use log.Fatal and install deferred code once you have the correct data:
rows, err := db.Query(queryString)
if err != nil {
log.Error(err)
return
}
defer rows.Close()

HTTP & RPC at the same time

I am trying to implement rpc and http server in my system.
So to listen servers at the same time I have run 2 goroutines
Here is a code
func main() {
// Recovering all errors during the process
defer errorHandler()
wg.Add(2)
go RPCConnect()
fmt.Println("Listening for RPC 127.0.0.1:" + config.rpcPort)
go HTTPConnect()
fmt.Println("Listening for HTTP 127.0.0.1:" + config.httpPort)
wg.Wait()
}
func RPCConnect() {
err := rpc.Register(pool)
if err != nil {
panic(err)
}
rpc.HandleHTTP()
listener, e := net.Listen("tcp", ":"+config.rpcPort)
if e != nil {
panic(e)
}
err = http.Serve(listener, nil)
if err != nil {
panic(err)
}
}
func HTTPConnect() {
var httpPool HTTPPool
r := mux.NewRouter()
r.HandleFunc("/create", httpPool.Create).Methods("POST")
r.HandleFunc("/generate", httpPool.Generate).Methods("POST")
r.HandleFunc("/list", httpPool.List).Methods("GET")
r.HandleFunc("/delete", httpPool.Delete).Methods("POST")
err := http.ListenAndServe("localhost:"+config.httpPort, r)
if err != nil {
panic(err)
}
}
I dont know is this a best way or not . Can somebody tell me more simple and flexible method?
I apologize in advance if the question is not relevant
Suggested approach is pretty well.
If you want to use only one port instead of two, there are several third-party tools to do it:
1) https://github.com/soheilhy/cmux
2) https://github.com/grpc-ecosystem/grpc-gateway

Unable to retrieve facebook live comments in real-time

I want to retrieve facebook live comments in real time. I have read this documentation
This is my implementation:
func getLiveComments(liveId, token string) {
url := fmt.Sprintf("https://streaming-graph.facebook.com/%s/live_comments?access_token=%s&comment_rate=one_per_two_seconds&fields=from{name,id},message",
liveId, url.QueryEscape(token))
resp, err := http.Get(url)
if err != nil {
fmt.Printf("Get: %s\n", err)
return
}
defer resp.Body.Close()
reader := bufio.NewReader(resp.Body)
for {
// got stuck here
line, err := reader.ReadBytes('\n')
if err != nil {
break
}
log.Println(string(line))
}
}
But it got stuck at line, err := reader.ReadBytes('\n').
I can use the liveId and token to get comments from facebook Graph API

How can I implement an inactivity timeout on an http download

I've been reading up on the various timeouts that are available on an http request and they all seem to act as hard deadlines on the total time of a request.
I am running an http download, I don't want to implement a hard timeout past the initial handshake as I don't know anything about my users connection and don't want to timeout on slow connections. What I would ideally like is to timeout after a period of inactivity (when nothing has been downloaded for x seconds). Is there any way to do this as a built in or do I have to interrupt based on stating the file?
The working code is a little hard to isolate but I think these are the relevant parts, there is another loop that stats the file to provide progress but I will need to refactor a bit to use this to interrupt the download:
// httspClientOnNetInterface returns an http client using the named network interface, (via proxy if passed)
func HttpsClientOnNetInterface(interfaceIP []byte, httpsProxy *Proxy) (*http.Client, error) {
log.Printf("Got IP addr : %s\n", string(interfaceIP))
// create address for the dialer
tcpAddr := &net.TCPAddr{
IP: interfaceIP,
}
// create the dialer & transport
netDialer := net.Dialer{
LocalAddr: tcpAddr,
}
var proxyURL *url.URL
var err error
if httpsProxy != nil {
proxyURL, err = url.Parse(httpsProxy.String())
if err != nil {
return nil, fmt.Errorf("Error parsing proxy connection string: %s", err)
}
}
httpTransport := &http.Transport{
Dial: netDialer.Dial,
Proxy: http.ProxyURL(proxyURL),
}
httpClient := &http.Client{
Transport: httpTransport,
}
return httpClient, nil
}
/*
StartDownloadWithProgress will initiate a download from a remote url to a local file,
providing download progress information
*/
func StartDownloadWithProgress(interfaceIP []byte, httpsProxy *Proxy, srcURL, dstFilepath string) (*Download, error) {
// start an http client on the selected net interface
httpClient, err := HttpsClientOnNetInterface(interfaceIP, httpsProxy)
if err != nil {
return nil, err
}
// grab the header
headResp, err := httpClient.Head(srcURL)
if err != nil {
log.Printf("error on head request (download size): %s", err)
return nil, err
}
// pull out total size
size, err := strconv.Atoi(headResp.Header.Get("Content-Length"))
if err != nil {
headResp.Body.Close()
return nil, err
}
headResp.Body.Close()
errChan := make(chan error)
doneChan := make(chan struct{})
// spawn the download process
go func(httpClient *http.Client, srcURL, dstFilepath string, errChan chan error, doneChan chan struct{}) {
resp, err := httpClient.Get(srcURL)
if err != nil {
errChan <- err
return
}
defer resp.Body.Close()
// create the file
outFile, err := os.Create(dstFilepath)
if err != nil {
errChan <- err
return
}
defer outFile.Close()
log.Println("starting copy")
// copy to file as the response arrives
_, err = io.Copy(outFile, resp.Body)
// return err
if err != nil {
log.Printf("\n Download Copy Error: %s \n", err.Error())
errChan <- err
return
}
doneChan <- struct{}{}
return
}(httpClient, srcURL, dstFilepath, errChan, doneChan)
// return Download
return (&Download{
updateFrequency: time.Microsecond * 500,
total: size,
errRecieve: errChan,
doneRecieve: doneChan,
filepath: dstFilepath,
}).Start(), nil
}
Update
Thanks to everyone who had input into this.
I've accepted JimB's answer as it seems like a perfectly viable approach that is more generalised than the solution I chose (and probably more useful to anyone who finds their way here).
In my case I already had a loop monitoring the file size so I threw a named error when this did not change for x seconds. It was much easier for me to pick up on the named error through my existing error handling and retry the download from there.
I probably crash at least one goroutine in the background with my approach (I may fix this later with some signalling) but as this is a short running application (its an installer) so this is acceptable (at least tolerable)
Doing the copy manually is not particularly difficult. If you're unsure how to properly implement it, it's only a couple dozen lines from the io package to copy and modify to suit your needs (I only removed the ErrShortWrite clause, because we can assume that the std library io.Writer implementations are correct)
Here is a copy work-alike function, that also takes a cancelation context and an idle timeout parameter. Every time there is a successful read, it signals to the cancelation goroutine to continue and start a new timer.
func idleTimeoutCopy(dst io.Writer, src io.Reader, timeout time.Duration,
ctx context.Context, cancel context.CancelFunc) (written int64, err error) {
read := make(chan int)
go func() {
for {
select {
case <-ctx.Done():
return
case <-time.After(timeout):
cancel()
case <-read:
}
}
}()
buf := make([]byte, 32*1024)
for {
nr, er := src.Read(buf)
if nr > 0 {
read <- nr
nw, ew := dst.Write(buf[0:nr])
written += int64(nw)
if ew != nil {
err = ew
break
}
}
if er != nil {
if er != io.EOF {
err = er
}
break
}
}
return written, err
}
While I used time.After for brevity, it's more efficient to reuse the Timer. This means taking care to use the correct reset pattern, as the return value of the Reset function is broken:
t := time.NewTimer(timeout)
for {
select {
case <-ctx.Done():
return
case <-t.C:
cancel()
case <-read:
if !t.Stop() {
<-t.C
}
t.Reset(timeout)
}
}
You could skip calling Stop altogether here, since in my opinion if the timer fires while calling Reset, it was close enough to cancel anyway, but it's often good to have the code be idiomatic in case this code is extended in the future.

Access HTTP response as string in Go

I'd like to parse the response of a web request, but I'm getting trouble accessing it as string.
func main() {
resp, err := http.Get("http://google.hu/")
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
ioutil.WriteFile("dump", body, 0600)
for i:= 0; i < len(body); i++ {
fmt.Println( body[i] ) // This logs uint8 and prints numbers
}
fmt.Println( reflect.TypeOf(body) )
fmt.Println("done")
}
How can I access the response as string? ioutil.WriteFile writes correctly the response to a file.
I've already checked the package reference but it's not really helpful.
bs := string(body) should be enough to give you a string.
From there, you can use it as a regular string.
A bit as in this thread
(updated after Go 1.16 -- Q1 2021 -- ioutil deprecation: ioutil.ReadAll() => io.ReadAll()):
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
See also GoByExample.
As commented below (and in zzn's answer), this is a conversion (see spec).
See "How expensive is []byte(string)?" (reverse problem, but the same conclusion apply) where zzzz mentioned:
Some conversions are the same as a cast, like uint(myIntvar), which just reinterprets the bits in place.
Sonia adds:
Making a string out of a byte slice, definitely involves allocating the string on the heap. The immutability property forces this.
Sometimes you can optimize by doing as much work as possible with []byte and then creating a string at the end. The bytes.Buffer type is often useful.
The method you're using to read the http body response returns a byte slice:
func ReadAll(r io.Reader) ([]byte, error)
official documentation
You can convert []byte to a string by using
body, err := ioutil.ReadAll(resp.Body)
bodyString := string(body)
Go 1.16+ update (February 2021)
Deprecation of io/ioutil
code should be
var client http.Client
resp, err := client.Get(url)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusOK {
bodyBytes, err := io.ReadAll(resp.Body)
// if u want to read the body many time
// u need to restore
// reader := io.NopCloser(bytes.NewReader(bodyBytes))
if err != nil {
log.Fatal(err)
}
bodyString := string(bodyBytes)
log.Info(bodyString)
}
reference
https://golang.org/doc/go1.16#ioutil
https://stackoverflow.com/a/52076748/2876087
string(byteslice) will convert byte slice to string, just know that it's not only simply type conversion, but also memory copy.

Resources