Do I need extra round trip to firestore for reading Created & Updated timestamps fields? - firebase

Ok so I have a REST API in GO which stores a ticket resource using firestore. For this I use: firestore go client
I want to be able to order my documents by date created / date updated, so by following the docs I store these 2 fields as timestamps in the document.
I use the tag serverTimestamp on these 2 fields. By doing this, the value should be the time at which the firestore server processed the request.
The HTTP response of the update operation should have this body:
{
"ticket": {
"id": "af41766e-76ea-43b5-86c1-8ba382edd4dc",
"title": "Ticket updated title",
"price": 9,
"date_created": "2023-01-06 09:07:24",
"date_updated": "2023-01-06 10:08:24"
}
}
So it means after I update the ticket document, besides an updated title or price I also need to have the updated value fo the date_updated field.
This is working for the moment but I'm curious if the way I coded this is the way to do it. As you can see in the code samples, I use a transaction to update a ticket. I didn't find a way to retrieve the updated value for the DateUpdated field, other than reading again the updated ticket.
The domain entity is defined as this:
package tixer
import (
"context"
"time"
"github.com/google/uuid"
)
type (
// TicketID represents a unique identifier for a ticket.
// It's a domain type.
TicketID uuid.UUID
// Ticket represents an individual ticket in the system.
// It's a domain type.
Ticket struct {
ID TicketID
Title string
Price float64
DateCreated time.Time
DateUpdated time.Time
}
)
I'll attach here the communication with firestore from the create and update perspective:
// Storer persists tickets in Firestore.
type Storer struct {
client *firestore.Client
}
func NewStorer(client *firestore.Client) *Storer {
return &Storer{client}
}
func (s *Storer) CreateTicket(ctx context.Context, ticket *tixer.Ticket) error {
writeRes, err := s.client.Collection("tickets").Doc(ticket.ID.String()).Set(ctx, createTicket{
Title: ticket.Title,
Price: ticket.Price,
})
// In this case writeRes.UpdateTime is the time the document was created.
ticket.DateCreated = writeRes.UpdateTime
return err
}
func (s *Storer) UpdateTicket(ctx context.Context, ticket *tixer.Ticket) error {
docRef := s.client.Collection("tickets").Doc(ticket.ID.String())
err := s.client.RunTransaction(ctx, func(ctx context.Context, tx *firestore.Transaction) error {
doc, err := tx.Get(docRef)
if err != nil {
switch {
case status.Code(err) == codes.NotFound:
return tixer.ErrTicketNotFound
default:
return err
}
}
var t persistedTicket
if err := doc.DataTo(&t); err != nil {
return err
}
t.ID = doc.Ref.ID
if ticket.Title != "" {
t.Title = ticket.Title
}
if ticket.Price != 0 {
t.Price = ticket.Price
}
return tx.Set(docRef, updateTicket{
Title: t.Title,
Price: t.Price,
DateCreated: t.DateCreated,
})
})
if err != nil {
return err
}
updatedTicket, err := s.readTicket(ctx, ticket.ID)
if err != nil {
return err
}
*ticket = updatedTicket
return nil
}
func (s *Storer) readTicket(ctx context.Context, id tixer.TicketID) (tixer.Ticket, error) {
doc, err := s.client.Collection("tickets").Doc(id.String()).Get(ctx)
if err != nil {
switch {
case status.Code(err) == codes.NotFound:
return tixer.Ticket{}, tixer.ErrTicketNotFound
default:
return tixer.Ticket{}, err
}
}
var t persistedTicket
if err := doc.DataTo(&t); err != nil {
return tixer.Ticket{}, err
}
t.ID = doc.Ref.ID
return toDomainTicket(t), nil
}
type (
// persistedTicket represents a stored ticket in Firestore.
persistedTicket struct {
ID string `firestore:"id"`
Title string `firestore:"title"`
Price float64 `firestore:"price"`
DateCreated time.Time `firestore:"dateCreated"`
DateUpdated time.Time `firestore:"dateUpdate"`
}
// createTicket contains the data needed to create a Ticket in Firestore.
createTicket struct {
Title string `firestore:"title"`
Price float64 `firestore:"price"`
DateCreated time.Time `firestore:"dateCreated,serverTimestamp"`
DateUpdated time.Time `firestore:"dateUpdate,serverTimestamp"`
}
// updateTicket contains the data needed to update a Ticket in Firestore.
updateTicket struct {
Title string `firestore:"title"`
Price float64 `firestore:"price"`
DateCreated time.Time `firestore:"dateCreated"`
DateUpdated time.Time `firestore:"dateUpdate,serverTimestamp"`
}
)
func toDomainTicket(t persistedTicket) tixer.Ticket {
return tixer.Ticket{
ID: tixer.TicketID(uuid.MustParse(t.ID)),
Title: t.Title,
Price: t.Price,
DateCreated: t.DateCreated,
DateUpdated: t.DateUpdated,
}
}

If I understand correctly, the DateUpdated field is a server-side timestamp, which means that its value is determined by the server (as a so-called field transformation) when the value is written to the storage layer. Since a write operation in the Firestore SDK doesn't return the resulting data of that operation, the only way to get that value back into your application is indeed to perform an extra read operation after the write to get it.
The SDK doesn't automatically perform this read is because it is a charged operation, which in many cases is not needed. So by leaving it up to your code to perform that read, you can decide whether to incur this cost or not.

Related

Firestore Golang run query with filters in transaction

I can't figure out how to run a firestore.Query in a transaction in the Golang Admin SDK.
The firestore.Transaction has a GetAll() method that takes an array of *firestore.DocumentRef, which I assume is how one would query multiple documents in a transaction. This works if I want to query an entire collection, since I can use tx.DocumentRefs to convert a *firestore.CollectionRef into document refs but there doesn't seem to be an equivalent method for queries (for example if I want to filter the collection).
In the NodeJS Admin SDK I could do something like:
admin.firestore().runTransaction(async (t) => {
const fooCollectionRef = admin.firestore().collection('foos').where('bar', '==', true);
const foosSnapshot = await t.get(fooCollectionRef);
// do stuff with the foos
})
How can I accomplish the same in Golang?
Use the Transaction.Documents method. It accepts a Queryier which can be a firestore.Query. See this method in the docs https://pkg.go.dev/cloud.google.com/go/firestore#Transaction.Documents
Following is a simple example:
err := client.RunTransaction(ctx, func(ctx context.Context, tx *firestore.Transaction) error {
col := client.Collection("myCollection")
query := col.Where("myCondition", "==", 1)
docs := tx.Documents(query)
for {
d, err := docs.Next()
if err != nil {
if err == iterator.Done {
break
}
//handle error
}
//do something with the document
}
return nil
})
if err != nil {
// handle error.
}

Swift code to Add item with quantity in Firebase Database

Using Swift code 5.1 I have managed to update Firestore Database with items in current users basket but not able to add/update quantity. Currently if I wanted to add an item that already exist in the basket it simply adds another line but I wanted to just update quantity.
Can you advise me on how to create a function that adds quantity?
Here are the codes I have so far. Only relevant sections of code pasted.
Firestore DB function in my Helper file:
enum FCollectionReference: String {
case User
case Category
case Items
case Basket
case Orders
}
func FirebaseReference(_ collectionReference: FCollectionReference) -> CollectionReference {
return Firestore.firestore().collection(collectionReference.rawValue)
}
Here's the code in in my Basket Model file using
class Basket {
var id: String!
var ownerId: String!
var itemIds: [String]!
var delivery: Float!
var admin: Float!
var quantity: Int!
init() {
}
init(_dictionary: NSDictionary) {
id = _dictionary[kOBJECTID] as? String
ownerId = _dictionary[kOWNERID] as? String
itemIds = _dictionary[kITEMIDS] as? [String]
delivery = _dictionary[kDELIVERY] as? Float
admin = _dictionary[kADMIN] as? Float
quantity = _dictionary[kQUANTITY] as? Int
}
}
//MARK: Helper functions
func basketDictionaryFrom(_ basket: Basket) -> NSDictionary {
return NSDictionary(objects: [basket.id, basket.ownerId, basket.itemIds, basket.quantity], forKeys: [kOBJECTID as NSCopying, kOWNERID as NSCopying, kITEMIDS as NSCopying, kQUANTITY as NSCopying,kDELIVERY as NSCopying, kADMIN as NSCopying])
}
//MARK: - Update basket
func updateBasketInFirestore(_ basket: Basket, withValues: [String : Any], completion: #escaping (_ error: Error?) -> Void) {
FirebaseReference(.Basket).document(basket.id).updateData(withValues) { (error) in
completion(error)
Codes in Item View Control to add items to basket:
#objc func addToBasketButtonPressed() {
//check if user is logged in or show login view
if MUser.currentUser() != nil {
downloadBasketFromFirestore(MUser.currentId()) { (basket) in
if basket == nil {
self.createNewBasket()
}else {
basket?.itemIds.append(self.item.id)
self.updateBasket(basket: basket!, withValues: [kITEMIDS: basket!.itemIds])
}
}
} else {
showLoginView()
}
}
private func updateBasket(basket: Basket, withValues: [String : Any]) {
updateBasketInFirestore(basket, withValues: withValues) { (error) in
if error != nil {
self.hud.textLabel.text = "Error: \(error!.localizedDescription)"
self.hud.indicatorView = JGProgressHUDErrorIndicatorView()
self.hud.show(in: self.view)
self.hud.dismiss(afterDelay: 2.0)
print("error updating basket", error!.localizedDescription)
}else {
self.hud.textLabel.text = "Added to Basket"
self.hud.indicatorView = JGProgressHUDSuccessIndicatorView()
self.hud.show(in: self.view)
self.hud.dismiss(afterDelay: 2.0)
}
}
}
To clarify my request, what do I need to change/re-arrange in my coding so the Database Cloud Firestore is arranged in order shown in my attached screen shot. First screen shot showing current layout in the last column and I'm trying to change this to layout demonstrated in the second screen shot?
I think you are asking how to update the value in a field within a Firestore document. If not, let me know and I will update the answer.
Here's some code that updates the qty of an item in inventory. Pass in the qty to add as a + Int and then to subtract as a - Int. The structure looks like this
root
inventory
item_0
qty: 0
and the code to update the qty node is:
func incrementQty(deltaQty: Int) {
let docToUpdate = self.db.collection("inventory").document("item_0")
docToUpdate.updateData( [
"qty": FieldValue.increment( Int64(deltaQty) )
])
}
call it like this
self.incrementQty(deltaQty: 4) //adds 4 to the existing qty
previously, incrementing values had to be wrapped into a transaction to make it safe but the FieldValue makes it much easier.
I am adding another answer based on comments and question clarification. My other answer still stands as an answer but it's a different approach.
Arrays are inherently hard to work with in NoSQL databases as they are often treated as a single object. They have limited functionality opposed to collections, documents and fields, and can't directly be sorted or have items inserted. And querying is well, challenging. Firestore does a great job at providing better interoperability with arrays but there are still usually better options.
Instead of an array, I would change the structure to this:
Baskets (collection)
basket_number (document in the Baskets collection, like you have now)
items //a collection of items in the basket
item_0 //a document with the docID being the the item number
item_qty: //qty of the item
item_1
item_qty:
item_2
item_qty:
So the downside of .updateData is that if the field being updated doesn't exist, it doesn't create the field, it simply throws an error. So we need to test to see if the document exists first, if so, update with updateData, if not create the item with an initial quantity.
Here's the code that does it - note for simplicity I am ignoring the top level Basket and basket_number since you already know how to do that part and focused on the items collection and down.
func incrementQty(itemNumberToUpdate: String, deltaQty: Int) {
let docToUpdate = self.db.collection("items").document(itemNumberToUpdate)
docToUpdate.getDocument(completion: { documentSnapshot, error in
if let err = error {
print(err.localizedDescription)
return
}
if let _ = documentSnapshot?.data() {
print("item exists, update qty")
docToUpdate.updateData([
"item_qty": FieldValue.increment( Int64(deltaQty) )
], completion: { err in
if let err = err {
print("Error updating document: \(err.localizedDescription)")
} else {
print("Item qty successfully updated")
}
})
} else {
print("no item exists, need to create")
docToUpdate.setData([
"item_qty": FieldValue.increment( Int64(deltaQty) )
], completion: { err in
if let err = err {
print("Error updating document: \(err.localizedDescription)")
} else {
print("Item successfully created with initial quantity")
}
})
}
})
}
Pass in an item number and the quantity to either modify the existing qty by, or will be the initial quantity.
self.incrementQty(itemNumberToUpdate: "item_0", deltaQty: 5)

How to listen to firestore through RPC

I want to listen to real time changes in firestore and I am also only allowed to use Go. Since firestore SDK for Go doesn't have any option to listen for real time changes, I decided to use the firestore v1beta1 sdk.
I have written the following code to do that
func TestRPCHandler(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
c, err := firestore.NewClient(context.Background())
databaseName := "projects/[project_name]/databases/(default)"
if err != nil {
panic(err)
}
stream, err := client.Listen(context.Background())
if err != nil {
panic(err)
}
request := &firestorepb.ListenRequest{
Database: databaseName,
TargetChange: &firestorepb.ListenRequest_AddTarget{
AddTarget: &firestorepb.Target{
TargetType: &firestorepb.Target_Documents{
Documents: &firestorepb.Target_DocumentsTarget{
Documents: []string{"projects/[project_name]/databases/(default)/[collection_name]"} ,
},
},
},
},
}
if err := stream.Send(request); err != nil {
panic(err)
}
if err := stream.CloseSend(); err != nil {
panic(err)
}
for {
resp, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
panic(err)
}
}
}
When I am doing this, the code does not detect any changes I bring about manually in the database. stream.Recv() just returns EOF and exits immediately. I even tried manually waiting by adding time.Sleep() but that does not help either.
You don't need the beta SDK or hacks to make this happen, I found the solution, it's pretty easy actually.
The https://firebase.google.com/docs/firestore/query-data/listen documentation does not contain an example for Go.
The source code of the firestore client API for Go has an unexported watchStream which we cannot directly use: https://github.com/googleapis/google-cloud-go/blob/master/firestore/watch.go#L130
Deep search of the repository shows that this is actually used on the DocumentSnapshotIterator and QuerySnapshotIterator at: https://github.com/googleapis/google-cloud-go/blob/master/firestore/docref.go#L644 and: https://github.com/googleapis/google-cloud-go/blob/master/firestore/query.go#L716.
The Collection contains a Snapshots method which returns the snapshot iterator that we want, after that all is easy, we just make an infivitive loop through its Next method.
Example:
cols, err := client.Collections(context.Background()).GetAll()
for _, col := range cols {
iter := col.Snapshots(context.Background())
defer iter.Stop()
for {
doc, err := iter.Next()
if err != nil {
if err == iterator.Done {
break
}
return err
}
for _, change := range doc.Changes {
// access the change.Doc returns the Document,
// which contains Data() and DataTo(&p) methods.
switch change.Kind {
case firestore.DocumentAdded:
// on added it returns the existing ones.
isNew := change.Doc.CreateTime.After(l.startTime)
// [...]
case firestore.DocumentModified:
// [...]
case firestore.DocumentRemoved:
// [...]
}
}
}
}
Yours, Gerasimos Maropoulos aka #kataras
Firebase's Get realtime updates with Cloud Firestore documentation currently indicates that Go is not yet supported.
// Not yet supported in Go client library

Query Firebase Firestore documents by the ID

As I got from 'Cloud Firestore Data Model' guide "each document is identified by a name." Is it possible to query a collection by that document identifier (which is the name or ID)?
For example, documents in the collection "Things" have IDs: 0, 1, 2 etc.:
Is it possible to query documents which IDs are less than 100?
You can query by documentId using the special sentinel FieldPath.documentId(), e.g.:
const querySnap = collection.where(firebase.firestore.FieldPath.documentId(), '<', '100').get();
But be aware that document IDs are strings and therefore this will include documents with ID '0' or '1', but not '2' since '2' > '100' lexicographically.
So if you want a numeric query, you'll need to write the document ID as a numeric field in the document and then do a normal query on it.
In python, you should use full documents names
from google.cloud import firestore
from google.cloud.firestore_v1.field_path import FieldPath
db = firestore.Client()
colRef = db.collection(u'docs')
filter = [db.document(u'docs/doc1'), db.collection(u'docs/doc3')]
query = colRef.where(FieldPath.document_id(), u'in', filter)
I was struggling to find this for the Golang Firebase SDK but finally got it. Hope this helps somebody out there!
package main
import (
"context"
"fmt"
"log"
"cloud.google.com/go/firestore"
firebase "firebase.google.com/go/v4"
"google.golang.org/api/option"
)
type (
Car struct {
ID string
Name string `firestore:"name"`
Make string `firestore:"make"`
Price float64 `firestore:"make"`
}
)
func main() {
ctx := context.Background()
// Use a service account
options := option.WithCredentialsFile("PATH/TO/SERVICE/FILE.json")
// Set project id
conf := &firebase.Config{ProjectID: "PROJECT_NAME"}
// Initialize app
app, err := firebase.NewApp(ctx, conf, options)
if err != nil {
log.Fatal(err)
}
// Get firestore client
client, err := app.Firestore(ctx)
if err != nil {
log.Fatal(err)
}
defer client.Close()
collectionRef := client.Collection("CAR_COLLECTION")
// Create docment list of documents from "CAR_COLLECTION"
var skipDocs []*firestore.DocumentRef
idList := []string{"001", "002", "003"}
for _, id := range idList {
skipDocs = append(skipDocs, collectionRef.Doc(id))
}
// firestore.DocumentID == "__name__"
docs, err := collectionRef.Where(firestore.DocumentID, "not-in", skipDocs).Documents(ctx).GetAll()
if err != nil {
log.Fatal(err)
}
var carList []Car
for _, doc := range docs {
var car Car
// Unmarshall item
doc.DataTo(&car)
car.ID = doc.Ref.ID
// Add car to list
carList = append(carList, car)
}
// Print car list
fmt.Println(carList)
}

Golang fatal error: concurrent map read and map write

I'm writing minecraft server in Go, when server is being stressed by 2000+ connections I get this crash:
fatal error: concurrent map read and map write/root/work/src/github.com/user/imoobler/limbo.go:78 +0x351
created by main.main /root/work/src/github.com/user/imoobler/limbo.go:33 +0x368
My code:
package main
import (
"log"
"net"
"bufio"
"time"
"math/rand"
"fmt"
)
var (
connCounter = 0
)
func main() {
InitConfig()
InitPackets()
port := int(config["port"].(float64))
ln, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
if err != nil {
log.Fatal(err)
}
log.Println("Server launched on port", port)
go KeepAlive()
for {
conn, err := ln.Accept()
if err != nil {
log.Print(err)
} else {
connCounter+=1
go HandleConnection(conn, connCounter)
}
}
}
func KeepAlive() {
r := rand.New(rand.NewSource(15768735131534))
keepalive := &PacketPlayKeepAlive{
id: 0,
}
for {
for _, player := range players {
if player.state == PLAY {
id := int(r.Uint32())
keepalive.id = id
player.keepalive = id
player.WritePacket(keepalive)
}
}
time.Sleep(20000000000)
}
}
func HandleConnection(conn net.Conn, id int) {
log.Printf("%s connected.", conn.RemoteAddr().String())
player := &Player {
id: id,
conn: conn,
state: HANDSHAKING,
protocol: V1_10,
io: &ConnReadWrite{
rdr: bufio.NewReader(conn),
wtr: bufio.NewWriter(conn),
},
inaddr: InAddr{
"",
0,
},
name: "",
uuid: "d979912c-bb24-4f23-a6ac-c32985a1e5d3",
keepalive: 0,
}
for {
packet, err := player.ReadPacket()
if err != nil {
break
}
CallEvent("packetReceived", packet)
}
player.unregister()
conn.Close()
log.Printf("%s disconnected.", conn.RemoteAddr().String())
}
For now server is only "limbo".
Generally speaking (without having access to the code where the error occurs) you have a few options. Here are two of them:
sync.RWMutex
Control access to the map with sync.RWMutex{}. Use this option if you have single reads and writes, not loops over the map. See RWMutex
Here a sample with access control to someMap via someMapMutex:
var (
someMap = map[string]string{}
someMapMutex = sync.RWMutex{}
)
go func() {
someMapMutex.Lock()
someMap["key"] = "value"
someMapMutex.Unlock()
}()
someMapMutex.RLock()
v, ok := someMap["key"]
someMapMutex.RUnlock()
if !ok {
fmt.Println("key missing")
return
}
fmt.Println(v)
syncmap.Map
Use a syncmap.Map{} instead of a normal map. This map is already taking care of race issues but may be slower depending on your usage. syncmap.Map{}s main advantage lies with for loops. See syncmap
var (
someMap = syncmap.Map{}
)
go func() {
someMap.Store("key", "value")
}()
v, ok := someMap.Load("key")
if !ok {
fmt.Println("key missing")
return
}
fmt.Println(v)
// with syncmap, looping over all keys is simple without locking the whole map for the entire loop
someMap.Range(func(key, value interface{}) bool {
// cast value to correct format
val, ok := value.(string)
if !ok {
// this will break iteration
return false
}
// do something with key/value
fmt.Println(key, val)
// this will continue iterating
return true
})
General Advice
You should test your server with -race option and then eliminate all the race conditions it throws. That way you can easier eliminate such errors before they occur.
go run -race server.go
See golang race detector

Resources