I'm working on improving code in an application that uses Firestore. I would like to ensure that Firestore will cleanup some bits of a transaction after the transaction is completed. Instead of doing this, I may opt to run the transaction and cleanup in one transaction. However, I believe this may also introduce a race condition if other architectural changes are not made.
Anyways, in working on this problem, I realized I don't know of a way (besides making a task queue) to dispatch a one-off task in Firestore.
I think the toy example below will help explain what I want to do.
// push cleanup task to Firestore to execute in 3000ms
await dispatchTransactionCleanup({id : "xxx", inMs : 3000}); // !important
// run the actual transaction, which should not take more than 1000 ms
await runMyTransaction({id : "xxx"});
// also dispatch a cleanup here, to reduce latency over the other cleanup.
// but, importantly this cleanup isn't guaranteed to run, e.g.,
// the server crashes between request to run transaction
// and this part of the program.
await runManualCleanup({id : "xxx"});
So, how would I dispatchTransactionCleanup to run a one-off task at some point in the future? Is this possible without my own task queue? Or, do I need to introduce additional structure?
Firestore doesn't have any sense of "task" or "queue". It just responds to the API calls you make, at the time you make them.
If you want a queuing mechanism, you'll have to build that, perhaps with some other software products. You already linked to one solution that uses Cloud Functions on the backend.
You might want to reconsider simply doing all the work in a single transaction. It's not clear why you think that this would be problematic. Maybe you could post a second question to explain in more detail what you tried that wouldn't work the way you expect.
Related
My application needs front-end searching. It searches an external API, for which I'm limited to a few calls per second.
So, I wanted to keep ALL queries, related to this external API, on the same Cloud Task queue, so I could guarantee the amount of calls per second.
That means the user would have to wait for second or two, most likely, when searching.
However, using Google's const { CloudTasksClient } = require('#google-cloud/tasks') library, I can create a task but when I go to check it's status using .getTask() it says:
The task no longer exists, though a task with this name existed recently.
Is there any way to poll a task until it's complete and retrieve response data? Or any other recommended methods for this? Thanks in advance.
No. GCP Cloud Tasks provides no way to gather information on the body of requests that successfully completed.
(Which is a shame, because it seems quite natural. I just wrote an email to my own GCP account rep asking about this possible feature. If I get an update I'll put it here.)
Transactions are used for atomic changes and when two clients may change the same data at the same time.
I want to test in the dev env if my transaction is having its expected behavior when there is a parallel transaction running from multiple clients requests. It runs only in my Cloud Functions. I can't let any undesired behavior of this nature to happen in the prod env, so I want to check in dev if everything is alright when it happens, even being unlikely.
Is it possible to force this test case?
Using JS/TS.
In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data. Refer to this documentation regarding updating data with transactions.
You can check this post that discusses concurrent read and write at the same time. Another link that also has an example on how to create and run a transaction using Node.js.
Lastly, you can consider creating two FirebaseApp instances, and running the same transaction in both and then synchronize between the two in the single process that they run in. Or, use testing tools that support parallel tests, like Node TAP.
So, I'm making multiplayer mobile game using Xamarin and Firebase. In game there are many moment when I'm letting players decide what to do and send their decision to the server (by putting decision enum in player-specific Firebase database node). Decision is time limited (short time, no longer than 20s).
I set listener to that specific node in my Firebase functions to check if all player decided or player decision comes after time deadline, but I need to deal with case when: some players send their decision in time - sersnmart were will not execute next action, and that one player just will not send his decision (eave game or something) - server won't be poke again to check deadline and invoke functions.
That why I'm looking for something else, I found method for schedule functions using crontab, but the minimal time interval there seems to be minutes, which is way more to long for me.
Second idea includes wait that specific time interval in previous Firebase thread, but it seems too bad way to deal with this.
Which way is best for dynamic invoking short-interval scheduled Firebase functions?
The best way to schedule Cloud Functions to run at a specific time it through the Google Cloud Tasks schedules. See Doug's blog post for a full description of this: How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL)
That said, I regularly use setTimeOut in my own Cloud Functions too when I need to delay an operation for a short period of time. Just keep in mind that you pay for the seconds that the function is sleeping, so cost-wise you'll want to trade that time off against what another invocation would cost.
So for now I decided to use setTimeout, free firebase plan seems to limit only functions invoke number, not working plan so this shouldn't be a problem. Depsite this, I'm still waiting for advice from you
My current application developed in Unity uses Firebase Realtime Database with database persistence enabled. This works great for offline use of the application (such as in areas of no signal).
However, if a new user runs the application for the first time without an internet connection - the application freezes. I am guessing, it's because it needs to pull down the database for the first time in order for persistence to work.
I am aware of threads such as: 'Detect if Firebase connection is lost/regained' that talk about handling database disconnection from Firebase.
However, is there anyway I can check to see if it is the users first time using the application (eg via presence of the persistent database?). I can then inform them they must go online for 'first time setup'?
In addition to #frank-van-puffelen's answer, I do not believe that Firebase RTDB should itself cause your game to lock up until a network connection occurs. If your game is playable on first launch without a network connect (ie: your logic itself doesn't require some initial state from the network), you may want to check the following issues:
Make sure you can handle null. If your game logic is in a Coroutine, Unity may decide to silently stop it rather than fully failing out.
If you're interacting with the database via Transactions, generally assume that it will run twice (once against your local cache then again when the cache is synced with the server if the value is different). This means that the first time you perform a change via a transaction, you'll likely have a null previous state.
If you can, prefer to listen to ValueChanged over GetValueAsync. You'll always get this callback on your main Unity thread, you'll always get the callback once on registration with the data in your local cache, and the data will be periodically updated as the server updates. Further, if you see #frank-van-puffelen answer elsewhere, if you're using GetValueAsync you may not get the data you expect (including a null if the user is offline). If your game is frozen because it's waiting on a ContinueWithOnMainThread (always prefer this to ContinueWith in Unity unless you have a reason not to) or an await statement, this could ValueChanged may work around this as well (I don't think this should be the case).
Double check your object lifetimes. There are a ton of reasons that an application may freeze, but when dealing with asynchronous logic definitely make sure you're aware of the differences between Unity's GameObject lifecycle and C#'s typical object lifecycle (see this post and my own on interacting with asynchronous logic with Unity and Firebase). If an objects OnDestroy is invoked before await, ContinueWith[OnMainThread], or ValueChanged is invoked, you're in danger of running into null references in your own code. This can happen if a scene changes, the frame after Destroy is called, or immediately following a DestroyImmediate.
Finally, many Firebase functions have an Async and synchronous variant (ex: CheckDependencies and CheckDependenciesAsync). I don't think there are any to call out for Realtime Database proper, but if you use the non async variant of a function (or if you spinlock on the task completing, including forgetting to yield in a coroutine), the game will definitely freeze for a bit. Remember that any cloud product is i/o bound by nature, and will typically run slower than your game's update loop (although Firebase does its best to be as fast as possible).
I hope this helps!
--Patrick
There is nothing in the Firebase Database API to detect whether its offline cache was populated.
But you can detect when you make a connection to the database, for example by listening to the .info/connected node. And then when that first is set to true, you can set a local flag in the local storage, for example in PlayerPrefs.
With this code in place, you can then detect if the flag is set in the PlayerPrefs, and if not, show a message to the user that they need to have a network connection for you to download the initial data.
Let's say I have a Cloud Firebase Function - called by a cron job - that produces 30+ tasks every time it's invoked.
These tasks are quite slow (5 - 6 second each in average) and I can't process them directly in the original because it would time out.
So, the solution would be invoking another "worker" function, once per task, to complete the tasks independently and write the results in a database. So far I can think of three strategies:
Pubsub messages. That would be amazing, but it seems that you can only listen on pubsub messages from within a Cloud Function, not create one. Resorting to external solutions, like having a GAE instance, is not an option for me.
Call the worker http-triggered Firebase Cloud Function from the first one. That won't work, I think, because I would need to wait for a response from the all the invoked worker functions, after they finish and send, and my original Function would time out.
Append tasks to a real time database list, then have a worker function triggered by each database change. The worker has to delete the task from the queue afterwards. That would probably work, but it feels there are a lot of moving parts for a simple problem. For example, what if the worker throws? Another cron to "clean" the db would be needed etc.
Another solution that comes to mind is firebase-queue, but its README explicitly states:
"There may continue to be specific use-cases for firebase-queue,
however if you're looking for a general purpose, scalable queueing
system for Firebase then it is likely that building on top of Google
Cloud Functions for Firebase is the ideal route"
It's not officially supported and they're practically saying that we should use Functions instead (which is what I'm trying to do). I'm a bit nervous on using in prod a library that might be abandoned tomorrow (if it's not already) and would like to avoid going down that route.
Sending Pub/Sub messages from Cloud Functions
Cloud Functions are run in a fairly standard Node.js environment. Given the breadth of the Node/NPM ecosystem, the amount of things you can do in Cloud Functions is quite broad.
it seems that you can only listen on pubsub messages from within a Cloud Function, not create one
You can publish new messages to Pub/Sub topics from within Cloud Functions using the regular Node.js module for Pub/Sub. See the Cloud Pub/Sub documentation for an example.
Triggering new actions from Cloud Functions through Database writes
This is also a fairly common pattern. I usually have my subprocesses/workers clean up after themselves at the same moment they write their result back to the database. This works fine in my simple scenarios, but your mileage may of course vary.
If you're having a concrete cleanup problem, post the code that reproduces the problem and we can have a look at ways to make it more robust.