In Expo, the react-native + extra development environment,
the SQLite API in react-native uses the notion of 'transaction'. However there is no discussion of committing that transaction.
Is the RN notion of transaction just a set of consecutive calls SQLite? or is it a real transaction in the SQLite sense (i.e. all commands succeed or they are rolled back). If the latter, how does one correctly test that the transaction does or doesn't need to be retried?
If not, then I presume one has to do transaction and commit commands on one's own. Still how does one correctly handle a failed transaction?
Related
Consider a python webapp using SQLite and the DB Browser for SQLite app. Sometimes a database change (insert, update, delete) is made in DB Browser and not committed. This locks up the python web server if it saves to the same table.
I am wondering if there is an SQLite magic cookie that will allow me to commit any outstanding transaction. The concept doesn't leap to the imagination, but there is no "user". One simply attaches to the SQLite database. Technically, transactions are owned by anyone attached to that database.
My goal is to issue a "commit" from python to ensure the database status is nominal.
Any ideas?
This question already has answers here:
Couple Firebase Firestore and Firebase Storage calls together into a batch?
(2 answers)
Closed last month.
I'm developing an app in Flutter, I have a method called **CreateUser **which takes as parameters the user information, his profile picture, and a list of strings, I need to save the information in the Realtime Database, the picture in the Cloud Storage and the list in the Firestore.
I would like all these operations to be successful, if one of these should fail then I would like the others to undo the data they wrote. How can I implement the rollback of the other operations? Can I use transactions?
I've tried using transactions but I'm not sure if I can use them on different Databases.
I need to save the information in the Realtime Database, the picture in the Cloud Storage, and the list in the Firestore.
That's indeed possible, by performing one operation, right after another, only when the operation succeeds. For example, as soon as the operation for writing data to the Realtime Database completes, then inside the callback, perform the addition of the image to Storage. As soon as the addition of the image to Storage succeeds, perform the last operation of writing the data to Firestore.
I would like all these operations to be successful, if one of these should fail then I would like the others to undo the data they wrote.
There is no built-in mechanism for that. If you thought you can add to a batch operation, a Realtime Database write operation, a Firebase Storage file upload
and Firestore write operation and be sure that all three are complete, so you can have consistent data, please note that this is not possible. These operations are a part of different Firebase services and unfortunately, at the moment I'm writing this answer there is no way you can make them atomic, meaning all succeed or all fail with an exception.
How can I implement the rollback of the other operations?
You have to write code for that because none of the Firebase products support cross-product transactional operations. To solve this, you'll have to nest the calls during your write/upload operations and handle the error if the next operation fails. This means that you either have to delete the data from the Realtime Database and the file from Storage if the write operation in Firestore fails. Or only delete the data from the Realtime Database if the file upload to Storage fails.
But note, at some point in time, there will be a failure that the client can't roll back one of the delete operations. The most common approach for these inevitable failures which might happen is to make your code robust by handling exceptions and performing occasional cleanups in both places, Firebase Storage and Firestore, considering that the first operation is the one that writes data to the Realtime Database.
As discussed with the Firebase engineers, the reason is quite clear. Even if the Realtime Database and Cloud Firestore are both Firebase products, they are still different products. Besides that, Firebase Storage is a service within Google Cloud. So now, 2023-01-12 there is no way you we can do that. Hopefully, it will be available in the near future.
Can I use transactions?
No, and that's for the exact same reason as above.
One way I might address this, is to use the Firestore document write operation to trigger a Workflow [1] that can handle the three operations and rollback depending on failure state. That way you can also have a constant transaction record follow the process.
If you wanted to provide app feedback say to the user, you could have your app. wait for a DB record of completion (or error) get written and based on that report back to the user.
[1] https://cloud.google.com/firestore/docs/solutions/workflows
According to firestore documentation:
a transaction is a set of read and write operations on one or more documents.
Also:
Transactions will fail when the client is offline.
Now the limitation in firestore is that:
In Cloud Firestore, you can only update a single document about once per second, which might be too low for some high-traffic applications.
So using cloud functions and running transactions to increment/decrement counters when the traffic is high will fail.
So they have discussed to use the approach of distributed counters.
According to the algorithm of distrbuted counter:
create shards
choose a shard randomly
run a transaction to increment/decrement the counter
get all the shards and aggregate the
result to show the value of a counter
Scenerio:
consider you have a counter which is to be updated when a document is added and that counter is being displayed in the UI. Now for good UX, I cannot block the UI when network is offline. So I must allow creation/updation of documents even when client is offline and sync these changes once client is online so that everyone else listening to these changes receive the correct value of the counter.
Now transactions fail when the client is offline.
So my question for best user experience (even when offline) is:
Do you really require a transaction to increment a counter? I know
transactions ensure that writes are atomic and are either
successful/unsuccessful and prevent partial writes. But what's the
point when they fail offline? I was thinking maybe write them to local cache and sync it once the network is back online.
Should this be done via client sdks of via cloud functions?
Do you really require a transaction to increment a counter?
Definitely yes! Because we are creating apps that can be used in a multi user environment, transactions are mandatory, so we can provide consistent data.
But what's the point when they fail offline?
When there is a loss of network connectivity (there is no network connection on user device), transactions are not supported for offline use. This is because a transaction absolutely requires round trip communications with server in order to ensure that the code inside the transaction completes successfully. So, transactions can only execute when you are online.
Should this be done via client sdks of via cloud functions?
Please note, that the Firestore SDK for Android has a local cache that's enabled by default. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
So all read operations will come from cache if there are no updates on the server. So Firestore provides this feature for handle offline data.
You can also write a function in Cloud Function that will increment the counter while a new document is added or to decrement the conter while a document is deleted.
I also recommend you to take a look:
How to count the number of documents under a collection in Firestore?
So you may also consider using Firebase realtime database for that. Cloud Firestore and Firebase realtime database work very well together.
Edit:
It allows one to upvote the answer even when the device is offline. After the network is online, it syncs to the server and the counter is updated. Is there a way i can do this in firestore when the device is offline.
This is also happening by default. So if the user tries to add/delete documents while offline, every operation is added to a queue. Once the user regains the connection, every change that is made while offline, will be updated on Firebase servers. With other words, all queries will be commited on the server.
Cloud fnctions are triggered only when the change is received and that can only happen when the device is online.
Yes, that correct. Once the device regains the network connection, the document is added/deleted from the database, moment in which the function fires and increases/decreases the counter.
Edit2:
Suppose I have made around 100 operations offline, will that not put a load on the cloud functions when the device comes online? What's your thought on this?
When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations. So I suggest use this database for its online capabilities.
Regarding Cloud Functions for those 100 offline operations, there will be no issues. Firebase servers work very well with concurent operations.
I am new to firebase and firestore and would like to know if the behavior I found is a bug or by design.
I am using angularfire2 in my ionic project where the user will be offline most of the time… so offline support is a big deal.
The problem: snapshotChanges is not called in a offline batch delete of subcolletion.
I have something like this in firestore /users/{userId}/projects/{projectId}/points/{pointId}
When a user inserts a new project or point a use set and the object is written fine, my lists are updated with the new instance and it works great thanks to snapshotChanges. When the user removes a project is the problem..
I execute a batch delete in the points of a project and after that I delete the project itself. This works fine online, but not offline. My lists are not being updated even though the operation completes with success. I could reproduce it multiple times, but only if the app is offline all the time ( the inserts and deletes are only local)
am I missing something?
The documentation states here:
Batched writes have fewer failure cases than transactions and use
simpler code. They are not affected by contention issues, because they
don't depend on consistently reading any documents. Batched writes
execute even when the user's device is offline.
This "Batched writes execute even when the user's device is offline" this makes me understand that the events of removal should be propagated to snapshotChanges.
It seems odd to me that Firestore would charge me for read queries to locally cached data, but I can't find any clarification to the contrary in the Firestore Pricing document. If I force Firebase into offline mode and then perform reads on my locally cached data, am I still charged for each individual entity that I retrieve?
Second, offline users in my app write many small updates to a single entity. I want the changes to persist locally each time (in case they quit the app), but I only need eventually consistent saves to the cloud. When a user reconnects to the internet and Firestore flushes the local changes, will I be charged a single write request for the entity or one per update call that I made while offline?
Firestore could potentially fit my use case very well, but if offline reads and writes are charged at the same rate as online ones it would not be an affordable option.
As the offical documentation says,
Cloud Firestore supports offline data persistence. This feature caches a copy of the Cloud Firestore data that your app is actively using, so your app can access the data when the device is offline. You can write, read, listen to, and query the cached data. When the device comes back online, Cloud Firestore synchronizes any local changes made by your app to the data stored remotely in Cloud Firestore.
So, every client that is using a Firestore database and sets PersistenceEnabled to true, maintains it's own internal (local) version of the database. When data is inserted/updated, it is first written to this local version of the database. As a result, all writes to the database are added to a queue. This means that all the operations that where stored there will be commited on Firebase servers once you are back online. This also means that those operations will be seen as independent operations and not as a whole.
But remeber, don't use Firestore as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
If Google Cloud Firestore priceing model does not fit your use case very well then use Firebase Realtime Database. As mentioned also in this post from the Firebase offical blog, one the reasons you still might want to use the Realtime Database is:
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
So it's up to you which option you choose.
According to this If you want to work completely offline with Cloud Firestore you can disable network by :
FirebaseFirestore.getInstance().disableNetwork()
but firestore will cause client offline error for first user get request, that you must consider this error as empty response.