I discovered a very useful function named get(Source source). If I pass CACHE, I can get data only from cache. But how about set(Source source)? I cannot find something similar. I need to save data locally and push it to the server only when needed. How to solve this? Or any other alternatives? Thanks
What you're trying to do is not supported by the Firestore client libraries. Then only kind of writes you can perform will always be synchronized to the server at the earliest opportunity. There is no operation that lets you write, then decide to synchronize later.
What you should do instead is write data to some other local storage (perhaps a database), then write those records to Firestore when you're ready.
Related
I'm trying to think of the best (read automated, cheapest and easy to use) way to back up Firestore data for a production app.
I'm aware I could automate exports through a scheduled cloud function and send them over to a gcloud bucket. The problem I have with this approach is that it does not allow for "incremental updates of the new and updated documents" but only for backing up entire collections. This means that most of the data will be backed up each and every time, even though it hasn't even changed since the last backup, skyrocketing the cost up for no reason.
The approach that came to mind was having a cloud function in "my-app" project that would listen to each and every change in the Firestore, and perform the same change in the Firestore of the "my-app-backup" project.
This way, I only back up the changed data. Furthermore, backed up data would never become stale (as it's backed up in real-time), unlike the first approach where automated backups happen e.g. daily or weekly.
Is this even possible, having a single cloud function in the first Firebase project writing data into another Firebase project? If not, perhaps write the data elsewhere(not in another Firebase project)? Does the approach even make sense, or do you have a better suggestion?
If you want to export updated documents only then you can store a field updatedAt and query documents where("updatedAt", ">", "lastExportTime"). Then you can periodically run a Cloud function to export these documents. This should only cost N reads (N = number of updated documents) every time the function runs.
Furthermore, backed up data would never become stale (as it's backed up in real-time)
This works too but can also get expensive if the document updates are too frequent.
I have been unable to find any documentation that discusses best practices when it comes to managing images using Firebase.
I am considering whether or not to add the filenames of uploaded images into the database or not.
If they're in the database then I can make one db get request to get the list of existing images. These would be put into the database on fileUploaded events and deleted from the database on delete.
Is it better for me to just do a few file exists requests directly on the Storage and not have the db overhead, for instance (pseudo code);
if ( gcs.exists('/storage_path/favicon.png') ) {
...
}
if ( gcs.exists('/storage_path/favicon-32x32.png') ) {
...
}
Or is it better to store those images filenames in the database, assume they exist, and pull them out in a document .get?
I would like this to be as lightweight as possible, I know in advance the list of filenames we want to know about (they're favicons) so looping and doing .exists is less code, but perhaps slower than putting and pulling from the database.
Any information you have on the efficiency of db document requests and storage exists requests (which I would assume are doing something similar behind the scenes anyway).
Please ask for more information if I'm not clear.
Based on the fact that Firestore clients try to maintain an open socket connection to the database, I'd give the edge to a database get being faster than checking for file existence in Cloud Storage. With Firestore, you're less likely to pay the cost of establishing an SSL connection to the cloud service.
From architectural point of view, I would save the file names in Firestore and retrieve them from the client directly based on the file name, its much simpler and you can add different types of images for different purposes without much thought down the road under same bucket
My application needs to build a couple of large hashmaps before processing a user's request. Ideally I want to store these hashmaps in-memory on the machine, which means it never has to do any expensive processing and can process any incoming requests quickly.
But this doesn't work for firebase because there's a chance a user triggers a new instance which sets off the very time-consuming preprocessing step.
So, I tried designing my application to use the firebase database, and get only the data it needs from the database each time instead of holding all the data in-memory. But, since the cloud functions are downloading loads of data from the database, I have now triggered over 1.7 GB in download for this month, just by myself from testing. This goes over the quota.
There must be something I'm missing; all I want is a permanent memory storage of some hashmaps. All I want is for those hashmaps to be ready by the time the function is called with a request. It seems like such a simple requirement; how come there is no way to do this?
If you want to store data in the container that runs your Cloud Functions, you can use its local tempfs, which is actually kept in memory. But this will disappear when the container is recycled, which happens when your function hasn't been access for a while. So this local file system will have to be rebuilt whenever the container spins up.
If you want permanent storage of values you generate, consider using Google Cloud Storage. It is probably a more cost effective option, and definitively the most scalable one.
Coming from a SQL background, I'm wondering how does one go about doing database migration in firebase?
Assume I have the following data in firebase {dateFrom: 2015-11-11, timeFrom: 09:00} .... and now the front-end client will store and expects data in the form {dateTimeFrom: 2015-011-11T09:00:00-07:00}. How do I update firebase such that all dateFrom: xxxx and timeFrom: yyyy are removed and replaced with dateTimeFrom: xxxxyyyy? Thanks.
You have to create your own script that reads, transform and write it back. You may eider read one node at the time or read the whole DB if it is not big. You may decide to leave the logic to your client when it access to it (if it ever does)
I think you are looking for this: https://github.com/kevlened/fireway
I think is a bad idea to pollute a project with conditionals to update data on the fly.
It is a shame firestore doesn't implement a process for this as it is very common and required to keep the app and db in sync.
FWIW, since I'm using Swift and there isn't a solution like Fireway (that I know of), I've submitted a feature request to the Firebase team that they've accepted as a potential feature.
You can also submit a DB migration feature request to increase the likelihood that they create the feature.
I am planning to create sqlite table on my android app. The data comes from the the server via webservice.
I would like to know what is the best way to do this.
Should I transfer the data from the webservice in a sqlite db file and merge it or should i get all the data as a soap request and parse it in to table or should I use rest call.
The general size of the data is 2MB with 100 columns.
Please advise the best case where I can quickly get this data, with less load on the device.
My Workflow is:
Download a set of 20000 Addresses and save them to device sqlite database. This operation is only once, when you run the app for the first time or when you want to refresh the whole app data.
Update this record when ever there is a change in the server.
Now I can get this data either in JSON, XML or as pure SqLite File from the server . I want to know what is the fastest way to store this data in to Android Database.
I tried all the above methods and I found getting the database file from server and copying that data to the database is faster than getting the data in XML or JSON and parsing it. Please advise if I am right or wrong.
If you are planning to use sync adapters then you will need to implement a content provider (or atleast a stub) and an authenticator. Here is a good example that you can follow.
Also, you have not explained more about what is the use-case of such a web-service to decide what web-service architecture to suggest. But REST is a good style to write your services and using JSON over XML is advisable due to data format efficiency (or better yet give protocol-buffer a shot)
And yes, sync adapters are better to use as they already provide a great set of features that you will want to implement otherwise when written as a background service (e.g., periodic sync, auto sync, exponential backoff etc.)
To have less load on the device you can implement a sync-adapter backed by a content provider. You serialize/deserialize data when you upload/download data from server. When you need to persist data from the server you can use the bulkInsert() method in content-provider and persist all your data in a transaction