I can upload a new object to the google storage bucket, and it will put the object in the database for the firebase project. But when I delete it does not update the google storage bucket -- any tips for how to get to the bottom of this?
Additionally there are a couple thousand objects in the google storage but not in the firebase database. How do I sync these two so the objects in the GS bucket are also in the firebase database
I tried a test and using gutils rsync uploaded an image, then deleted it in the firestore and when i tried uploading it again I get
Building synchronization state...
At destination listing 10000...
At destination listing 20000...
At destination listing 30000...
At destination listing 40000...
Starting synchronization...
Copying file://./.DS_Store [Content-Type=application/octet-stream]...
- [1/1 files][ 6.0 KiB/ 6.0 KiB] 100% Done
UPDATE:
I believe it has to do with an issue with 'application/octet-stream' vs image/jpeg and deletions -- any advice for how to switch those over to image/jpeg
You can write two Cloud Functions triggers, one that responds to changes in your storage bucket (a storage trigger), and another the responds to changes in your database (a Firestore trigger). If either function detects a change, it can make a corresponding update in the other product.
If you want to perform a one-time synchronization, you will just have to write code to find the differences and make the appropriate changes. A Cloud Functions trigger would not be the best way to do this.
Related
I'm trying to implement a system that allows react-native clients to upload files to a specific folder in Cloud Storage and allows clients to download from them. I can't do this directly from the client because I first need to query Firestore to validate that the user is 'friends' with the owner of the folder which will allow for read/write permissions.
I decided to use Cloud Functions as a middle layer to encapsulate this business logic and I expected to also be able to use it as a middle layer to upload the files. However, I feel like I may be misunderstanding how to best use these services together to solve this problem.
Current Plan:
Client uploads file to Cloud Function (assuming they are permitted after Cloud Function queries Firestore and validates)
Cloud Function uploads file to Cloud Storage
Client can then request file from Cloud Function, which validates permissions using Firestore and downloads file from CloudStorage
Cloud Function sends file to client
Questions:
Can/Should I use Cloud Functions in this way as a middle layer to upload files after validating permissions store in Firestore?
Is there an alternative solution using firebase that would mitigate the 10MB download limit with Cloud Functions but still allow me to authenticate uploads/downloads to and from Cloud Storage using my custom business logic on relationships in Firestore?
Any help or guidance here is appreciated! I'm very new to firebase, cloud architecture, and architecture in general.
This is definitely a valid and technically feasible approach. Whether you should do it, only you can determine of course.
An alternative is to use Firebase Storage's security rules to enforce the access control on the files. But you won't be able to read anything from Firestore in those rules, so you'll have to ensure everything needed to grant/deny access is in the path of the requested file, in the file's metadata, and/or in the user's token.
Even if you implement the download of files in Cloud Functions, you can still perform uploads through Firebase. You could in that case for example have the user write to a temporary location, which then triggers a Cloud Function that can do whatever additional checks you want on the file.
I am making an android app where the user can upload images under some category. It will be saved on firestore with category collection with image description and path of the image of firebase storage. I want to delete the photo which are 7 days old automatically using cloud function. I don't want to involve the client device to trigger the function.
You will end up a writing a lot of code for this. A full discussion is outside the scope of a Stack Overflow answer. The general procedure goes like this:
Write a Storage trigger that fires when the user uploads the file.
In that trigger, use Cloud Tasks to schedule an invocation of an HTTPS function using its URL, delayed after 7 days. The task must be configured to deliver a payload to the function that tells it which file to delete.
In the HTTPS trigger, receive the payload, and delete the file from Storage.
It will be similar to an existing detailed pattern for giving a Firestore document a TTL, except you are using a Storage file instead of a Firestore document.
let's say I have an image /path/image1.png in the firebase storage. I want to copy this image and create a new image with a different name but the same content as /path/image2.png. I'm using AngularFire. How will I achieve this? Please help
Firebase Storage (nowadays called Cloud Storage for Firebase) is a set of client-side SDKs that allow you to access Cloud Storage from within your application.
There is no Firebase API to create a copy of a file in Cloud Storage. It's a valid use-case though, so I'd recommend you file a feature request for it.
In the meantime, the two options I can think of are:
Read the data to the client, and write it to the new location. This is definitely wasting bandwidth for the client, so I'd only consider it if your files are quite small, and your clients have a decent connection.
Use one of the (non-Firebase) server-side SDKs for Cloud Storage to create a copy of the file. For example, you could wrap this Node.js code in a callable Cloud Function and then call that from your application code.
await storage
.bucket(srcBucketName)
.file(srcFilename)
.copy(storage.bucket(destBucketName).file(destFilename));
I have an app that has 50k - 60k document reads a day and I can't afford another plan now, so I'm looking for a way to optimize reads, getting less reads possible I can.
The database has 4 collections with some subcollections, I think I might have around 1000 documents in the whole server.
It is there anyway: I can read documents from cloud Firestore (first time the user opens the app) and store on the Firestore cache, after that, make the app only load data from Firestore cache, and if there is a new document(s), it reads these documents only, store to the cache, and keep reading from the cache?
The app could only load data from cache (using the parameter "source") and never directly from Firestore, if possible.
Firestore.instance.collection("images").getDocuments(source: Source.cache);
Is is there anyway: I can read documents from cloud Firestore (first time the user opens the app) and store on the Firestore cache
That's the default behavior. According to the official documentation regarding Firestore offline persistence:
For Android and iOS, offline persistence is enabled by default. To disable persistence, set the PersistenceEnabled option to false.
For the web, offline persistence is disabled by default. To enable persistence, call the enablePersistence method. Cloud Firestore's cache isn't automatically cleared between sessions. Consequently, if your web app handles sensitive information, make sure to ask the user if they're on a trusted device before enabling persistence.
So there is nothing special that you need to do. Once you open a stream on one of your collections, the data is added to the cache.
after that, make the app only load data from Firestore cache, and if there is a new document(s), it reads these documents only, stores to the cache, and keeps reading from the cache?
That's again the default behavior, but this work as long as the documents in your database are not changed. If a document in the database is changed, you'll be charged with a document read for each document. The mechanism is the same.
Besides that, you can tell Firestore to read data only from the cache if needed, as explained in the answer from the following post:
How to cache Firebase data in Flutter?
It seems odd to me that Firestore would charge me for read queries to locally cached data, but I can't find any clarification to the contrary in the Firestore Pricing document. If I force Firebase into offline mode and then perform reads on my locally cached data, am I still charged for each individual entity that I retrieve?
Second, offline users in my app write many small updates to a single entity. I want the changes to persist locally each time (in case they quit the app), but I only need eventually consistent saves to the cloud. When a user reconnects to the internet and Firestore flushes the local changes, will I be charged a single write request for the entity or one per update call that I made while offline?
Firestore could potentially fit my use case very well, but if offline reads and writes are charged at the same rate as online ones it would not be an affordable option.
As the offical documentation says,
Cloud Firestore supports offline data persistence. This feature caches a copy of the Cloud Firestore data that your app is actively using, so your app can access the data when the device is offline. You can write, read, listen to, and query the cached data. When the device comes back online, Cloud Firestore synchronizes any local changes made by your app to the data stored remotely in Cloud Firestore.
So, every client that is using a Firestore database and sets PersistenceEnabled to true, maintains it's own internal (local) version of the database. When data is inserted/updated, it is first written to this local version of the database. As a result, all writes to the database are added to a queue. This means that all the operations that where stored there will be commited on Firebase servers once you are back online. This also means that those operations will be seen as independent operations and not as a whole.
But remeber, don't use Firestore as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
If Google Cloud Firestore priceing model does not fit your use case very well then use Firebase Realtime Database. As mentioned also in this post from the Firebase offical blog, one the reasons you still might want to use the Realtime Database is:
As we noted above, Cloud Firestore's pricing model means that applications that perform very large numbers of small reads and writes per second per client could be significantly more expensive than a similarly performing app in the Realtime Database.
So it's up to you which option you choose.
According to this If you want to work completely offline with Cloud Firestore you can disable network by :
FirebaseFirestore.getInstance().disableNetwork()
but firestore will cause client offline error for first user get request, that you must consider this error as empty response.