I have a search index with several documents. I can delete using the python documentation here https://cloud.google.com/appengine/training/fts_adv/lesson3, however I was curious as to if anyone knows a way to do it via gcloud or the admin console?
I have tried via the gcloud datastore indexes cleanup command. and generated a index.yaml file following directons via:https://cloud.google.com/datastore/docs/tools/indexconfig, but it does not clear the documents away like the delete method does within python.
I didnt think the datastore one would work, for it lives within the appengine, so I am thinking i am looking in the wrong place so any help would be much appreciated! ^_^
There is no gcloud command that will allow you to delete the documents on Datastore. There are only commands to delete the indexes or the operations which you can find over here.
If you are using Datastore of Firestore in Datastore mode, you can use a Dataflow job to delete a bulk of entities. There are some template jobs that you can use to do this as mentioned in this document.
If you want to delete a document from Firestore in Native mode, you would need to do so through the delete() method as mentioned in the documentation, nevertheless, you would need to keep in mind that deleting a document does not delete any subcollection, you would need to retrieve the documents and subcollections in order to delete them manually as expressed here.
Hope you find this useful.
Related
I have understood that each time i deploy a firebase cloud function, firebase automatically save some data in firebase storage. I would like to know excatly which data is saved and if it is safe to delete, or any suggestion on what to deleate.
I have a small project with 200MB of cloud storage used for user data and 10 cloud functions doing a variety of task. At the begining of October, I did few firebase deploy action and now I see a whooping 900MB of us.artifacts data. See below the screeneshot from firebase and google cloud. It looks like I have 256 items there with a mysterious .dms extension (disk image) and I really do not understand what this exaclty is, if i need to keep it or not.
I am currently considering to remove all these file and run a fresh firebase deploy but not sure if this is going to create any problem
Many thanks to anyone who can shine a bit of light here?
As suggested in the comments, you can remove the files and that would not cause any issues.
I have to Firebase projects, one is for dev and another is for production. I create a bunch of collections and after they successfully pass the test, I'll need to move them into production database. How can I do this without using Cloud Shell, or are there any alternative suggestions of database?
Thank you!
You can export the collection using cloud shell
Manage export ad import
In case you want to move collections between your dev and your production databases, without using Cloud Shell, there is an alternative that you can follow.
For you to achieve that, you will need to follow the below steps.
Create a Cloud Storage bucket to hold the data from your source project.
Export the data from your source project to the bucket.
Give your destination project permission to read from the bucket.
Import the data from the bucket into your destination project.
With these steps, you should be able to migrate data between your projects and have this way, the structure you want of a Development database and a Production database, where you can easily transfer the data. I would recommend you to check the official documentation Move Data Between Projects, to get the whole tutorial on how to achieve the above steps, in case you have doubts on how to achieve them.
Let me know if the information helped you!
I have an open-source project that uses two separate Firebase projects for a test environment and the production one.
Ultimately, I want to have other developers be able to pull down the project and actually be able to run it WITHOUT me needing to give each individual developer access.
I see a number of solutions in this question: How to add collaborators to a Firebase app?. Those all seem to require each person's email.
I understand why it maybe isn't a thing, but I am wondering if there is a way to just give access to everyone for only the test project so that contributing is super low-friction. Something similar to Firestore database rules that allow read/write in a public fashion to get started for testing.
I've tried making a new IAM account in the Google Cloud Console, and I think that partially worked for the Firebase Cloud Functions access to Admin SDK, but my collaborators get hung up trying to run firebase use <test-firebase-project> saying that they don't have access.
I see a lot of other config options for IAM, but nothing sticking out to me for this public access scenario.
Can anyone confirm this either is or isn't a thing?
Thanks!
EDIT
To add some more detail to my project...
I am using Authentication, Firestore, and Cloud Functions. The only js package I will use is the Auth one, which will be loaded from a CDN (so I believe that doesn't apply to my question).
I want to give access to people to run the Cloud Functions locally. There is a pre-build step that I have made in vanilla Node that calls a Cloud Function (running locally), which uses the Firebase Admin SDK to call my Firestore database. I then write that response to a JSON file that my front end uses.
When my collaborators pull down the project, and install the Firebase CLI, when they try to serve the Cloud Functions locally, they get hit with a "no access" type of error. I was trying to resolve this by adding a service account into the /functions directory, but apparently that didn't help.
My collaborators don't need access to the console at all. I basically just need them to be able to run the Cloud Function locally (and make sure they can't access my production Firebase project).
I am happy to add more detail, but I kind of feel like I am just rambling. Thanks again.
There is no way to grant everyone contributor access to your Firebase console. You will either have add each individual user, or create your own dashboard that uses the API to show the relevant data.
I have a Firebase project which basically have two environments: Staging and Production. The way I organized them is by creating different Firebase projects. Each of my projects uses Firebase Cloud Functions and Firestore. Except for that, I have each of the projects associated with a specific GIT branch. Both of the branches are integrated into CI/CD pipeline in Google Cloud Build.
So, in order to make it absolutely clear, I will share a simple diagram:
As you can see, I have the source code for the cloud functions under source control and there's nothing to worry about there. The issue comes in when I have the following situation:
A Firestore schema change is present on Staging
Cloud function (on Staging) is adjusted to the new schema.
Merge staging branch into production.
Due to the old Firestore schema on production, the new functions there won't work as expected.
In order to work around it, I need to manually go to the production Firestore instance and adjust the schema there (there's a risk to mess up production data).
In the perfect case, I would have that operation automated and existing project data would be adjusted to a new schema which comes in dynamically after merge.
Is that possible somehow? Something like migrations in .NET Core.
Cloud Firestore is schema-less - documents have no enforced schema. Code is able to write whatever fields it wants at any time that it wants. (For web and mobile clients, this is gated by security rules, but for backend code, there are no restrictions.) As such, there is no such thing as a formal migration in Cloud Firestore.
The "schema" of your documents is effectively defined by your code that reads and writes those documents. This means that migrating a data to a new format means that you're going to have to write code to perform the required changes. There is really no easy way around this. All you can really do is design your updates so that they are not disruptive to existing code when it comes time to move them to another environment. This means your code should be resilient to breaking changes, or simply do not perform breaking changes until after all code has been updated to deal with those changes.
You have to use Google Cloud to download an archive of the Firestore data. Run a migration script yourself on the archive, and then upload the archive to restore your Firestore database.
https://cloud.google.com/firestore/docs/manage-data/export-import
Google Cloud gives you a lot of command line access for managing your Firestore service.
// manage indexes
gcloud firestore indexes
// export all data to a bucket
gcloud firestore export gs://[BUCKET_NAME]
// import data from a bucket
gcloud firestore import gs://[BUCKET_NAME]/[filename]
// manage admin "functions" currently running (i.e. kill long processes)
gcloud firestore operations
To download/upload your JSON archive from Google Cloud buckets
// list files in a bucket
gsutil ls gs://[BUCKET_NAME]
// download
gsutil cp gs://[BUCKET_NAME]/[filename] .
// upload
gsutil cp [filename] gs://[BUCKET_NAME]/[filename]
Once you setup Google Cloud to be accessible from your build scripts. It's possible to automate data migration scripts to download, transform and then upload data.
It's recommended to maintain a "migrations" document in your Firestore so you can track which reversion of the migration needs to be done.
To avoid heavy migration tasks try adding a "version" property to documents, and then use the converter callbacks on the query builder to mutate data to the latest schema on "client side". While this won't help you handle changes with Firestore rules or functions. It's often easier to make tiny changes that are mostly cosmetic.
I would like to understand how does the Alfresco Repo works whenever any document is uploaded. How exactly the content is stored in file system metadata in DB and indexes in Solr respectively ?
You have to dive deep down in Alfresco documentation and code as well for getting all these clarifications.
Alfresco stores the documents in the physical directories in proper folder structures(which you can define in your code as well). CMIS services expose the functionalities for all operations which you can perform on documents. CMIS internally uses cache in CMIS session for better performance.
Documents metadata and node references all are being stored in DB(PostgreSQL) and indexing of documents are being done automatically through Solr in latest versions.
You have to specify the fields which you want in Solr document for indexing. Searching documents in Solr is faster than DB but queryConsistency is Eventual for Solr. So, as per your use case you have to decide whether query Solr or DB.
So, whenever any operation(CRUD) on any document is being done, it reflects in DB first then async indexing of that document happens in Solr. This Async indexing leads to Eventual consistency.