I have a project my-proj in firebase with a firestore database with 153k documents and 50Mb.
I used gcloud cli to export my current firestore in production to a bucket:
gcloud firestore export gs://my-proj.appspot.com/backup
It created a folder in the gs://my-proj.appspot.com/ bucket with a backup.overall_export_metadata and a all_namespaces folder.
I can go to gcloud web interface and check that the saved bucket/directory has 153K documents and uses 50Mb.
I then go to my project folder and run the cli command:
gsutil -m cp -r gs://my-proj.appspot.com/backup ./data
Copying gs://my-proj.appspot.com/backup/all_namespaces/all_kinds/all_namespaces_all_kinds.export_metadata...
Copying gs://my-proj.appspot.com/backup/all_namespaces/all_kinds/output-2...
...
Copying gs://my-proj.appspot.com/backup/all_namespaces/all_kinds/output-7...
/ [10/10 files][ 52.2 MiB/ 52.2 MiB] 100% Done
Operation completed over 10 objects/52.2 MiB.
Also...
❯ du -h ./data/backup
56M ./data/backup/all_namespaces/all_kinds
56M ./data/backup/all_namespaces
56M ./data/backup
So, it copied 56M of data from the cloud bucket to my local folder and I suppose there are 153K documents there.
I then start my firestore emulator with:
firebase emulators:start --import ./data/backup
Emulators start...
I can then go to localhost:4000 > Firestore Emulator > Go to emulator.
In the emulator all collections are there, except the one with 150K documents, which is most of my database. When I click in the link of this collection, it starts loading... then it stops and there is no error message. Just an empty collection.
There is no error in firestore-debug.log.
¯_(ツ)_/¯
P.S. The same procedure with a 3k entries and 1.3Mb file, worked.
Related
Using Firebase tools 11.21.0 and FIREBASE_STORAGE_EMULATOR_HOST=localhost:9199 and maven dependency
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>2.17.1</version>
</dependency>
I started the firebase emulator. And tried a simple file store:
emulatorStorage = StorageOptions.newBuilder()
.setProjectId(projectId)
.setHost("http://localhost:9199")
.setCredentials(NoCredentials.getInstance())
.build()
.getService();
And tried to save a file:
byte[] compress = "test".getBytes();
Blob blob = emulatorStorage.create(
BlobInfo.newBuilder(index, filename)
.setContentType("text/plain")
.build()
,compress,
Storage.BlobTargetOption.doesNotExist());
but even with the content type set I get this every time:
com.google.cloud.storage.StorageException: Failed to parse multipart request body part. Missing content type.
at com.google.cloud.storage.StorageException.translate(StorageException.java:163)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:297)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.create(HttpStorageRpc.java:379)
at com.google.cloud.storage.StorageImpl.lambda$internalCreate$2(StorageImpl.java:208)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.Retrying.run(Retrying.java:60)
at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1476)
at com.google.cloud.storage.StorageImpl.internalCreate(StorageImpl.java:205)
at com.google.cloud.storage.StorageImpl.create(StorageImpl.java:151)
and through debug I know that it is talking to the local emulator:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
POST http://localhost:9199/upload/storage/v1/b/demo-project.appspot.com/o?ifGenerationMatch=0&projection=full&uploadType=multipart
{
"code" : 400,
"message" : "Failed to parse multipart request body part. Missing content type."
}
What am I missing in the save operation, is the content type wrong? Or is this likely to be a bug in the emulator or compatibility issue with cloud-storage libs?
I have found github issue you raised and followed steps from there with both firebase v11.21.0 And v11.19.0 but I can successfully upload video files using firebase emulators:start --project demo-project --debug.
As per our conversation above in comments, it seems like you have mistaken firebase-tools-linux as a command. the doc you followed to set up the forebase for linux will just have downloadable file name as firebase-tools-linux It is just file name it does not mean that for linux we have to use firebase-tools-linux as a command for linux machines.If you observe step 3 in the doc you shared, It is pointing to log in and test the CLI where we have to use firebase login only. Hence try with firebase emulators:start --project demo-project --debug command.
Steps I have taken
Step 1:
Cloned source code from the github you shared. Changed directory to firebase-emulator-debug.
Step 2:
Ran below command
`firebase emulators:start --project demo-project --debug`.
Step 3:
Successfully uploaded video file of 2.3 MB from emulator.
FYI, I have also used the linux machine only for the above steps.
As mentioned by #Gridcell Coder, The Cloud Storage for Firebase emulator only supports a very small subset of the Cloud API and it is intended to only be used via the firebase-admin package. Admin SDK is not yet supported for Cloud Storage for Firebase.
I have a Firebase code project with Functions meant to be deployed to multiple Firebase projects over multiple regions.
I used to set the deployment region like this:
return functions
.region(process.env.REGION)
// ...
and used this command to deploy:
$ REGION=us-central1 firebase deploy --only functions
it worked like a charm until recently. Now it seems to completely ignore REGION=us-central1 even if I export it before I run firebase deploy.
EDIT 2022-06-13 - Possible solution
I changed the code to dump the contents of process.env to a file during deployment. This is what I got:
{
"FIREBASE_CONFIG": "{\"projectId\":\"REDACTED\",\"storageBucket\":\"REDACTED.appspot.com\",\"locationId\":\"us-central\"}",
"GCLOUD_PROJECT": "REDACTED",
"CLOUD_RUNTIME_CONFIG": "{REDACTED}",
"__CF_USER_TEXT_ENCODING": "REDACTED"
}
so definitely is not the same list of variables I have in my local environment.
I could use the locationId from FIREBASE_CONFIG to get the target location, or CLOUD_RUNTIME_CONFIG (it contains the dump of the functions .config() object, so I could set the target there).
I also believe that I could use the .env and .env.{project alias or ID} files and their contents would be available in process.env at deployment time.
As per Osvaldo López's suggestion, here are some other details:
Running on MacOS
No errors are reported
No recent changes to the CLI
Any input would be very welcome! Thanks.
I have a simple method like
exports.updatePlayer = functions.firestore
.document('matches/{docId}')
.onCreate((change, context) => {
console.log('fired------------', context.params.docId);
});
If I add a new doc to matches, I can see the console message in logs.
Question is how I can run it locally? I have this cmd in package.json
"serve": "firebase emulators:start --only functions,database",
When I run `npm run server', console says
> functions# serve D:\cric\fs-functions\functions
> firebase emulators:start --only functions,database
i emulators: Starting emulators: functions
! database: Not starting the database emulator, make sure you have run firebase init.
+ hub: emulator hub started at http://localhost:4400
! Your requested "node" version "8" doesn't match your global version "12"
+ functions: functions emulator started at http://localhost:5001
i functions: Watching "D:\cric\fs-functions\functions" for Cloud Functions...
! functions: The Cloud Firestore emulator is not running, so calls to Firestore will affect production.
+ functions[allMatches]: http function initialized (http://localhost:5001/xxxx-xxxxx/us-central1/allMatches).
i functions[updatePlayer]: function ignored because the firestore emulator does not exist or is not running.
+ emulators: All emulators started, it is now safe to connect.
But now if I add something to firestore collection, I am not getting any console message on my local terminal. Also not that
i functions[updatePlayer]: function ignored because the firestore emulator does not exist or is not running.
I want to run this trigger locally, so can write/test my code fast.
Try to update your NodeJs version
Firebase announced the deprecation of Node.js 8 runtime on Cloud Functions and the decommissioning of the Node.js 6 runtime.
To ensure that your functions are on a supported version of Node.js, migrate to the Node.js 10 runtime as soon as possible.
Update from Node.js 8 to Node.js 10 specifying the Node.js 10 runtime explicitly by setting the engines field in package.json
NodeJs End-of-Life Releases
With my vue.js app I followed the exact steps from the official vue documentation to build and deploy it on firebase hosting.
My .firebaserc looks as prescribed:
{
"projects": {
"default": "trackomad"
}
}
However, the deployment gets stuck after one file and exits with the message:
$ firebase deploy
=== Deploying to 'trackomad'...
i deploying firestore, hosting
i firestore: checking firestore.rules for compilation errors...
i firestore: reading indexes from firestore.indexes.json...
✔ firestore: rules file firestore.rules compiled successfully
i firestore: uploading rules firestore.rules...
✔ firestore: deployed indexes in firestore.indexes.json successfully
i hosting[trackomad]: beginning deploy...
i hosting[trackomad]: found 9 files in dist
⠸ hosting: uploading new files [1/2] (50%)
Error: Task ce6ebf8fe181eb301634f0abb5063dd4b23f6b1b724d2f85872f69053c6832e8 failed: retries exhausted after 6 attempts
This project of mine already had a deployment before that was a simple dummy project as it is provided by firebase documentation. Initially, I tried to just deploy the wholly new vue app over it and got the same errors. I then deleted the old deployment as described here. The only thing that changed is that it now gets stuck at ⠸ hosting: uploading new files [1/2] (50%) and not as previously at ⠸ hosting: uploading new files [1/4] (25%)
Can I debug this behaviour somehow or are there any other possibilities to understand what's wrong?
It's an incident on hosting, you can see the status here https://status.firebase.google.com/incident/Hosting/19015
I'm familiar with firebase and fire storage, but new to G-Cloud-SDK. I found this topic https://groups.google.com/forum/#!msg/firebase-talk/oSPWMS7MSNA/RnvU6aqtFwAJ and tried to copy the results, but get "IOError: [Errno 2] No such file or directory: u'cors-json-file.json'" from the terminal.
I found a bit more information in the docs https://cloud.google.com/storage/docs/configuring-cors#configure-cors-gsutil and it describes the same process, and I get the same error.
I'm finding it hard to do any troubleshooting with Google Cloud SDK since I have already setup functions in Firebase.
gsutil cors set cors.json gs://example-bucket
Example: If the cors.json file in the d: drive then the path will be like
d:/folderName/cors_filename. The gs url is your bucket url.
I did not move my local directory to my 'cors-json-file.json' ...