I'm having some serious issues understanding why the emulator won't import the exported firestore data I am feeding it. It keeps throwing a memory error. Any help/ideas?
These are the logs given by firestore-debug.log when trying to run command:
firebase emulators:start --import=./production_data_export
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.nio.ByteBuffer.allocate(Unknown Source)
at com.google.cloud.datastore.emulator.impl.LevelDBLogReaderChannel.readRecord(LevelDBLogReaderChannel.java:84)
at com.google.cloud.datastore.emulator.impl.LevelDBLogReaderChannel.readProtoMessage(LevelDBLogReaderChannel.java:63)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseEntityProtoFile(ExportImportUtil.java:194)
at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:66)
at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:90)
folder structure of production_data_export
code for firebase-export-metadata.json
{
"version": "10.2.0",
"firestore": {
"version": "1.13.1",
"path": "",
"metadata_file": "mock-db.overall_export_metadata"
}
}
If you are trying to import a huge dataset, then you can try tweaking the memory size using export JAVA_TOOL_OPTIONS="-Xmx4g" (this sets it up for 4GB I think - not sure tho.... so you can increase or reduce based on what your machine is capable of). if you are on a linux or mac machine. I think for windows it is set JAVA_TOOL_OPTIONS="-Xmx4g"
Got the idea from the official issues ticket: https://github.com/firebase/firebase-tools/issues/4336#issuecomment-1075439749
Related
I have a mobile and web app that use firebase realtime database and there are some long running tasks which are served on servers with the help of firebase-queue and firebase-admin. The long running tasks is to find out who else is using the mobile app in a person's contact book. So, if you install the app, the app will send a task to the server with your contact book data and ask it to find the people in the contact book who are also using the app. Every now and then I see two types of errors in the logs. First error is below which also cause the node process to stop.
/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15373
return queue[i].onComplete(new Error(abortReason), false, null);
^
Error: maxretry
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15373:52
at exceptionGuard (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4018:9)
at repoRerunTransactionQueue (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15386:9)
at repoRerunTransactions (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15279:5)
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15260:13
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:7061:17
at PersistentConnection.onDataMessage_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:7088:17)
at Connection.onDataMessage_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5882:14)
at Connection.onPrimaryMessageReceived_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5876:18)
at WebSocketConnection.onMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5778:27)
at WebSocketConnection.appendFrame_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4491:18)
at WebSocketConnection.handleIncomingFrame (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4539:22)
at Client.mySock.onmessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4438:19)
at Client.dispatchEvent (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2883:30)
at Client._receiveMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:3042:10)
at Client$2.<anonymous> (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2924:49)
at Client$2.emit (node:events:539:35)
at Client$2.emit (node:domain:475:12)
at Client$2.<anonymous> (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2186:14)
at pipe (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1503:40)
at Pipeline$1._loop (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1510:3)
at Pipeline$1.processIncomingMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1479:8)
at Extensions$1.processIncomingMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1645:20)
at Client$2._emitMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2177:22)
at Client$2._emitFrame (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2137:19)
at Client$2.parse (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1863:18)
at Client$2.parse (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2369:60)
at IO.write (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:186:16)
at TLSSocket.ondata (node:internal/streams/readable:754:22)
at TLSSocket.emit (node:events:527:28)
at TLSSocket.emit (node:domain:475:12)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
There isn't much information about what exactly caused the maxretry error. The error happens at random after a few days of running the script. It doesn't happen right away.
The second error that I see that isn't as disruptive as the one above is
[2022-06-01T10:30:49.722Z] #firebase/database: FIREBASE WARNING: transaction at /queue/tasks/-N3TDQCAdt4y-akb0_MK failed: disconnect
This doesn't stop the node process and I can see that a transaction failed but not sure why did it disconnect and how can I resolve this problem.
I am using firebase-admin 9.6.0.
I am trying to create a luminus project with sqlite and cljs support with this command:
lein new dbscript +sqlite +cljs
but I keep getting this error message all the time:
Failed to resolve version for dbscript:lein-template:jar:RELEASE: Could not find metadata dbscript:lein-template/maven-metadata.xml in local (/home/jonas/.m2/repository)
Failed to read artifact descriptor for dbscript:lein-template:jar:RELEASE
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Could not find template dbscript on the classpath.
and I cannot find any solution to this. Any suggestions?
I am trying to deploy my application built in ASP.Net 4.6.1. So I am using HWC Buildpack.
Below is my manifest.yml
---
applications:
- name: DRSN
random-route: true
memory: 128M
buildpack:
https://github.com/cloudfoundry/hwc-buildpack.git
env:
DOTNET_CLI_TELEMETRY_OPTOUT: 1
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
The error that I am receiving is below.
Waiting for API to complete processing files...
Staging app and tracing logs...
Cell 0f7012eb-9e32-4fdf-ba92-85aee4639139 creating container for instance 34107c3c-1acb-4aa5-b435-b06516abcfcb
Cell 0f7012eb-9e32-4fdf-ba92-85aee4639139 successfully created container for instance 34107c3c-1acb-4aa5-b435-b06516abcfcb
Downloading app package...
Downloading build artifacts cache...
Downloaded build artifacts cache (231B)
Downloaded app package (19.5M)
Failed to compile droplet: Failed to compile droplet: fork/exec /tmp/buildpackdownloads/6c6dca8d638ac0d145d6581f9eb9a96a/bin/compile: permission denied
Exit status 223
Cell 0f7012eb-9e32-4fdf-ba92-85aee4639139 stopping instance 34107c3c-1acb-4aa5-b435-b06516abcfcb
Cell 0f7012eb-9e32-4fdf-ba92-85aee4639139 destroying container for instance 34107c3c-1acb-4aa5-b435-b06516abcfcb
Error staging application: App staging failed in the buildpack compile phase
Can anyone help me resolve this issue? Am I not correct in my manifest.yml? Or is it something else?
I believe that the problem is that you're telling the system to use the HWC buildpack, but at the same time you're not setting the Windows stack (at least based on what info I can see). That means it's going to default to the Linux stack, which I believe is why you're seeing the fork/exec /tmp/buildpackdownloads/... error.
Try adding stack: windows to your manifest.yml or -s windows to your cf push command (for future reference, when you need help always include the full cf push command you're running).
PS: you shouldn't use https://github.com/cloudfoundry/hwc-buildpack.git that is telling the system to grab the master branch in whatever state it's currently in. That's a.) not reproducible and b.) not guaranteed to be in a working state. You should either use the platform provided buildpack names (from cf buildpacks) or append #<branch_or_tag> to the end of the URL so it picks a specific branch. All CF Buildpacks have tags for each release. It's strongly recommended you use a tagged release.
I built this React App, that allows users to add todo item, and mark an item as completed/active. Todo items are stored in cloud firestore.
I want to enable full-text search on the app, and one way is to create a cloud function that writes index to Algolia whenever cloud firestore witnesses a create operation.
Inside my app root directory, I initialize cloud functions and then implement the onTodoAdded function. After done, I run firebase serve under functions directory and get the errors:
=== Serving from '/Users/evan/source/react-redux-todos'...
i functions: Preparing to emulate functions.
Warning: You're using Node.js v10.6.0 but Google Cloud Functions only supports v6.11.5.
i hosting: Serving hosting files from: build
✔ hosting: Local server: http://localhost:5000
error: SUPERVISOR error Error: listen EADDRINUSE 127.0.0.1:5001
at Server.setupListenHandle [as _listen2] (net.js:1335:14)
at listenInCluster (net.js:1383:12)
at GetAddrInfoReqWrap.doListen [as callback] (net.js:1509:7)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:61:10)
error: supervisorPort (5001) is already in use
i functions: No HTTPS functions found. Use firebase functions:shell if you would like to emulate other types of functions.
For a complete debug log, find here.
However, this command works fine: firebase serve --only functions.
=== Serving from '/Users/evan/source/react-redux-todos'...
i functions: Preparing to emulate functions.
Warning: You're using Node.js v10.6.0 but Google Cloud Functions only supports v6.11.5.
i functions: No HTTPS functions found. Use firebase functions:shell if you would like to emulate other types of functions.
Since it says supervisorPort(5001) is already in use, I lsof -i:5001 and finds no process. I also shut down my laptop and restart, gets the same error.
I find the similar issue here, however didn't find the solution.
I still don't know the reason, but adding --port=9000 solves the problem. For example, firebase serve --only functions --port=9000. Credits to restPort (5001) is already in use #142.
I was having this same error after updating to macOS Mojave 10.14.1 (was fine with 10.14). error: supervisorPort (5001) is already in use and it led me down this rabbit hole of additional updates and configuration that eventually got it to work.
1 Update Node Packages
I updated these items to these versions (latest as of this writing)
firebase-functions: 2.1.0
firebase: 5.5.8
I deleted by node_modules folder and reinstalled all the packages.
2 Firestore Emulator Setup Required
Next time I attempted to serve, firebase gave me this error:
Setup required, please run: firebase setup:emulators:firestore
Because of the weird permissions issues I initially had with mac, I actually have to run
node_modules/.bin/firebase setup:emulators:firestore
3 Then.... Fix Java Runtime Issues
On my next serve attempt, I was presented with firestore: No Java runtime present, requesting install. Mac presents a popup leading to a page where you can "download java for mac". This isn't what you actually need. You really need the Java SE Development Kit which enables command line interface.
Then... You may get these errors the next time you try to serve.
error: restPort (5002) is already in use
error: SUPERVISOR error Error: listen EADDRINUSE 127.0.0.1:5001
at Object._errnoException (util.js:1022:11)
at _exceptionWithHostPort (util.js:1044:20)
at Server.setupListenHandle [as _listen2] (net.js:1351:14)
at listenInCluster (net.js:1392:12)
at GetAddrInfoReqWrap.doListen [as callback] (net.js:1501:7)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:97:10)
error: supervisorPort (5001) is already in use
But Firebase still ended up continuing to successfully serve somehow, despite these errors. And it still shows my app over http://localhost:5000.
Hope this helps!
We are trying to use Web Deploy to improve our deployment.
Everything works internally but once we try to deploy to our online environment than msdeploy hangs on files lager than 1 MB.
If we remove those files, the sync works with msdeploy.
The client tries a few times to upload the file and then gives the following error:
Warning: Retrying the sync because a socket error (10054) occurred.
Retrying operation 'Serialization' on object MSDeploy.contentPath (sourcePath).
Attempt 4 of 25.
Info: Using ID '789c2466-2bae-4b56-b67d-1a1743cdb0f6' for connections to the remote server.
On the server we see the following error:
Content-Type: application/msdeploy
Version: 8.0.0.0
MSDeploy.VersionMin: 7.1.600.0
MSDeploy.VersionMax: 7.1.1070.1
MSDeploy.Method: Sync
MSDeploy.RequestId: 789c2466-2bae-4b56-b67d-1a1743cdb0f6
MSDeploy.RequestCulture: en-US
MSDeploy.RequestUICulture: en-US
Skip: objectName="^configProtectedData$"objectName="filePath",absolutePath=".*web.config$"objectName="filePath"
Provider: contentPath, Path: demo.com
Tracing deployment agent exception. Request ID '789c2466-2bae-4b56-b67d-1a1743cdb0f6'. Request Timestamp: '5/07/2011 11:29:56'. Error Details:
System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection
at System.Net.HttpResponseStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.BufferedStream.Dispose(Boolean disposing)
at System.IO.Stream.Close()
at System.IO.StreamWriter.Dispose(Boolean disposing)
at System.IO.StreamWriter.Close()
at System.Xml.XmlTextWriter.Close()
at Microsoft.Web.Deployment.TraceEventSerializer.Dispose()
at Microsoft.Web.Deployment.DeploymentAgent.HandleSync(DeploymentAgentWorkerRequest workerRequest)
at Microsoft.Web.Deployment.DeploymentAgent.HandleRequestWorker(DeploymentAgentAsyncData asyncData)
at Microsoft.Web.Deployment.DeploymentAgent.HandleRequest(DeploymentAgentAsyncData asyncData)
Anyone any suggestions to fix the problem?
I have not had this issue so cannot give you a specific fix but the microsoft documentation says
Why does remote deployment fail for large files, although local deployment succeeds?
Typically when you deploy to production servers you are deploying across a firewall.
If you get end-of-stream errors on large files, check the settings on the firewall.
Unfortuately it doesnt give specifics regarding the firewall setting to check