Cloud Functions: Function worker crashed with exit code: 3 - firebase

in Node.js command prompt
Your environment has been set up for using Node.js 6.14.0 (x64) and
npm.
run
firebase functions:shell
get error:
i functions: Preparing to emulate functions.
error: FATAL ERROR: Zone Allocation failed - process out of memory
error: Function worker crashed with exit code: 3
error: code=500, status=INTERNAL, message=Function worker crashed
with exit code: 3
undefinedFATAL ERROR: Zone Allocation failed - process out of memory,
errors=[Function worker crashed with exit code: 3
undefinedFATAL ERROR: Zone Allocation failed - process out of memory]
error: FATAL ERROR: Zone Allocation failed - process out of memory
error: Function worker crashed with exit code: 3
! functions: Error from emulator. [object Object]
No functions emulated.

Update: hmmm.... I just noticed that the message comes from the Cloud Functions shell. I don't think my steps below will solve the problem there, but will leave them up just in case.
As the error message says, your Cloud Functions fails because:
process out of memory
You can either reduce the memory your code uses, or increase the memory that is available to your functions. Given that you didn't share details about the function itself, I'll focus on the second option.
To increase the memory available to your functions:
Go to the Cloud Functions console
Select your function
Click Edit
Change the value in the "Memory allocated" dropdown
Click Save

Related

Google Cloud Composer (Airflow) - Scalability issues

I'm facing some issues on my Cloud Composer instance resulting in failed tasks.
Details of instance configuration :
Composer image : composer-2.0.29-airflow-2.3.3 / Airflow version : 2.3.3
Airflow.cfg :
parallelism = 32 / dag_concurrency = 100 / worker_concurrency = 24
In terms of resources :
I have 60 DAGs which can contains up to 55 tasks that needs to run in parallel.
They don't do any compute, only some light PythonOperator/GCSOperator/BigQueryOperator.
I often encounter this type of errors :
*** Log file is not found: gs://xxx/xxx/attempt=2.log.
*** The task might not have been executed or worker executing it might have finished abnormally (e.g. was evicted).
*** Please, refer to https://cloud.google.com/composer/docs/how-to/using/troubleshooting-dags#common_issues hints to learn what might be possible reasons for a missing log.
All of my tasks have 3 retries but when it happens for a reason it stops at 2 retries and send a failure error. I don't understand why. Example of error in mail sent :
Try 2 out of 3
Exception:
Executor reports task instance finished (failed) although the task says its queued. (Info: None) Was the task killed externally?
I also receives random zombie tasks Detected as zombie
My metrics are the following :
When I clear the task, it succeeds as it should.
(I don't have access to GKE but if it helps I can ask to have access)
Any advice to prevent this errors and understand what happens ?

firebase-admin fails with error maxretry without much information about the underlying cause and stops the node script also

I have a mobile and web app that use firebase realtime database and there are some long running tasks which are served on servers with the help of firebase-queue and firebase-admin. The long running tasks is to find out who else is using the mobile app in a person's contact book. So, if you install the app, the app will send a task to the server with your contact book data and ask it to find the people in the contact book who are also using the app. Every now and then I see two types of errors in the logs. First error is below which also cause the node process to stop.
/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15373
return queue[i].onComplete(new Error(abortReason), false, null);
^
Error: maxretry
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15373:52
at exceptionGuard (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4018:9)
at repoRerunTransactionQueue (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15386:9)
at repoRerunTransactions (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15279:5)
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:15260:13
at /Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:7061:17
at PersistentConnection.onDataMessage_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:7088:17)
at Connection.onDataMessage_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5882:14)
at Connection.onPrimaryMessageReceived_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5876:18)
at WebSocketConnection.onMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:5778:27)
at WebSocketConnection.appendFrame_ (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4491:18)
at WebSocketConnection.handleIncomingFrame (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4539:22)
at Client.mySock.onmessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:4438:19)
at Client.dispatchEvent (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2883:30)
at Client._receiveMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:3042:10)
at Client$2.<anonymous> (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2924:49)
at Client$2.emit (node:events:539:35)
at Client$2.emit (node:domain:475:12)
at Client$2.<anonymous> (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2186:14)
at pipe (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1503:40)
at Pipeline$1._loop (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1510:3)
at Pipeline$1.processIncomingMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1479:8)
at Extensions$1.processIncomingMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1645:20)
at Client$2._emitMessage (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2177:22)
at Client$2._emitFrame (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2137:19)
at Client$2.parse (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:1863:18)
at Client$2.parse (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:2369:60)
at IO.write (/Users/varungupta/Projects/myapp-server/node_modules/#firebase/database-compat/dist/index.standalone.js:186:16)
at TLSSocket.ondata (node:internal/streams/readable:754:22)
at TLSSocket.emit (node:events:527:28)
at TLSSocket.emit (node:domain:475:12)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
There isn't much information about what exactly caused the maxretry error. The error happens at random after a few days of running the script. It doesn't happen right away.
The second error that I see that isn't as disruptive as the one above is
[2022-06-01T10:30:49.722Z] #firebase/database: FIREBASE WARNING: transaction at /queue/tasks/-N3TDQCAdt4y-akb0_MK failed: disconnect
This doesn't stop the node process and I can see that a transaction failed but not sure why did it disconnect and how can I resolve this problem.
I am using firebase-admin 9.6.0.

How to fix Airflow logging?

I have S3 remote logging enabled, Airflow is installed on an EC2. My dags are running however, they don't always create a log and then fails. The error is as follows:
*** Falling back to local log
*** Log file does not exist: /home/ec2-user/airflow/logs/REMOVED/REMOVED/2022-03-07T07:00:00+00:00/2.log
*** Fetching from: http://ip-10-105-32-92.eu-west-1.compute.internal:8793/log/REMOVED/REMOVED/2022-03-07T07:00:00+00:00/2.log
*** Failed to fetch log file from worker. Client error '404 NOT FOUND' for url 'http://ip-10-105-32-92.eu-west-1.compute.internal:8793/log/REMOVED/REMOVED/2022-03-07T07:00:00+00:00/2.log'
For more information check: https://httpstatuses.com/404
After a few attempts (3-5), it eventually does end up working.
I have even disabled the remote logging in an attempt to debug, and it still doesn't work. Any suggestions?
Apache Airflow version: 2.2.4
We use a describe stacks API call to get the latest ECS Task Definition, and I've noticed we have lots of these errors:
An error occurred (Throttling) when calling the DescribeStacks operation (reached max retries: 4): Rate exceeded

Grakn mem issue windows local installation

I don't have a tech background. I installed grakn locally on windows into a folder, I am able to start the server and use the workbase, however when I tried to do the compute query in the documentation tutorial (social_network), it throws an error:
2020-08-18 16:59:33,614 [transaction-listener] ERROR g.c.s.r.SessionService$TransactionListener - Runtime Exception in RPC TransactionListener: java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
I then set env variables:
"SERVER_JAVAOPTS": -Xmx4G
"STORAGE_JAVAOPTS": -Xmx4G
Outcome nok:
Starting Storage....FAILED!
Unable to start Storage.
Process exited with code '1': 'Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx4G
The specified size exceeds the maximum representable size.
An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
Process exited with code '1': 'Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Invalid maximum heap size: -Xmx4G
The specified size exceeds the maximum representable size.
I then set env variables:
"SERVER_JAVAOPTS": -Xmx2G
"STORAGE_JAVAOPTS": -Xmx2G
Outcome again nok:
Starting Storage....FAILED!
Unable to start Storage.
Process exited with code '1': ''
An error has occurred during boot-up. Please run 'grakn server status' or check the logs located under the 'logs' directory.
Process exited with code '1': ''
I then set env variables to this:
"SERVER_JAVAOPTS": -Xmx1G
"STORAGE_JAVAOPTS": -Xmx1G
Outcome now is ok:
Starting Storage......SUCCESS
Starting Grakn Core Server.....SUCCESS
...but why can't I assign more mem? I have totally 32GB and usage is around 20%.
Many thanks in advance.
There are two possible reasons for this:
JVM is unable to allocate 4G of RAM because it's occupied by something else. However, as you've said only 20% or RAM is used, this is an unlikely reason.
32-bit JVM are unable to address more than 4GB of RAM 1, so you should check if it applies to you and reinstall your JVM to be 64-bit if it does.

Xcode Linker error: can't allocate region

I want to run an app on the iPad2 but at linking I got this error:
collect2: ld terminated with signal 6 [Abort trap]
ld(69392) malloc: *** mmap(size=16777216) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
I dont know whats the reason for this error. It look like allocating 16777216 bytes (16MB) and the iPad2 should handle that!
Are you sure you got the error at linking, and that it didn't successfully link, install and begin to run, THEN get the error?
The error you have is because malloc can't allocate another 16M block, and THAT is almost certainly because you have either crazy memory fragmentation (possible, but not as common) or a memory leak (very common!)
It would be odd to see this coming form the linker/XCode tools (unless you're running betaware, in which case who knows?!) It's more likely in your app.

Resources