I just updated my Cloud Composer environment to composer-1.10.3-airflow-1.10.6 and started periodically seeing the yellow warning box in the web ui that the scheduler hasn't been responding. Looking in the logs for airflow-scheduler I'm seeing the errors below, and it appears to be happening after composer mounts a bucket with FUSE.
I'm also seeing a large number of restarts happening, which I'm guessing is being caused by the errors above.
Has anyone else run into this or know what would be causing this error/how to fix?
Related
I've developed NextJS Project and already deployed to Vercel. Today when I updated my code and pushed to Github, and begin to deploy on Vercel, it shows me error. Build logs is attached in screenshot.
I've tried to delete node_modules, cleaning cache and all, but it didn't work.
The strange thing I'm not seeing any error and the reason for failing. In my local host, everything is working perfectly and no errors whatsoever, but Vercel is not giving me any error but just one notification:
Error: Command "npm run build" exited with 1
Can somebody let me know please what could be the reason of it? I can see there are some warnings but these warnings I suppose are not the main cause of this strange issue.
Thanks for reply. After spending 2-3 hours , I had intentionally put an error in my code, deleted one dependency from node_modules to see the build logs of Vercel. And removed a component from my code. So during deployment on Vercel of ERROR Code, it has shown me the errors indeed. And when I undo my changes and again started to deploy, that works like a charm. Not sure what was the reason still. Might be Vercel got some bugs at that period of time in my account. LOL
Removed one of the dependency from package.json
Then Deployed again to Vercel. Then it has shown me some error that makes sense.
Screenshot is attached.
Errors shown in Vercel
After fixing that error, it worked !
Background:
We are have noticed that in production and in some staging environments, that the onClick handlers on a SSR page are failing to be called. By comparing working and non-working (npm run start) deployments we've determined that the non-working deployments are not calling the render() methods of the components on the page immediately after page reload, whereas the working deployments do. We are assuming that means that hydration is not occurring.
Development deployments (npm run dev) work flawlessly
We are seeing no browser-side console messages.
We are seeing no server side warnings
The npm build --debug suggests only to upgrade to the new ChunkGraph API. Otherwise no warnings.
We are now wondering...
QQ:
Short of starting with a fresh create-next-app again and gradually evolving the app towards our current implementation until it breaks, what are the suggested ways of determining the root cause(s) of a problem like this?
For some reason, my hydration issues only showed up in production, and only on the server. Running next build and next start didn't show any errors. Since user3624334's answer didn't really help me, I figured I'd reply with how I ended up finding the culprit.
Since I deploy to Vercel, and use Vercel's review apps, I ended up creating a Pull Request where I remove half of the page that was having the hydration issues, and continued bisecting until I found out which component was causing the issues. It took me three tries to find the component.
Once I had the component it was quite easy to figure out what was causing the issues (for me the hydration issue was caused by a timestamp being rendered in different ways).
This also explains why user3624334's answer didn't work for me, since I opened the page straight after building, at which point the time was still the same as during next build!
One challenge here is that it appears that when SSR page hydration fails in next.js, a warning message will not necessarily be generated.
I hope someone has a better suggestion than the following, but I will offer this human script that worked several times for me:
git checkout -b "hydration-problem-02" to create a branch for this effort
add a console.log('render called') to the top of the render() method of pages/_app.js. This should produce a message on each hydrate, making it easy to notice when the page does/does-not hydrate.
Then repeatedly:
simplify a significant portion of the app.
npm run dev to run in dev mode
confirm that your change did not break the app in unexpected ways
rm .next/ node_modules/ package-lock.json
npm install && npm run build && npm run start
confirm that the above command completed successfully
refresh the page in the browser
check for the message in the browser console.
git commit -am "msg that indicates what you changed and if it helped"
if the problem remains, repeat these steps
Doing the commits above should help make the process more fool-proof. Ex. If you think you might have forgotten to test in production mode, you can go back to that commit and retest to confirm you did not mis-test.
Once you have a working and non-working commit of the app, you can do a binary search for the error, either adding or removing smaller and smaller pieces of function.
The root causes of quiet hydration failures I've seen include:
two modules that mutually import each other
an import '..../bootstrap.js' in a component rather than _app.js
Just recently I am starting to experience an issue when trying to deploy my functions through Firebase. When running the command firebase deploy --only functions one of my four functions deploy correctly, but the other three throw this error in the firebase-tools console:
Deployment error.
Build failed: Build error details not available. Please check the logs at url to my GCP logs
When I look at the logs the error message that shows for each function that is failing is:
ERROR: build step 3 "us.gcr.io/fn-img/buildpacks/nodejs10/builder:nodejs10_20200922_20_RC00" failed: step exited with non-zero status: 46
I have searched and googled for quite a while now and I cannot find anyone reporting the same error code, having the same non-zero status returned, or having functions fail on step 3. There doesn't seem to be much for documentation on troubleshooting the errors either.
Hopefully someone here can point me in the right direction.
Thanks!
The error seems to be caused by a missing (or not accessible) file in the history/restore functionality of Firebase. Don't know the reason for this.
A workaround that worked for me was:
Remove the functions in the web console
Deploy normally > firebase deploy
Answering since I can't comment in response to xaphod, Markus's fix worked for me, thank you mate! Deleting the function means it creates a new one rather than trying to restore it, which is the failed step 3 in the build process.
In response to xaphod, I did setup a lifecycle rule for old files in the artifacts bucket, it was by far my biggest bucket in storage and was pushing the free tier limit. I set it up a bit short probably, delete 5 days after last updated time, but surprisingly didn't run into this issue until today (possibly because I usually just deploy them all whenever I update one). I removed the lifecycle rule for now, although I'd rather it not run up the storage space again, I might end up contacting support to see if there's a way to lifecycle them without deleting the most recent, not sure why it doesn't just clear out the old images on updates though.
Found the post that suggested setting up lifecycle rules, firebase storage artifacts is huge, and did some testing as well, deleted all the files in us.artifacts and deployed all my functions (got errors on all them, only deleted the function that I needed to update earlier and deployed it by itself). Doesn't cause any errors, just takes a bit longer, so I think the issue can be fixed by deleting them all or deleting nothing, probably going to set the lifecycle rule to delete immediately and deal with the increased deploy time, rather than the cost of firebase storage.
Update: The problems passed for a while, but then returned with a vengeance since yesterday. Deploys now really take forever and always fail with Server Error. connect ETIMEDOUT or Upload Error: Cannot read property 'response' of undefined or something else.
After experimenting with connecting via a US location using HideMyAss I found that that completely resolved my issues though! Note that this issue not only occurs when deploying from our office in Amsterdam, but also from our office in Rotterdam. In the meanwhile I have also heard from more people experiencing issues with other Google services.
I have replied to the related Firebase Support email with this information and hope they will look into it. In the meanwhile I guess I'll have to keep on using HideMyAss..
--
Deploying Functions has been taking increasingly longer times after adding more of them. Occasionally at first, but recently for some periods every time I try to $ firebase deploy --only functions one of the functions being deployed at random fails with:
⚠ functions[foo]: Deploy Error: Failure in the
execution environment
When I try again an hour or so later it deploys without a problem (still takes 2 minutes to deploy which seems a little slow).
Perhaps the deploy process is timing out; it always fails after a long time, never quickly.
Perhaps my location outside of America is resulting in latency related issues in the deployment process? Doesn't seem very likely though..
I'm also looking into Firebase Functions logging "Function execution took 60002 ms, finished with status: 'timeout'" and other performance issues, so I wonder if these are all related.
PS: I also reported this to https://firebase.google.com/support/ but the last report I made there is still un-answered after 15 days, so I'm going to go ahead and post it here as well. I included a firebase-debug.log with that report, but rather not publicly sharing that here (not sure if there any tokens in there etc).
Also having issues
I have been having similar issues today and I am deploying from the United States.
For example a function will fail showing these two errors:
Deploy Error: Failure in the execution environment
Error: Functions did not deploy properly.
I found that once when this happened it was because my internet connection was dropping and then totally dropped during a firebase deploy.
Then again it happened because I was trying to deploy at the exact same time that at lot of my cloud functions happened to kick off doing stuff on their servers.
Once the cloud function had failed, then even when my internet resumed and my functions weren't busily running, it would not let me redeploy to get the jammed function running again. No matter what the broken copy of it was stuck in their servers with the tag:
Failure in execution environment
My Solution
I found that once you have a problem like that you can actually rename the function. On the next deploy it will effectively delete the old broken function and reload it as a working copy with the new name.
I would think if your project required it to have the same name you could do this process one more time, deleting the copy with the new name and restoring the copy with the name you needed. Or even block commenting out the function to delete it on the first deploy and then uncommenting it on another deploy to reinstall it.
How that helps you
I'm hoping that if you are still having issues from the Netherlands it is helpful to know that it may be a slow connection issue or a busy server issue. As I have found these two things to cause problems for me from within the States.
Also my solution of deleting and redeploying the function might help to speed up a deploy if the issue is with a copy that is on their servers. It would be interesting to know if that helps because even though the function looks ok on the server maybe it had issues during the previous deploy that are jamming up future deploys.
Sorry for the late response to this, hopefully you are no longer having these issues with firebase (I hate firebase btw, always issues like this).
I have deployed an angular-fullstack (Yeoman generator) app to Openshift successfully, however, when I make changes and try to push the changes to Openshift via the command "grunt buildcontrol:openshift" I get the following error...
Unable to process task.
Warning: Required config property "buildcontrol.openshift" missing. Use --force to continue.
Has anyone come across this error and found a solution? If I try and force the task it does not do anything additional. I'm not quite sure what I need to be looking for, or adding.