For example, if YARN was stopped by some unknown reasons when running query1. At this time, query2 allocates resources to run. YARN restarts after a few minutes. Will query1 and query2 both run normally?
Once the query succeeds in obtaining the resource(called vseg in HAWQ), the query will run no matter the status of YARN is down.
HAWQ Resource Manager will periodically contact Yarn.
If the down of Yarn has not been found by HAWQ Resource Manager, and HAWQ Resource Manager still contains the resources, both query1 and query2 can run successfully. But if the down of Yarn is found by HAWQ Resource Manager, following queries will all hang until Yarn becomes back.
Related
I recently created a new production cloud composer environment and migrated my existing dags to it and have been getting the error
INFO - Task exited with return code Negsignal.SIGSEGV
on some tasks that work perfectly fine in my dev environment (also on cloud composer). The production environment is loaded from a snapshot of the dev environment so they should be identical. Is there anything configuration wise that might not have come over in the snapshot that I can look into so I can try and get this resolved?
For added context, I have upgraded the scheduler, web server and worker specs in the prod environment to be more than the dev environment and this is still causing the same issue.
Are the instances the same size? This message means that some tasks run out of resources so you have either to scale up or modify your code so it is more frugal and you don't run out of resources.
Also have you checked how the dags are scheduled? Maybe you have many dags running at the same time and they use up all your resources.
I'm doing firebase & react project.
However everytime I do git push origin master I get this email.
**
Run failed: Deploy to Firebase Hosting on merge - master
**
As I click the email and it says
**
Deploy to Firebase Hosting on merge: All jobs have failed
**
Though I get this email, it still works fine. My github gets updated fine. Also if I do firebase deploy, my website gets updated fine. Everything works just fine but getting that email.
Is that a problem?
https://firebase.google.com/docs/hosting/test-preview-deploy
I guess you have to read this one.
I know posting links ain't cool, but this is an official documentation and I believe it will stay up to date forever.
Or this one:
https://firebase.google.com/docs/hosting/github-integration
I am not really sure what your issue is.
I had a similar issue.
Firebase Hosting Setup - "Set up the workflow to run a build script before every deploy? (y/n)
I wrote npm ci && npm run generate and now it works.
npm ci is required for since your node_modules is missing in repo.
In my case I am using nuxt.js and my script is npm run generate instead or more common npm run build
I am using Cloud Composer to schedule multiple DAGs. These DAGs are built dynamically using this method and they use custom plugins.
I would like to know how to proceed when adding / modifying a plugin which concerns all DAGs (let's say it adds a new task to each DAGs)?
Do we need to pause all the running DAGs when doing so?
What I did so far when adding /modifying a plugin, is :
Upload the plugins into the plugins bucket of the Composer cluster (using gcloud composer command)
Do a dummy update in the Airflow config -> add a dummy value to the airflow.cfg (using gcloud composer commands)
I did that to force the DAGs to pause, and once the update is finished then the DAGs are resumed but with the new plugins and hence the new tasks (or if its not in this dagrun then it's the next one). Is it useless?
Thanks if you can help.
As explained in the architecture diagram, the Airflow webserver where you view your DAG and plugin code runs in a Google-managed tenant project, whereas the Airflow workers which actually run your DAG and plugin code are directly in your project.
When a DAG/Plugin is placed in the Composer bucket, the Airflow webserver(which falls under tenant project) validates the code and updates any new scheduling changes in the Airflow database.
At the same time, the Airflow scheduler (in your project) asks the Airflow database for the next DAG to run and notifies the Airflow workers to perform the scheduled work. The Airflow workers (in your project) then grab the DAG/Plugin code from the Composer bucket and compile them in order to run that specific task.
Thus, any updates made to your DAGs/Plugin code is read separately by the Airflow webserver and Airflow workers, at different times.
If you do not see your new code in the Airflow webserver, it should still be picked up by the Workers when they grab the code fresh on the new task run.
Therefore you shouldn't have to restart Composer for the workers to pick up the changes.
You cannot force a worker to grab and re-compile the new code mid task execution.
There are two ways to refresh the Airflow Webserver to see the Plugin code changes if it is not updating:
Set the reload_on_plugin_change property to True in the [webserver] section via the ‘AIRFLOW CONFIGURATIONS OVERRIDE’ tab in the Console.
OR, you can specifically add/remove/update a PYPI Package via the ‘PYPI PACKAGES’ Console tab. Non PYPI Package changes will not trigger the web server restart. Note this will also initiate an entire Composer environment restart which may take ~20 minutes.
Standard procedure to start with
meteor app -p 3000&
This works, except when I close the ssh connection, application is no more running? I have no clue why this is happening.
Awakening Edit:
I use PM2 for traditional node apps, but if I have to setup process monitors, logs, database all by myself. I could just go back to reactjs and socketio and rock it with node.
Consider use mup package from Arunoda to easily deploy and run your app in production.
You could launch meteor with nohup (no hang-up) which serves this purpose.
nohup meteor --production &
But it's not a good idea to run a site in production with meteor anyway.
What should I do to run meteor forever ?
You can use forever, a Node.js tool designed to run node apps as services.
I also want to point that forever is getting old and I've heard of better recent alternatives but it seems to still be a pretty common tool. You could also use systemd which integrates better with the UNIX service ecosystem but that's anoter story.
But first, you'll have to "demeteorize" your meteor application like this :
cd my-project-meteor
meteor bundle --directory ../my-project-node
this is going to take some time
cd ../my-project-node/programs/server
npm install
this is going to take some time too
So now you have a plain node app, that you can run with node main.js
Let me mention that it might be a good idea to use the node version used by meteor which is 0.10.29 as of meteor 0.9.1 You can install it traditionally or you could use the node version that is shipped with the meteor tool.
sudo ln -s ~/.meteor/packages/meteor-tool/1.0.27/meteor-tool-os.linux.x86_64/dev_bundle/bin/node /usr/bin/node
sudo ln -s ~/.meteor/packages/meteor-tool/1.0.27/meteor-tool-os.linux.x86_64/dev_bundle/bin/npm /usr/bin/npm
Note that this way of "installing" node + npm on your system is problematic because :
it assumes you're doing only meteor related stuff.
it is dependant on the release process of the meteor tool (you'll need to rerun these commends if the meteor tool is updated).
You can install the forever tool using npm :
-g means globally : give access to forever to all users on the system
sudo npm install -g forever
To launch your node app as a service, you can use the following command, which sets correctly some environment variables and run the app using forever :
sudo export PORT=80 MONGO_URL=mongodb://localhost/my-project-mongodb ROOT_URL=http://localhost forever start my-project-node/main.js
You can monitor it using forever stop my-project-node/main.js
Also, what's the point of using 3rd party database service like https://mongolab.com/?
When using the meteor tool, it launches a mongod process automatically for you, and the underlying node process executed by meteor representing your app connects to this mongo instance.
When we want to launch our meteor app as a node app, we have to handle the mongo stuff ourself, which kinda answer the question : why not using another service to handle it for us, they know better, right ?
Doesn't it slow down the website, because now application has to connect to their database instead of local database ?
Of course, relying on a 3rd party database service has its inconvenients, and this is one of them. Network communications will always be slower than interprocess communications taking place on localhost (this is especially true on these SSD backed cheap VPS you can find nowadays).
And how exactly do I connect to mongolab for example ?
By setting an appropriate value to the environment variable MONGO_URL, the database service provider will give you an url that corresponds to your online mongodb, this is what you need to pass to the node process in command line if you want meteor to connect to your distant database and work as usual.
If you want to launch a dedicated local mongod instance to let your application connect to it, well this is another topic but you'll have to follow these steps :
first install mongodb correctly on your server, using the reference documentation for the OS version. By correctly I mean choose the same version as meteor is using currently (2.4.9) and let it run as a service so that it will actually restart when your server reboots.
test that mongod is running by launching a client with the mongo command.
pass the correct MONGO_URL when launching your app with forever (something like mongodb://localhost/my-project-mongodb)
Understand now why meteor deploy is amazing :D
Answer copy from here
When developing with Meteor locally, one execute meteor reset locally to refresh the database.
Can one run this command on a production level app deployed on Bluemix without digging into the Mongo console?
The meteor reset command actually deletes the local mongo database files in .meteor/local. Since the database isn't stored in the application container when running in Cloud Foundry, there isn't an equivalent operation as a one liner.
Seems like your only option is to retrieve the connection credentials from your application cf env appname and then stop the application, connect with the mongo command line, and use one of the methods described in this answer to clean your data out.