We have a production environment (cluster) where there are two physical servers and 3 (three) plone/zope instances running inside each one.
We scheduled a job (with apscheduler) that needs to run only in a unique instance, but is executing by all 6 (six) instances.
To solve this, I think I need to verify if the job is running in the server1 and if it is a instance that listens on a specific port.
So, how to get programmaticly informations about a zope/plone instance?
Related
Trying to split out Airflow processes onto 2 servers. Server A, which has been already running in standalone mode with everything on it, has the DAGs and I'd like to set it as the worker in the new setup with an additional server.
Server B is the new server which would host the metadata database on MySQL.
Can I have Server A run LocalExecutor, or would I have to use CeleryExecutor? Would airflow scheduler has to run on the server that has the DAGs right? Or does it have to run on every server in a cluster? Confused as to what dependencies there are between the processes
This article does an excellent job demonstrating how to cluster Airflow onto multiple servers.
Multi-Node (Cluster) Airflow Setup
A more formal setup for Apache Airflow is to distribute the daemons across multiple machines as a cluster.
Benefits
Higher Availability
If one of the worker nodes were to go down or be purposely taken offline, the cluster would still be operational and tasks would still be executed.
Distributed Processing
If you have a workflow with several memory intensive tasks, then the tasks will be better distributed to allow for higher utilizaiton of data across the cluster and provide faster execution of the tasks.
Scaling Workers
Horizontally
You can scale the cluster horizontally and distribute the processing by adding more executor nodes to the cluster and allowing those new nodes to take load off the existing nodes. Since workers don’t need to register with any central authority to start processing tasks, the machine can be turned on and off without any downtime to the cluster.
Vertically
You can scale the cluster vertically by increasing the number of celeryd daemons running on each node. This can be done by increasing the value in the ‘celeryd_concurrency’ config in the {AIRFLOW_HOME}/airflow.cfg file.
Example:
celeryd_concurrency = 30
You may need to increase the size of the instances in order to support a larger number of celeryd processes. This will depend on the memory and cpu intensity of the tasks you’re running on the cluster.
Scaling Master Nodes
You can also add more Master Nodes to your cluster to scale out the services that are running on the Master Nodes. This will mainly allow you to scale out the Web Server Daemon incase there are too many HTTP requests coming for one machine to handle or if you want to provide Higher Availability for that service.
One thing to note is that there can only be one Scheduler instance running at a time. If you have multiple Schedulers running, there is a possibility that multiple instances of a single task will be scheduled. This could cause some major problems with your Workflow and cause duplicate data to show up in the final table if you were running some sort of ETL process.
If you would like, the Scheduler daemon may also be setup to run on its own dedicated Master Node.
Apache Airflow Cluster Setup Steps
Pre-Requisites
The following nodes are available with the given host names:
master1 - Will have the role(s): Web Server, Scheduler
master2 - Will have the role(s): Web Server
worker1 - Will have the role(s): Worker
worker2 - Will have the role(s): Worker
A Queuing Service is Running. (RabbitMQ, AWS SQS, etc)
You can install RabbitMQ by following these instructions: Installing RabbitMQ
If you’re using RabbitMQ, it is recommended that it is also setup to be a cluster for High Availability. Setup a Load Balancer to proxy requests to the RabbitMQ instances.
Additional Documentation
Documentation: https://airflow.incubator.apache.org/
Install Documentation: https://airflow.incubator.apache.org/installation.html
GitHub Repo: https://github.com/apache/incubator-airflow
All airflow processes need to have the same contents in their airflow_home folder. This includes configuration and dags. If you only want server B to run your MySQL database, you do not need to worry about any airflow specifics. Simply install the database on server B and change your airflow.cfg's sql_alchemy_conn parameter to point to your database on Server B and run airflow initdb from Server A.
If you also want to run airflow processes on server B, you would have to look into scaling using the CeleryExecutor.
Is there a possibility using the command --alarm-action 'log: //' to run any script or create a VM instances on OpenStack, for example:
Can I do something like this
$ ceilometer alarm-threshold-create --name cpu_high/\ --description 'instance running hot' --meter-name cpu_util --threshold 70.0 --comparison-operator gt --statistic avg --period 600 --evaluation-periods 3 --alarm-action './script.sh' --query resource_id=INSTANCE_ID
where --alarm-action './script.sh' launches script.sh
It's not possible for a Ceilometer action to run a script.
The OpenStack APIs have generally been designed under the assumption that the person running the client commands (a) is running them remotely, rather than on the servers themselves, and (b) is not an administrator of the system. In particular (b) means that permitting you to run arbitrary scripts on a server would be a terrible security problem, because you would first need a way to get a script installed on the server and then there would need to be a way to prevent you from trying to run, say, /sbin/reboot.
For this reason, the ceilometer action needs to be URL. You could set up a simple web server that would receive the signal from ceilometer and execute a script in response.
If you deploy resources using Heat, you can set up autoscaling groups and have ceilometer alarms trigger an autoscaling action (creating new servers or removing servers, for example).
I have 2 windows machines and one Unix machine. On Unix machine autosys JILS are defined.
I have one cleanup job, that I want to run on both the windows machine # same time every day.
I don’t want to create 2 different jobs (JIL’s). In JIL can I add two machine names like machine: WindowMachine1, WindowMachine2 , is this correct or is there any other way to do this?
Defining multiple machines as a single virtual entity will cause AutoSys to treat them as a pool and load balance when sending them work, causing your cleanup job to only run on one of the two machines.
To have a job run on both machines simultaneously, you will need two separate CMD jobs, one sent to each machine.
We're using nServiceBus in our development environment, where we have a frontend publishing messages to a service (subscriber). Life is good.
FrontendWebServer -> MiddlewareServer
In our production environment, we'll be running two frontends and two middleware servers for failover.
FrontendWebServer -> LoadBalancer(F5) -> MiddlewareServer
FrontendWebServer -> LoadBalancer(F5) -> MiddlewareServer
This works well for URLs, but because we need to use machine names for MSMQ we're stuck.
We don't want to specify a physical middleware machine name in each frontend config (because it makes managing configs harder, and if one middleware server goes down, it will also stop messages its particular frontend).
We tried to use the nServiceBus distributor (installed on each frontend), but it seems that a subscriber can only listen to one distributor.
Any ideas how we can get around this problem without using separate configs?
I would push the F5 up in front of the web servers to balance that load. For the cluster, just reference the Clustered Server name and services not the individual machines. For example if you have Node1 and Node2 you may call the Cluster NSBNode or something like that.
If you make that cluster the Distributor, you can then add multiple Worker nodes behind it to load balance further. Again in this case also make reference to the Cluster queue names(queue#ClusterName).
I am aware that nodes can be started from the shell. What I am looking for is a way to start a remote node from within a module. I have searched, but have been able to find nothing.
Any help is appreciated.
There's a pool(3) facility:
pool can be used to run a set of
Erlang nodes as a pool of
computational processors. It is
organized as a master and a set of
slave nodes..
pool:start/1,2 starts a new pool.
The file .hosts.erlang is read to
find host names where the pool nodes
can be started. The slave nodes are
started with slave:start/2,3,
passing along Name and, if provided,
Args. Name is used as the first
part of the node names, Args is used
to specify command line arguments.
With pool you get load distribution facility for free.
Master node may be started this way:
erl -sname poolmaster -rsh ssh
Key -rsh here specifies an alternative to rsh for starting a slave node on a remote host. We used SSH here. Make sure your box have working SSH keys, and you can authenticate to the remote hosts using these keys.
If there are no hosts in the file .hosts.erlang, then no slave nodes are started, and you can use slave:start/2,3 to start slave nodes manually passing arguments if needed.
You could, for example start a remote node:
Arg = "-mnesia_dir " ++ M,
slave:start(H, Name, Arg).
Ensure epmd(1) is up and running on the remote boxes in order to start Erlang nodes.
Hope that helps.
A bit more low level that pool is the slave(3) module. Pool builds upon the functionality in slave.
Use slave:start to start a new slave.
You should probably also specify -rsh ssh on the command-line.
So use pool if you need the kind of functionality it offers, if you need something different you can build it yourself out of slave.