We have a requirement where we need to send airflow metrics to datadog. I tried to follow the steps mentioned here
https://docs.datadoghq.com/integrations/airflow/?tab=host
Likewise, I included statsD in airflow installation and updated the airflow configuration file (Steps 1 and 2)
After this point, I am not able to figure out how to send my metrics to datadog. Do I follow the Host configurations or containarized configurations? For the Host configurations, we have to update the datadog.yaml file which is not in our repo and for containerized version, they have specified how to do it for Kubernetics but we don't use Kubernetics.
We are using airflow by creating a docker build and running it over on Amazon ECS. We also have a datadog agent running parallely in the same task (not part of our repo). However I am not able to figure out what configurations I need to make in order to send the StatsD metrics to datadog. Please let me know if anyone has any answer.
Related
We are using Jfrog artifactory selfhosted instance with license for our project and many customers are using for thir package and binary management.
Since this is hosted i our private selfhosted environments over linux platform, regularly we may need to have a maintenance window atleast 2 times in a month to apply patches to our servers and all. So we are considering for high availability for our currently running Jfrog instance which should resolve this downtime during the maintenance. Also we are looking for some better managemental scenarios as below and couldnt find any helpful guidance from the docs.
How the Jfrog server insance service status can be monitored along with auto restart if the service is in failed state after the server reboot.
Is there any way to set and populate a notification messsage to the sustomers regarding the sceduled maintenance.
How can we enable the high availability for JFrog Artifactory and Xray. ?
Here are some of the workaround you can follow to mitigate the situation
To monitor the health of the JFrog services you can use the below rest API
curl -u : -XGET
http://<Art_IP>:8046/router/api/v1/topology/health -H 'Content-Type:
application/json'
If you are looking for a more lightweight check you can use
curl -u: -XGET
http://<Art_IP>:8081/artifactory/api/system/ping
By default, the systemctl scripts check for the availability of the services and restart them when they see a failure. The same applies to the system restart as well.
There is no option for a pop-up message however, you can set a custom message as a banner in the Artifactory. Navigate to Administration -> General settings -> Customer message. Here is the wiki link
When you add another node to the mix, Artifactory/Xray becomes a cluster to balance the load (or as a failover) however it is the responsibility of the load balancer/Reverse proxy to manage the traffic between the cluster nodes according to the availability of the backend node.
I'm a newbie at using Airflow. I went through many Airflow tutorials, and I can say that all are about development environments using a docker-compose file or files. I'm facing a problem at work setting up a production environment properly. My goal is to have a cluster composed of 3 EC2 virtual machines. Can anyone share best practices for installing Airflow on that cluster?
I went through many tutorials on the internet.
Airflow has 4 main components:
Webserver: stateless service which expose the UI and the REST API of Airflow
Scheduler: stateless service which processes the dags and runs them and it's the main component
Worker: stateless service which execute the tasks
Metadata: the database of Airflow where the state is stored, and it manages the communication between the 3 other components
And Airflow has 4 main executors:
LocalExecutor: the scheduler runs tasks by itself by spawning a process for each task, and it works in a single host -> not suitable for your need
CeleryExecutor: the most used scheduler, you can create one or multiple scheduler (for HA), and a group of celery workers to run the tasks, you can scale it on different nodes
DaskExecutor: similar to CeleryExecutor but it uses Dask instead of Celery, not much used, and there is no many resources around it
KubernetesExecutor: it runs each task in a K8S pod, and since it's based on Kubernetes, it's very scalable, but it has some drawbacks.
For you use case, I recommend using CeleryExecutor.
If you can use EKS instead of EC2, you can use the helm chart to install and configure the cluster. And if not, you have other options:
run the services directly on the host:
pip install apache-airflow[celery]
# run the webserver
airflow webserver
# run the scheduler
airflow scheduler
# run the worker
airflow celery worker
You can decide how many scheduler, workers and webserver you want to execute, and you can distribute them on the 3 nodes, for ex: node1(1 scheduler, 1 webserver, 1 worker), node2(1 scheduler, 2 workers), node3(1 webserver, 2 workers), and you need a DB, you can use postgres from AWS RDS, or create it on one of the nodes (not recommended).
using docker: same as the first solution, but your run containers instead of running the services directly on the host
using docker swarm: you can connect the 3 nodes to create a swarm cluster, and manage the config from one of the nodes, this gives you some feature which are not provided by the first 2 solutions, and it's similar to K8S. (doc)
For the 3 solutions, you need to create airflow.cfg file contains the configurations and the DB creds, and you should set the exeutor conf to CeleryExecutor.
I am writing this question because I couldn't find a way to change the default Hasura console port when using Hasura docker image.
The page I'm referring is this
There is no variable defined to change the Hasura default console port in the above page.
The reason I'm requesting this feature is to separate query histories of my two hasura projects. If I could manage to run these two consoles on two different ports I will be able to save the query/mutation history seperately.
I have not used Hasura image previously, but I have used docker a lot to run MySQL instances etc. What I typically do depends on whether or not I am using docker-compose or just plain docker run.
If using docker compose you can specify a port mapping for each container, e.g. map 9695 -> 9005 would look like so:
hasura:
image: hasura
ports:
- 9005:9695
or if using docker run following these docs e.g.
docker run --expose 9695:9005 hasura ...
Hasura's Dockerfile has graphql-engine serve as its CMD. You can override this default command by your own command and pass --server-port or any config to serve, as per the reference page.
setting the port will change the hasura api endpoint, not the hasura console
That's not entirely true. If HASURA_GRAPHQL_ENABLE_CONSOLE=true Hasura will serve the console on the /console route of the server API endpoint. However, for production, it's recommended to disable the console and run Hasura CLI locally, connected to the production instance.
I was wondering if the Airflow's scheduler and webserver Daemons could be launched on different server instances ?
And if it's possible, why not use serverless architecture for the flask web server ?
There is a lot of resources about multi nodes cluster for workers but I found nothing about splitting scheduler and webserver.
Has anyone already done this ? And what may be the difficulties I will be facing ?
I would say the minimum requirement would be that both instance should have
Read(-write) access to the same AIRFLOW_HOME directory (for accessing DAG scripts and the shared config file)
Access to the same database backend (for accessing shared metadata)
Exactly the same Airflow version (to prevent any potential incompatibilities)
Then just try it out and report back (I am really curious ;) ).
woocommerce webhooks aren't firing at all for me, even on a fresh install. I did the following:
Create a new MySQL database
Install WP from the zip file.
Set up WP.
Install Woocommerce.
Enable REST API and create a key.
Added "Coupon created" webhook, made sure it's set to active, and set it to a publicly accessible site.
When I create a coupon, the webhook does not fire, and no entry is created in the log. I tried this with orders as well and also doesn't work.
I think it's a machine configuration problem, but not sure what to change. The machine is an EC2 instance and has all ports opened in its security group policy.
Weirdest of all is that on a different EC2 instance does work, but it's a production machine and I want to have a dev server work so I can test out things. The only config differences between the production and dev machines that I can think of are the subnets and the firewall, but I don't understand why the subnet should matter and I opened all the firewall ports on the dev machine.
what Linux distributions are you running for prod and dev?
CentOS with SELinux enabled with not allow HTTPD scripts and modules to connect to network by default.
setsebool -P httpd_can_network_connect on
If above is not valid, please identify network problems by trying connecting to AWS RDS via SSH CLI. If you can open a connection via SSH CLI, the problem will be with your application. If you can't, it will be network problem. First thing to check in that case is AWS RDS security group. For testing you can open 3306 to public.
Let me know how it goes.