any one knows how to send metrics from airflow to prometheus, I'm not finding much documents about it, I tried the airflow operator metrics on Grafana but it doesnt show any metrics and all it says no data points.
By default, Airflow doesn't have any support for Prometheus metrics. There are two ways I can think of to get metrics in Prometheus.
Enable StatsD metrics and then export it to Prometheus using statsd exporter.
Install third-party/open-source Prometheus exporter agents (ex. airflow-exporter).
If you are going with 2nd approach then the Airflow Helm Chart also provides support for that.
Edit
If you're using statsd exporter here is a good resource for Grafana Dashboard and exporter config.
This is how it worked for me -
Running airflow in docker using this doc
Added this configuration inside the docker-compose file downloaded in the previous step AIRFLOW__SCHEDULER__STATSD_ON: 'true'
AIRFLOW__SCHEDULER__STATSD_HOST: statsd-exporter
AIRFLOW__SCHEDULER__STATSD_PORT: 9125
AIRFLOW__SCHEDULER__STATSD_PREFIX: airflow
Under environment section
Now run the statsd_export
docker run -d -p 9102:9102 -p 9125:9125 -p 9125:9125/udp \ -v $PWD/statsd_mapping.yml:/tmp/statsd_mapping.yml \ prom/statsd-exporter --statsd.mapping-config=/tmp/statsd_mapping.yml
Get the statsd_mapping.yml contents from Here
Now do docker-compose up to run the airflow and try to run some worflow and you should see logs at http://localhost:9102/metrics
If you installed your Airflow with statsd support:
pip install 'apache-airflow[statsd]'
you can expose Airflow statsd metrics in the scheduler section of your airflow.cfg file, something like this:
[scheduler]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
Then, you can install a tool called statsd_exporter, that captures statsd-format metrics and converts them to Prometheus-format, making them available at the /metrics endpoint for Prometheus to scrape.
There is a docker image available on DockerHub called astronomerinc/ap-statsd-exporter that already maps Airflow statsd metrics to Prometheus metrics.
References:
https://airflow.apache.org/docs/stable/metrics.html
https://github.com/prometheus/statsd_exporter
https://hub.docker.com/r/astronomerinc/ap-statsd-exporter/tags
Related
When my application is up, telegraf works fine and collects data related to jolokia since my application opens the port 11722 that telegraf uses to get the metrics. But then, when my application is down, telegraf starts to get errors since it can't connect to Jolokia. My telegraf version is 1.5.3 and this is a Production environment, so I don't have much flexibility to change the version. Is there a way to collect the jolokia metrics just when my application is up and running?
I've tried to create a script to check if jolokia was running and use with a tag that then I could use with my agent, but this didn't work:
[[inputs.exec]]
commands = ["sh /local/1/home/svcegctp/telegraf/inputs/scripts/check_jolokia.sh"]
timeout = "1s"
data_format = "influx"
name_override = "jvm_status"
[inputs.exec.tags]
running = "true"
(...)
[[inputs.jolokia2_agent]]
# Add agents URLs to query
urls = ["http://localhost:11722/jolokia"]
[inputs.jolokia2_agent.tags]
running = "true"
This is my script:
check_jolokia.sh
#!/bin/bash
if curl -s -u <username>:<password> http://localhost:11722/jolokia/version >/dev/null 2>&1; then
echo "jvm_status running=true"
else
echo "jvm_status running=false"
fi
Airflow Version: 2.2.4
Airflow running in EKS
Issue: Logs not showing in the UI while tasks running
The issue with the logs is that airflow is only writing the logs to the log file rather than standard out as well. This is what's preventing us from being able to see the logs in the web UI while the task is running.
When i get into the pod , i do see log inside the pod
Is there any solution to finding the setting or configuration needed to output to both?
I log as below
kubectl logs detaskdate0.3d55e5ba89ca4ad491bb3e1cadfdaaec -n airflow
Added new context arn:aws:eks:us-west-2:XXXXXXXX:cluster/us-west-2-airflow-cluster to /home/airflow/.kube/config
[2022-05-20 19:56:43,529] {dagbag.py:500} INFO - Filling up the DagBag from /opt/airflow/dags/tss/dq_tss_mod_date_dag.py
I'm trying to run dagster using celery-k8s and using the examples/celery-k8s as a start. upon running the pipeline from playground I get
Initialization of resources [s3, io_manager] failed.
botocore.exceptions.NoCredentialsError: Unable to locate credentials
I have configured aws credentials in env variables as mentioned in the document
deployments:
- name: "user-code-deployment-test"
image:
repository: "somasays/dagster-usercode-example"
tag: "0.5"
pullPolicy: Always
dagsterApiGrpcArgs:
- "-f"
- "/workspace/repo.py"
port: 3030
env:
AWS_ACCESS_KEY_ID: AAAAAAAAAAAAAAAAAAAAAAAAA
AWS_SECRET_ACCESS_KEY: qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
AWS_DEFAULT_REGION: eu-central-1
and I can also see these values are set in the env variables of the pod and can also access the s3 location after pip install awscli and aws s3 ls see the screenshot below the job pod however throws Unable to locate credentials
Please help
The deployment configuration applies to the user code servers. Meanwhile the celery executor runs your pipeline code in separate kubernetes jobs. To provide your secrets there, you will want to configure the env_secrets field of the celery-k8s executor in your pipeline run config.
See https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/dagster_k8s/job.py#L321-L327 for details on the config.
The airflow documentation states:
Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are available at /api/experimental/. Please note that we expect the endpoint definitions to change.
https://airflow.apache.org/api.html#experimental-rest-api
However it doesn't state in which version the API appears. We are running Airflow v1.8.0
But whenever I browse to /api/ or /api/experimental/ I get a 404 and the spinning circles.
I tried curling the same URLs but that only confirmed the same, /api/ gives me a 404:
$ curl -I -L -s http://${AIRFLOWIP}:8080/admin/ | grep HTTP
HTTP/1.1 200 OK
$ curl -I -L -s http://${AIRFLOWIP}:8080/api/ | grep HTTP
HTTP/1.1 404 NOT FOUND
We have this setting in airflow.cfg:
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
I don't know whether the API is only available in a later version of Airflow or if we have stood it up incorrectly.
Can someone let me know in which version of airflow we can find the experimental API?
The first experimental endpoints were added in 1.8.0, with a few more endpoints added in following releases. There is no endpoint for the root paths of /api/ and /api/experimental/ on any version, so those curls are not expected to work. However, there is a /api/experimental/test/ endpoint which you can hit to confirm the API is available.
If you're going to be using the experimental API, I think the code is the best reference at the moment.
The Airflow API is no more at the experimental phase.
Stable version here Airflow REST API.
We are trying to setup a Single Node Kaa server(version 0.10.0) in an Ubuntu 16.04 machine.
Followed the documentation given here
We were unable to connect to the admin UI after starting the kaa node service.
On investigating further we could see that the Mongodb and zookeeper services were not started. So we manually started those services. After that we were able to connect to Kaa admin UI. Do we need any additional steps to get these service running on kaa-node start ?
I setup kaaproject with the guide for my Ubuntu 16.04.1 LTS VM and Zookeeper was not running by default on my server also, so I had to install the deamon (which starts zookeeper also on startup):
sudo apt-get install zookeeperd
Check if zookeeper is running:
netstat -ntlp | grep 2181
This should result in an output like this:
With mongodb I had the problem, that there was not enough space available for the journal files. I fixed this by increasing the available disk space + setting smallfiles=true in the /etc/mongod.conf
Probably you have some troubles with configurations for services. Check if auto-startup is enabled for MongoDB / Zookeeper by the next command:
$ systemctl is-enabled ${service-name}
if you see this:
$ disabled
then auto-startup is disabled for specified service and you should try next in order to enable it:
$ systemctl enable ${service-name}