I am new in the field, I would like to know how can I use keycloak with apache airflow,
I saw some tutorial that talks about it but are not detailed,
I would like your help to help me understand how can I work with keycloak and apache airflow
What I tried and studied
https://awslife.medium.com/airflow-authentication-with-rbac-and-keycloak-2c34d2012059
https://aksharau.substack.com/p/configuring-airflow-with-keycloak
https://github.com/apache/airflow/issues/14002
Related
Hello guys and I hope you're having a great day. I have a question about using Openstack API in Python.
I'm using python-novaclient for getting server details and flavor details. And I want to get the volume details too but I don't know how to do it, I've tried to collect volume details but it failed somehow and I need to ask you guys if you have any idea.
This information is what I want to get:
volume_id, attached to (w/c volume), name, status and volume_type (CEPH or LVM)
I used python-cinderclient, but I only got the volume_id.
Here's the code:
volumes = cinder.volumes.list()
Can someone help me to get the other data? Other than running Openstack command-line in the server, I just need some Python module to get these data.
Thanks in advance.
I've finally figured it out, and I'm going to answer this for anyone who is interested in Openstack SDK or other Python API for Openstack.
First, for authentication you need to use Keystone API, the documentation is all over the internet so no need to worry, you could just oversee in your Openstack for credentials needed. And for my question, I use the function get_volume from Connection class. Please see the documentation
for this. You can read other documentation as well on the internet.
So, here is the example of how to get volumes details:
vol = conn.get_volume(volume_id)
print(vol)
In Airflow http (and other) connections can be defined as environment variables. However, it is hard to use an https schema for these connections.
Such a connection could be:
export AIRFLOW_CONN_MY_HTTP_CONN=http://example.com
However, defining a secure connection is not possible:
export AIRFLOW_CONN_MY_HTTP_CONN=https://example.com
Because Airflow strips the scheme (https) and in the final connection object the url gets http as scheme.
It turns out that there is a possibility to use https by defining the connection like this:
export AIRFLOW_CONN_MY_HTTP_CONN=https://example.com/https
The second https is called schema in the airflow code (like in DSN's e.g. postgresql://user:passw#host/schema). This schema is then used as the scheme in the construction of the final url in the connection object.
I am wondering if this is by design, or just an infortunate mixup of scheme and schema.
For those who land in this question in the future, I confirm that #jjmurre 's answer works well for 2.1.3 .
In this case we need URI-encoded string.
export AIRFLOW_CONN_SLACK='http://https%3a%2f%2fhooks.slack.com%2fservices%2f...'
See this post for more details.
Hope this can save other fellows an hour which I've spent on investigating.
You should use Connections and then you can specify schema.
This is what worked for me using bitnami airflow:
.env
MY_SERVER=my-conn-type://xxx.com:443/https
docker-compose.yml
environment:
- AIRFLOW_CONN_MY_SERVER=${MY_SERVER}
I've successfully installed Prometheus in Google Container Engine and I have these targets up:
kubernetes-apiservers
kubernetes-cadvisor
kubernetes-nodes
Now I would like to scrape Nginx stats from each of the Docker containers inside this Kubernetes cluster (which seems like a sensible thing to do).
But how can I make Prometheus automatically pull the metrics from all the Nginx instances running in all of the Docker containers?
From my research so far, the answer involves kubernetes_sd_config but I simply could not find enough documentation on how to put the pieces together.
Thank you!
Edit: This is not about exposing the Nginx stats. This is just about scraping any stats that are exposed by all Docker containers.
You are correct that you need to use the kubernetes_sd_config directive. Before continuing let me just say that what you should be asking is "Automatically scape all pods from Kubernetes". This is because a pod is considered the lowest unit of scale in Kubernetes. Regardless it is clear what you are trying to do.
So the kubernetes_sd_config can be used to discover all pods with a given tag like so:
- job_name: 'some-app'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: python-app
action: keep
The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at app pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').
Hope that helps. You can follow blog post here for more detail. Also for more information about kubernetes_sd_config check out docs here.
Note: it is worth mentioning that kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.
I'm trying to configure the file cygnus.conf, but, I don't know the FQDM/IP of the Namenodes, hive sever and CKAN API endpoint.
I was search in the catalogue and forge, and I can't see anything about that.
thank you, and the best reggarts.
For the Namenode and the Hive server just use the same endpoint you used to setup your Cosmos account: http://cosmos.lab.fi-ware.org. Regarding CKAN, I'm not sure there is a public CKAN instance running in FI-LAB. Let me check it.
I'm running a .net server on Amazon's Elastic Beanstalk and when I try to instantiate AmazonDynamoDBClient it fails. According to the information I've been following from these two pages:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.iam.roles.apps.html
http://docs.aws.amazon.com/AWSSdkDocsNET/latest/DeveloperGuide/net-dg-roles.html
It should retrieve the credentials from the IAM role assigned to the EC2 instance.
It has the DynamoDBFullAccess template as a policy.
If I supply the credentials in the code itself through the web.config file it works, but only when debugging locally and not when deployed to elastic beanstalk.
Here is the code that breaks it:
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
or
AmazonDynamoDBClient client = new AmazonDynamoDBClient(new InstanceProfileAWSCredentials());
Neither works. And I can't seem to retrieve crash logs from elastic beanstalk, so it is hard to debug. I'm pretty sure that I'm following the descriptions in those two links to the letter and am confused.
I've been trying to solve this for the better part of a day and would really appreciate some help. If there is any information missing, please let me know.
Thank you.
If you go to aws console and select ec2 view instances you will see the ec2 instance for your elastic beanstalk. The name of that instance will be either "Default-Environment" or whatever name you chose for your elastic beanstalk environment. Then you can ssh to that ec2 instance and view logs. For example my tomcat logs are stored at /var/log/tomcat7
You can also scp your credential file to your ec2 host just for debug and run the app. It should work because you already have it working on your local machine. but I guess this approach is not recommended and may has security concern.
This should at least get you started. may be an EC2 expert can help you solve the real problem :)