I am sending mail using airflow emailOperator. Is there any way to send "reply-to" parameter also. i tried default_param={'key':'value'} but throwing exception as invalid parameter.
I read this https://github.com/apache/airflow/blob/main/airflow/operators/email.py
https://github.com/apache/airflow/blob/main/tests/operators/test_email.py
But I don't know where i am doing mistake.
version : 2.0.2
email_op = EmailOperator(
task_id='send_email',
to=destination['emails'],
subject=destination['subject'],
html_content=destination['message'],
files=[file.name],
default_param={'Reply-To': 'reply_to#example.com'},
)
email_op.execute(context)
It would be passed via dictionary custom_headers, as per the Official Source Code and Official Docs
I actually took the Reply-To directly from the official Airflow unit test.
There was a bug patched in release 2.3.0 with the send_email function to correctly pass the custom_headers. This might be your root cause...
Here is the link to the issue
Here is the pull request to resolve it
For new comers here is how you would send the custom_headers when using 2.3.0 and above.
from airflow.operators.email_operator import EmailOperator
t1 = EmailOperator(
task_id='send_email',
to="email#example.com",
subject="Airflow Example",
html_content=""" <h3>Email Test</h3> """,
cc="email#example",
custom_headers = {
'Reply-To': 'reply_to#example.com'
}
)
Related
My django webserver url accept query-parameters. For ex. "mydomain.com?emp_name=abc&dept=admin".
I want to automate this write a test using pytest-djnago to see if the url accepts the query parameters or not. Please suggest.
There are two ways to achive this.
My setup :
Django 3.2.12
django-test-curl 0.2.0
gunicorn 20.1.0
pytest 7.0.1
pytest-django 4.5.2
1) By using pytest-curl-report plugin (https://pypi.org/project/pytest-curl-report/)
I was getting below error after installing the "pytest-curl-report 0.5.4" plugin.
Error:
`pluggy._manager.PluginValidationError: Plugin 'curl-report' for hook
'pytest_runtest_makereport'
hookimpl definition: pytest_runtest_makereport(multicall, item, call)
Argument(s) {'multicall'} are declared in the hookimpl but can not be found in the hookspec`
I tired multiple install/unistall but it didnt helped.
Also I didnt found any fix this issue on google, so I skipped this one and decided to use django-test-curl plugin. See point no. 2)
2) By using django-test-curl plugin ("https://github.com/crccheck/django-test-curl")
$ pip3 install django-test-curl
Usage
from django_test_curl import CurlClient
class SimpleTest(TestCase):
def setUp(self):
self.client = CurlClient()
def test_details(self):
response = self.client.curl("""
curl http://localhost:8000/customer/details/
""")
self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.context['customers']), 5)
In our project we are using DataprocClusterCreateOperator which was under contrib from airflow.contrib.operators import dataproc_operator. It is working fine with airflow version 1.10.14.
We are in a process of upgrading to Airflow 2.1.2 wherein while testing or dags which requires spinning of DataProc Cluster we found error as airflow.exceptions.AirflowException: Invalid arguments were passed to DataprocClusterCreateOperator (task_id: <task_id>). Invalid arguments were: **kwargs: {'config_bucket': None, 'autoscale_policy': None}
I am not able to see any links for this operator support in Airflow 2 so that I can identify the new params or the changes which happened.
Please share the relevant link.
We are using google-cloud-composer version 1.17.2 having Airflow version 2.1.2.
Since Airflow 2.0, 3rd party provider (like Google in this case) operators/hooks has been moved away from Airflow core to separate providers packages. You can read more here.
Since you are using Cloud Composer, the Google providers package is already installed.
Regarding the DataprocClusterCreateOperator, it has been renamed to DataprocCreateClusterOperator and moved to airflow.providers.google.cloud.operators.dataproc so you can import it with:
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator
The accepted parameters differ from the one included in Airflow 1.x. You can find an example of usage here.
The supported parameters for the DataprocCreateClusterOperator in Airflow 2 can be found here, in the source code. The cluster configuration parameters that can be passed to the operator can be found here.
The DataprocClusterCreateOperator has been renamed as DataprocCreateClusterOperator since January 13, 2020 as per this Github commit and has been ported from airflow.contrib.operators to airflow.providers.google.cloud.operators.dataproc import path.
As given in #itroulli's answer, an example implementation of the operator can be found here.
I am trying to set two default values for all my dags. For that, I have created the file airflow_local_settings.py in my home directory of Airflow with the following code (trying to follow the example in https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#dag-level-cluster-policy):
def dag_policy(dag: DAG):
dag.catchup = False
dag.default_args['email'] = 'blabla'
However, I get the error:
Error: name 'DAG' is not defined
If I leave the code as:
def dag_policy(dag: DAG):
dag.catchup = False
dag.default_args['email'] = 'blabla'
Then the dags run with catchup=True and no email. How could I solve it? Thanks
DAG Policy was added in PR and available only in Airflow >= 2.0 since you are running 1.10.14 this feature is not available for you.
So... this morning... I got an email saying:
Our records show that you own projects with App Engine applications or
Cloud Functions that are still calling the pre-GA v0.1 and v1beta1
endpoints of the App Engine and Cloud Functions metadata server.
We’re writing to let you know that these legacy endpoints are
scheduled to be turned down on April 30, 2020. After April 30, 2020,
requests to the v0.1 and v1beta1 endpoints will no longer be
supported, and may return HTTP 404 NOT FOUND responses.
I'm only using Firebase Functions to send messages... and the email went on to identify my sendMessage function as the culprit. But I can't... for the life of me... figure out WHERE I need to update the endpoints. My sendMessage function is as follows:
exports.sendMessage = functions.database.ref('/messages/{receiverUid}/{senderUid}/{msgId}')
.onWrite(async (change, context) => {
const message = change.after.val().body;
const receiverUid = change.after.val().receiverUid;
const senderUid = change.after.val().senderUid;
const msgId = change.after.val().msgId;
if (!change.after.val()) {
return console.log('Sender ', senderUid, 'receiver ', receiverUid, 'message ', message);
}
console.log('We have a new message: ', message, 'for: ', receiverUid);
I've tried following some of the Curl suggestions from this link: https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server
...but every time I try one of them I get:
curl: (6) Couldn't resolve host 'metadata.google.internal'
So... at this point... I have no idea what it is I'm supposed to change or where I'm supposed to look. Any help would be appreciated.
I had this same problem, and didn't see any of the libraries I was using listed here.
In my case, the culprit turned out to be firebase-admin. I was using version 7.3.0, and I found this gem:
$ grep -rni "computeMetadata/" *
firebase-admin/lib/auth/credential.js:30:var GOOGLE_METADATA_SERVICE_PATH = '/computeMetadata/v1beta1/instance/service-accounts/default/token';
So, I updated my Cloud Functions libraries as shown here:
npm install firebase-functions#latest --save
npm install firebase-admin#latest --save-exact
and then, voila!
$ grep -rni "computeMetadata/" *
node_modules/firebase-admin/lib/auth/credential.js:30:var GOOGLE_METADATA_SERVICE_PATH = '/computeMetadata/v1/instance/service-accounts/default/token';
Then I redeployed and problem solved.
I searched at the https://github.com/firebase/firebase-functions repo latest version (3.3.0), and I found the file: spec/fixtures/https.ts. Inside this file there are some mock functions, which use the old: /computeMetadata/v1beta1 endpoint.
This might mean that firebase-functions modules package should be updated to use the /computeMetadata/v1 endpoint instead.
Fwiw I found this old dependency in package.json was dragging in other very old packages:
"#google-cloud/functions-emulator": "^1.0.0-beta.6",
In particular it brought in gcs-resumable-upload v 0.10.2, which is below the v 0.13.0 recommended by google (see https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server#apps-to-update). Probably others too.
The fix was to either:
remove #google-cloud/functions-emulator, or
switch to its modern replacement, #google-cloud/functions-framework
I am trying to send notifications tom slack when DAG run fails in airflow in google cloud composer. The version of airflow used is 1.9 so I cannot use slack webhooks. But when I add my code , i get this strange error : No module named 'slackclient'
I am not sure how to make this work in google cloud composer. I tried installing the slack package by adding PyPi variables in composer. But till now nothing works.
Anybody please help?
My code:
from slackclient import SlackClient
from airflow.operators.slack_operator import SlackAPIPostOperator
slack_channel= 'gsdgsdg'
slack_token = 'ssdfhfdrtxcuweiwvbnw54135f543589zdklchvfö'
def task_fail_slack_alert(context):
slack_msg = \
"""
:red_circle: Task Failed.
*Task*: {task}
*Dag*: {dag}
*Execution Time*: {exec_date}
*Log Url*: {log_url}
""".format(task=context.get('task_instance'
).task_id, dag=context.get('task_instance').dag_id,
ti=context.get('task_instance'),
exec_date=context.get('execution_date'),
log_url=context.get('task_instance').log_url)
failed_alert = SlackAPIPostOperator(
task_id = 'airflow_etl_failed',
channel = slack_channel,
token = slack_token,
text = slack_msg
)
return failed_alert.execute(context=context)
The SlackAPIPostOperator requires the slackclient library installed, which is usually done using
$ pip install apache-airflow[slack]
BUT as you're running Google Cloud Composer this clearly won't work. Instead, you can install slackclient using the environment's PyPI package installer.
Navigate to the
Google Cloud Console -> Google Cloud Composer -> Your Airflow environment
and then choose the "PYPI PACKAGES" tab. You can then specify the slackclient module to be installed, and maybe pin it at a version (>=1.3.2 ?)