I am trying to send notifications tom slack when DAG run fails in airflow in google cloud composer. The version of airflow used is 1.9 so I cannot use slack webhooks. But when I add my code , i get this strange error : No module named 'slackclient'
I am not sure how to make this work in google cloud composer. I tried installing the slack package by adding PyPi variables in composer. But till now nothing works.
Anybody please help?
My code:
from slackclient import SlackClient
from airflow.operators.slack_operator import SlackAPIPostOperator
slack_channel= 'gsdgsdg'
slack_token = 'ssdfhfdrtxcuweiwvbnw54135f543589zdklchvfö'
def task_fail_slack_alert(context):
slack_msg = \
"""
:red_circle: Task Failed.
*Task*: {task}
*Dag*: {dag}
*Execution Time*: {exec_date}
*Log Url*: {log_url}
""".format(task=context.get('task_instance'
).task_id, dag=context.get('task_instance').dag_id,
ti=context.get('task_instance'),
exec_date=context.get('execution_date'),
log_url=context.get('task_instance').log_url)
failed_alert = SlackAPIPostOperator(
task_id = 'airflow_etl_failed',
channel = slack_channel,
token = slack_token,
text = slack_msg
)
return failed_alert.execute(context=context)
The SlackAPIPostOperator requires the slackclient library installed, which is usually done using
$ pip install apache-airflow[slack]
BUT as you're running Google Cloud Composer this clearly won't work. Instead, you can install slackclient using the environment's PyPI package installer.
Navigate to the
Google Cloud Console -> Google Cloud Composer -> Your Airflow environment
and then choose the "PYPI PACKAGES" tab. You can then specify the slackclient module to be installed, and maybe pin it at a version (>=1.3.2 ?)
Related
I am sending mail using airflow emailOperator. Is there any way to send "reply-to" parameter also. i tried default_param={'key':'value'} but throwing exception as invalid parameter.
I read this https://github.com/apache/airflow/blob/main/airflow/operators/email.py
https://github.com/apache/airflow/blob/main/tests/operators/test_email.py
But I don't know where i am doing mistake.
version : 2.0.2
email_op = EmailOperator(
task_id='send_email',
to=destination['emails'],
subject=destination['subject'],
html_content=destination['message'],
files=[file.name],
default_param={'Reply-To': 'reply_to#example.com'},
)
email_op.execute(context)
It would be passed via dictionary custom_headers, as per the Official Source Code and Official Docs
I actually took the Reply-To directly from the official Airflow unit test.
There was a bug patched in release 2.3.0 with the send_email function to correctly pass the custom_headers. This might be your root cause...
Here is the link to the issue
Here is the pull request to resolve it
For new comers here is how you would send the custom_headers when using 2.3.0 and above.
from airflow.operators.email_operator import EmailOperator
t1 = EmailOperator(
task_id='send_email',
to="email#example.com",
subject="Airflow Example",
html_content=""" <h3>Email Test</h3> """,
cc="email#example",
custom_headers = {
'Reply-To': 'reply_to#example.com'
}
)
In our project we are using DataprocClusterCreateOperator which was under contrib from airflow.contrib.operators import dataproc_operator. It is working fine with airflow version 1.10.14.
We are in a process of upgrading to Airflow 2.1.2 wherein while testing or dags which requires spinning of DataProc Cluster we found error as airflow.exceptions.AirflowException: Invalid arguments were passed to DataprocClusterCreateOperator (task_id: <task_id>). Invalid arguments were: **kwargs: {'config_bucket': None, 'autoscale_policy': None}
I am not able to see any links for this operator support in Airflow 2 so that I can identify the new params or the changes which happened.
Please share the relevant link.
We are using google-cloud-composer version 1.17.2 having Airflow version 2.1.2.
Since Airflow 2.0, 3rd party provider (like Google in this case) operators/hooks has been moved away from Airflow core to separate providers packages. You can read more here.
Since you are using Cloud Composer, the Google providers package is already installed.
Regarding the DataprocClusterCreateOperator, it has been renamed to DataprocCreateClusterOperator and moved to airflow.providers.google.cloud.operators.dataproc so you can import it with:
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator
The accepted parameters differ from the one included in Airflow 1.x. You can find an example of usage here.
The supported parameters for the DataprocCreateClusterOperator in Airflow 2 can be found here, in the source code. The cluster configuration parameters that can be passed to the operator can be found here.
The DataprocClusterCreateOperator has been renamed as DataprocCreateClusterOperator since January 13, 2020 as per this Github commit and has been ported from airflow.contrib.operators to airflow.providers.google.cloud.operators.dataproc import path.
As given in #itroulli's answer, an example implementation of the operator can be found here.
Snowflake is not showing in the connections dropdown.
I am using MWAA 2.0 and the providers are already in the requirements.txt
MWAA uses python 3.7 dont know if this can be a thing
Requirements.txt:
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
asn1crypto
azure-common
azure-core
azure-storage-blob
boto3
botocore
certifi
cffi
chardet
cryptography
greenlet
idna
isodate
jmespath
msrest
numpy
oauthlib
oscrypto
pandas
pyarrow
pycparser
pycryptodomex
PyJWT
pyOpenSSL
python-dateutil
pytz
requests
requests-oauthlib
s3transfer
six
urllib3
apache-airflow-providers-http
apache-airflow-providers-snowflake
#apache-airflow-providers-snowflake[slack]
#apache-airflow-providers-slack
snowflake-connector-python >=2.4.1
snowflake-sqlalchemy >=1.1.0
\
If anyone is in this trouble, instead of choosing Snowflake in the dropdown, you can choose AWS as the connection and will work fine.
It took me a while to finally figure this one out after trying many different parameter combinations.
My full Snowflake URL is:
https://xx12345.us-east-2.aws.snowflakecomputing.com
The correct format for the Host field is:
xx12345.us-east-2.snowflakecomputing.com
For the Extra field, this is what worked for me:
{
"account": "xx12345.us-east-2.aws",
"warehouse": "my_warehouse_name",
"database": "my_database_name"
}
Make sure you put Amazon Web Services for the Conn Type, like #AXI said.
Also, I have these modules defined in my requirements.txt file:
apache-airflow-providers-snowflake==1.3.0
snowflake-connector-python==2.4.5
snowflake-sqlalchemy==1.2.4
My Airflow version is 2.0.2.
According to MWAA docs, it should be enough to add apache-airflow-providers-snowflake==1.3.0 to the requirements file. When I added it to the existing MWAA env, where I had already tried many different combinations of packages, it helped partially. It was possible to create a connection using CLI, but not with UI.
But, when I created a new clean MWAA env with the requirements file as stated in mentioned AWS doc, it worked well. The connection was available in UI.
Importing the operator in the following way:
from airflow.contrib.operators.gcs_download_operator import GoogleCloudStorageDownloadOperator
Then trying to use it in a DAG:
download_file = GoogleCloudStorageDownloadOperator(bucket='us-central1-scale-training-d7d12089-bucket',
google_cloud_storage_conn_id='google_cloud_default',
object='params.json',
filename='params.json')
Receiving this error:
'GoogleCloudStorageDownloadOperator' is not defined
Edit: I am using Google Cloud Composer so I assume the relevant dependecies are installed.
If you haven't already, you also need to add the GCP dependency to Airflow:
pip install apache-airflow[gcp_api]
There's more information about installation in the docs: https://airflow.apache.org/installation.html
I have configured electron with vuejs and i would like to add firebae to my project so i have
In my main.js
import * as firebase from 'firebase'
let firebasconfig = {
//config from firebase project
};
Vue.prototype.$firebase = firebase.initializeApp(firebasconfig);
But now am getting an error
Error: Failed to load gRPC binary module because it was not installed
for the current system
Expected directory: electron-v1.8-linux-x64-glibc
Found: [node-v59-linux-x64-glibc]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module
I have tried running rebuild but still fails
What elese do i need to do for this to work
I encouter the same issue with
vue-electron 1.0.6
firebase 5.4.2
vuefire 1.4.5
node 6.10.3
The following manipulation has worked for me. I have downgraded :
firebase to 4.6.0
vuefire to 1.4.4
Then ran npm rebuild and yarn and now it works.
I'm not sure all the things I have done are necessary. But for sure it looks like a problem with firebase.