Importing the operator in the following way:
from airflow.contrib.operators.gcs_download_operator import GoogleCloudStorageDownloadOperator
Then trying to use it in a DAG:
download_file = GoogleCloudStorageDownloadOperator(bucket='us-central1-scale-training-d7d12089-bucket',
google_cloud_storage_conn_id='google_cloud_default',
object='params.json',
filename='params.json')
Receiving this error:
'GoogleCloudStorageDownloadOperator' is not defined
Edit: I am using Google Cloud Composer so I assume the relevant dependecies are installed.
If you haven't already, you also need to add the GCP dependency to Airflow:
pip install apache-airflow[gcp_api]
There's more information about installation in the docs: https://airflow.apache.org/installation.html
Related
In our project we are using DataprocClusterCreateOperator which was under contrib from airflow.contrib.operators import dataproc_operator. It is working fine with airflow version 1.10.14.
We are in a process of upgrading to Airflow 2.1.2 wherein while testing or dags which requires spinning of DataProc Cluster we found error as airflow.exceptions.AirflowException: Invalid arguments were passed to DataprocClusterCreateOperator (task_id: <task_id>). Invalid arguments were: **kwargs: {'config_bucket': None, 'autoscale_policy': None}
I am not able to see any links for this operator support in Airflow 2 so that I can identify the new params or the changes which happened.
Please share the relevant link.
We are using google-cloud-composer version 1.17.2 having Airflow version 2.1.2.
Since Airflow 2.0, 3rd party provider (like Google in this case) operators/hooks has been moved away from Airflow core to separate providers packages. You can read more here.
Since you are using Cloud Composer, the Google providers package is already installed.
Regarding the DataprocClusterCreateOperator, it has been renamed to DataprocCreateClusterOperator and moved to airflow.providers.google.cloud.operators.dataproc so you can import it with:
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator
The accepted parameters differ from the one included in Airflow 1.x. You can find an example of usage here.
The supported parameters for the DataprocCreateClusterOperator in Airflow 2 can be found here, in the source code. The cluster configuration parameters that can be passed to the operator can be found here.
The DataprocClusterCreateOperator has been renamed as DataprocCreateClusterOperator since January 13, 2020 as per this Github commit and has been ported from airflow.contrib.operators to airflow.providers.google.cloud.operators.dataproc import path.
As given in #itroulli's answer, an example implementation of the operator can be found here.
I'm getting an error in express-serve-static core when trying to build my firebase cloud function locally.
How do I fix this?
I've tried reinstalling the node packages but without luck.
node_modules/#types/express-serve-static-core/index.d.ts:31:10 - error TS2305: Module '"../qs"' has no exported member 'ParsedQs'.
31 import { ParsedQs } from "qs";
You can fix this by installing #types/qs.
npm i #types/qs#6.9.3
Try to return to #types/express-serve-static-core#4.17.5 (instead of the latest 4.17.7).
If you don't have this package in you dependencies, install it explicitly.
npm install -D #types/express-serve-static-core#4.17.5
I had the same problem and returning to the version 4.17.5 worked for me.
I am trying to send notifications tom slack when DAG run fails in airflow in google cloud composer. The version of airflow used is 1.9 so I cannot use slack webhooks. But when I add my code , i get this strange error : No module named 'slackclient'
I am not sure how to make this work in google cloud composer. I tried installing the slack package by adding PyPi variables in composer. But till now nothing works.
Anybody please help?
My code:
from slackclient import SlackClient
from airflow.operators.slack_operator import SlackAPIPostOperator
slack_channel= 'gsdgsdg'
slack_token = 'ssdfhfdrtxcuweiwvbnw54135f543589zdklchvfö'
def task_fail_slack_alert(context):
slack_msg = \
"""
:red_circle: Task Failed.
*Task*: {task}
*Dag*: {dag}
*Execution Time*: {exec_date}
*Log Url*: {log_url}
""".format(task=context.get('task_instance'
).task_id, dag=context.get('task_instance').dag_id,
ti=context.get('task_instance'),
exec_date=context.get('execution_date'),
log_url=context.get('task_instance').log_url)
failed_alert = SlackAPIPostOperator(
task_id = 'airflow_etl_failed',
channel = slack_channel,
token = slack_token,
text = slack_msg
)
return failed_alert.execute(context=context)
The SlackAPIPostOperator requires the slackclient library installed, which is usually done using
$ pip install apache-airflow[slack]
BUT as you're running Google Cloud Composer this clearly won't work. Instead, you can install slackclient using the environment's PyPI package installer.
Navigate to the
Google Cloud Console -> Google Cloud Composer -> Your Airflow environment
and then choose the "PYPI PACKAGES" tab. You can then specify the slackclient module to be installed, and maybe pin it at a version (>=1.3.2 ?)
I have configured electron with vuejs and i would like to add firebae to my project so i have
In my main.js
import * as firebase from 'firebase'
let firebasconfig = {
//config from firebase project
};
Vue.prototype.$firebase = firebase.initializeApp(firebasconfig);
But now am getting an error
Error: Failed to load gRPC binary module because it was not installed
for the current system
Expected directory: electron-v1.8-linux-x64-glibc
Found: [node-v59-linux-x64-glibc]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module
I have tried running rebuild but still fails
What elese do i need to do for this to work
I encouter the same issue with
vue-electron 1.0.6
firebase 5.4.2
vuefire 1.4.5
node 6.10.3
The following manipulation has worked for me. I have downgraded :
firebase to 4.6.0
vuefire to 1.4.4
Then ran npm rebuild and yarn and now it works.
I'm not sure all the things I have done are necessary. But for sure it looks like a problem with firebase.
I try to import node-rsa npm package to my project: "import NodeRSA from 'node-rsa';" but i get following error: Cannot find module 'crypto'.
crypto is a node builtin module, so its not installed over npm. Does that mean i can't use 'node-rsa' directly?
https://atmospherejs.com/planifica/node-rsa works without any problems.
planifica:node-rsa - "Browserified Node.js RSA library carefully packed for Meteor"
thanks