composer workflow fails at dataproc operator - airflow

Question

I have a composer environment setup in gcp and it's running a DAG as follows
with DAG('sample-dataproc-dag',
default_args=DEFAULT_DAG_ARGS,
schedule_interval=None) as dag: # Here we are using dag as context
# Submit the PySpark job.
submit_pyspark = DataProcPySparkOperator(
task_id='run_dataproc_pyspark',
main='gs://.../dataprocjob.py',
cluster_name='xyz',
dataproc_pyspark_jars=
'gs://.../spark-bigquery-latest_2.12.jar'
)
simple_bash = BashOperator(
task_id='simple-bash',
bash_command="ls -la")
submit_pyspark.dag = dag
submit_pyspark.set_upstream(simple_bash)
and this is my dataprocjob.py
from pyspark.sql import SparkSession
if __name__ == '__main__':
spark = SparkSession.builder.appName('Jupyter BigQuery Storage').getOrCreate()
table = "projct.dataset.txn_w_ah_demo"
df = spark.read.format("bigquery").option("table",table).load()
df.printSchema()
my composer pipeline fails at the dataproc step. In the composer log stored in the gcs, this is what I see
[2020-09-23 21:40:02,849] {taskinstance.py:1059} ERROR - <HttpError 403 when requesting https://dataproc.googleapis.com/v1beta2/projects/lt-dia-pop-dis-upr/regions/global/jobs?clusterName=dppoppr004&alt=json returned "Not authorized to requested resource.">#-#{"workflow": "sample-dataproc-dag", "task-id": "run_dataproc_pyspark", "execution-date": "2020-09-23T21:39:42.371933+00:00"}
Traceback (most recent call last):
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 930, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/airflow/airflow/contrib/operators/dataproc_operator.py", line 1139, in execute
super(DataProcPySparkOperator, self).execute(context)
File "/usr/local/lib/airflow/airflow/contrib/operators/dataproc_operator.py", line 707, in execute
self.hook.submit(self.hook.project_id, self.job, self.region, self.job_error_states)
File "/usr/local/lib/airflow/airflow/contrib/hooks/gcp_dataproc_hook.py", line 311, in submit
num_retries=self.num_retries)
File "/usr/local/lib/airflow/airflow/contrib/hooks/gcp_dataproc_hook.py", line 51, in __init__
clusterName=cluster_name).execute()
File "/opt/python3.6/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/opt/python3.6/lib/python3.6/site-packages/googleapiclient/http.py", line 851, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://dataproc.googleapis.com/v1beta2/projects/lt-dia-pop-dis-upr/regions/global/jobs?clusterName=dppoppr004&alt=json returned "Not authorized to requested resource.">

Answer 1

It seems that the issue you are presenting corresponds to the Dataproc permissions you have granted to your application.
According to docummentation, you need different role privileges to perform Dataproc tasks, for example:
dataproc.clusters.create permits the creation of Cloud Dataproc clusters in the containing project
dataproc.jobs.create permits the submission of Dataproc jobs to Dataproc clusters in the containing project
dataproc.clusters.list permits the listing of details of Dataproc clusters in the containing project
If you want to create a submit a dataproc job, you need the 'dataproc.clusters.use' and 'dataproc.jobs.create' permission.
In order to grant the correct privileges to your user account, you can follow the docummentation for updating your service account you are using in your code and add the correct permissions.

Answer 2

From a first read this looks like the permissions on the Google Cloud account that you are invoking the Dataproc APIs aren't adequate for the Operator.