I've been trying to get this operator working for some time since switching to airflow 2.0 BigQueryInsertJobOperator.
The error I'm seeing shows there is something missing from our connection, oddly enough this connection works in another DAG where we are using google's api to access google sheets:
export AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT=
"google-cloud-platform://?extra__google_cloud_platform__project=\analytics&extra__google_cloud_platform__keyfile_dict=
{\"type\": \"service_account\", \"project_id\": \"analytics\",
\"private_key_id\": \"${GCLOUD_PRIVATE_KEY_ID}\", \"private_key\": \"${GCLOUD_PRIVATE_KEY}\",
\"client_email\": \"d#lytics.iam.gserviceaccount.com\", \"client_id\": \"12345667\",
\"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",
\"token_uri\": \"https://accounts.google.com/o/oauth2/token\",
\"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",
\"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/d#lytics.iam.gserviceaccount.com\"}"
This is the error I'm seeing:
{
"error": {
"code": 401,
"message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"errors": [
{
"message": "Login Required.",
"domain": "global",
"reason": "required",
"location": "Authorization",
"locationType": "header"
}
],
"status": "UNAUTHENTICATED"
}
}
is there a way I can look up what else might be required in terms of formatting, etc, perhaps a really good example on how to get the correct connection setup for this Operator??
In my logs I'm seeing this error which makes me think perhaps it might not be a credential issue?
File "/usr/local/lib/python3.8/site-packages/google/cloud/_http.py", line 438, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.BadRequest: 400 POST https://bigquery.googleapis.com/bigquery/v2/projects/vice-analytics/jobs?prettyPrint=false: Required parameter is missing
Create a service account json key, which contains all the required info posted in your error message.
https://cloud.google.com/iam/docs/creating-managing-service-account-keys
Then you can paste the json key into the Airflow UI: Admin -> Connections in the json key field and reference this in your dag with: gcp_conn_id="name of connection you created"
Or add the json key as an env variable (on macos):
export GOOGLE_APPLICATION_CREDENTIALS="link to your json key file"
Related
I am getting this response back when trying to execute a WebAPI request, but can't figure out why its erroring out. Moreover the error is not very helpful, as it doesn't tell which entity or which field, and I cannot identify any fields missing.
{
"error": {
"code": "90040100",
"message": "Attribute {0} cannot be found for table {1}.",
"innererror": {
"code": "90040100",
"message": "Attribute {0} cannot be found for table {1}.",
"type": "InvalidAttribute"
}
}
}
I tried this myself a few times but got the same errors that you describe here. According to the Microsoft documentation this is explicitly not supported.
Calling actions and functions using the portals Web API is not supported.
I am trying to get the firebase token through postman with email and password by using below API
https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key={API_KEY}
Below is the body
{"email":"{YOUR_EMAIL_ADDRESS}","password":"{PASSWORD}","returnSecureToken":true}
I am using proper email and password which I used to log into firebase console. But I am getting EMAIL_NOT_FOUND error as below.
{
"error": {
"code": 400,
"message": "EMAIL_NOT_FOUND",
"errors": [
{
"message": "EMAIL_NOT_FOUND",
"domain": "global",
"reason": "invalid"
}
]
}
}
From the firebase document, we will get this error when "There is no user record corresponding to this identifier. The user may have been deleted."
But I am able to see the email under "Users and permissions" and the role as 'owner'. I am using the same email ID. Can anyone please help me to fix this issue.
You need to add the user in the authentication list:
https://console.firebase.google.com/project/*****/authentication/users
Firebase add user panel:
Mavens,
I have managed to change the response for fault and error from xml to json using the below link
https://docs.wso2.com/display/AM260/Error+Handling
However, I am now left with how to change the payloads for the faults or errors generated by the system itself
Something which is generated by default from the system
{ "fault": { "code": 900901, "message": "Invalid Credentials", "description": "Access failure for API: /employee/1.0.0, version: 1.0.0 status: (900901) - Invalid Credentials. Make sure you have provided the correct security credentials" }}
to something customized like this
{ "status": "fault", "description":{ "code": 900901, "message": "Invalid Credentials", "description": "Access failure for API: /employee/1.0.0, version: 1.0.0 status: (900901) - Invalid Credentials. Make sure you have provided the correct security credentials" }}
Update 1 :
Having a look at: https://medium.com/#azinneera/transforming-backend-error-messages-in-wso2-api-manager-91bf8d744ad4
I want to retrieve calendar events of the logged in user. The API doc here suggests that I need to pass 'primary' as calendarId for the logged in user.
Therefore the URL becomes https://www.googleapis.com/calendar/v3/calendars/primary/
I am getting a 401 on this URL.
GET https://www.googleapis.com/calendar/v3/calendars/primary/ 401 ()
Can someone please suggest what am I doing wrong.
I think you need to pass a different calendarId (email address of a calendar). To get your calendarID, you can follow the steps specified here.
And to know about the error you encountered, here is the document about 401: Invalid Credentials.
{
"error": {
"errors": [
{
"domain": "global",
"reason": "authError",
"message": "Invalid Credentials",
"locationType": "header",
"location": "Authorization",
}
],
"code": 401,
"message": "Invalid Credentials"
}
}
Suggested actions:
Get a new access token using the long-lived refresh token.
If this fails, direct the user through the OAuth flow, as described in Authorizing requests with OAuth 2.0.
If you are seeing this for a service account, check that you have successfully completed all the steps in the service account
page.
Let's say I have a Model with two mandatory fields:
class ExampleModel(EndpointsModel):
attr1 = ndb.StringProperty(required=True)
attr2 = ndb.StringProperty(required=True)
Then I want to use endpoints-proto-datastore to query on either attr1 or attr2:
#ExampleModel.query_method(query_fields=('attr1', 'attr2'),
path='example', name='list')
def example_list(self, query):
return query
This fails if I only provide one of the fields - from API Explorer it's a required field, but the API itself returns:
{
"error": {
"code": 400,
"errors": [
{
"domain": "global",
"message": "Error parsing ProtoRPC request (Unable to parse request content: Message CombinedContainer is missing required field attr2)",
"reason": "badRequest"
}
],
"message": "Error parsing ProtoRPC request (Unable to parse request content: Message CombinedContainer is missing required field attr2)"
}
}
Obviously I could mark them as not required, then handle the check within the application code - but I was wondering if someone else had come up with a better solution.
Many thanks
This is an old question but I ran into the same confusion. This was the answer I found. Basically if you want to make something mandated on Post but not get you need to make a custom proto class. Which can only be used with method and not query_method.