I am trying to get XCOM value for specific DAG using API?
Can't find any way of doing it.
Any idea?!
Airflow exposed API probably since version 2.
Airflow API
There is also exposed API for the xcom values, more details here
Related
Is it possible to monitor the state of an already running remote job in Livy with Java API? How can this be done?
I looked over Livy Java API docs. A JobHandle would let me pool the state of the app. However, the only way I can see to obtain it is via LivyClient.submit while I need to get a handle of the job that was already submitted outside of Java code. I'm afraid that Java API doesn't support it and creating REST calls in Java code is the only option. If anyone has found a way to get information about batch jobs via Livy Java API, please show the code used.
does anyone know how to enable the Message Ordering property on a subscription when working with Firebase and NodeJs?
If I navigate to the GCP console and open the queue's subscription detail, I can change the parameter manually but I'd rather have it done when my functions deploy.
This link (https://cloud.google.com/pubsub/docs/ordering#enabling_message_ordering) also talks about a parameter that can be provided when creating a subscription but that doesn't seem to be available the way I usually create/deploy my functions.
GCP Documentation
Own Code
It sounds like you're asking if pubsub functions deployed with the Firebase CLI (and not gcloud) can request ordering. Currently, this is not possible. When you deploy a pubsub function with the Firebase CLI, it will automatically create the pubsub topic for you. However, there is no option to set the ordering setting.
If you would like see this implemented, you should file a feature request on the Firebase CLI GitHub and/or contact Firebase support directly.
Alternatively, you can switch to use gcloud to establish the trigger and create its pubsub topic with your customizations.
You should note that enabling ordering does not necessarily ensure that function invocations will be perfectly serialized with respect to incoming messages. Cloud Functions is designed for incoming events to be handled in parallel as fast as possible by using multiple server instances as needed. There is no guarantee that pubsub messages will be processed fully in order. The ordering only applies to pubsub topic consumer code that runs in a single process. You could try to set the max instances of a function to 1, but that is not a hard guarantee.
If you need fully ordered serialized execution of some code in response to pubsub events, maybe Cloud Functions isn't the product you want to use. Maybe a single App Engine or Compute Engine instance would work better.
I want to use Firebase' REST API to send HTTP operations to fetch data from my realtime database that passes some criteria. Below is a picture of how the DB is set up. An example could be fetching only the records that have animalType="Cat". Is there a way of altering the URL I use to send the GET request to include this? Something along the lines of https://database-name-38a0t.firebaseio.com/animals?animalType=Cat
For info: I'm building an app using react native expo.
You need orderBy and equalTo in your query params as outlined in the docs:
https://database-name.firebaseio.com/animals?orderBy="animalType"&equalTo="Cat"
I currently working on a sync third party data project, that I decided to implement queue to ensure the message has received to the worker, but the worker need to do only one task at the time, which means the previous task should finished and acknowledged before executing the next one.
so the question is, how can I config the firebase pub/sub trigger to execute task one by one ?.
If I misunderstand on the concept of google pub/sub feel free to tell me :)
What you're trying to do isn't really compatible with the way pubsub functions work. There are other Google Cloud products that can help you limit the rate of invocation of some code, but it won't be implemented with a pubsub type function.
GCP offers a product called Cloud Tasks, which lets you set up a queue that can be configured with a rate limit. The queue can dispatch a task invocation to App Engine, or to an HTTP type Cloud Function (beta). So, if you've deployed an HTTP function, you could use its endpoint URL to configure a Cloud Tasks queue that invokes that function only once at a time.
You can control the number of instances of Cloud Functions with the MAX-INSTANCES flag of gcloud command line tool. See the documentation on Using max instances for full details.
The Firebase SDK for Cloud Functions doesn't wrap this option (yet). But you can deploy the function using the Firebase CLI, and then change the setting in the Cloud console, by following the steps shown in Clearing max instances limits.
I'm teaching myself to use Flutter and I'm making an app that queries The Movie Database API. Currently, I'm having the client query the API on launch but I'm thinking this is not the most efficient way of doing it, and I would rather have the client query a backend service like Firebase to get the same data.
I would appreciate some guidance into where to start in order to setup a periodical process to query the API and use the results as entries into a Firestore DB. I've looked online but I might be using suboptimal keywords since I haven't found a good tutorial or example for this.
Thanks.
You can use Firebase Cloud Functions to build code that runs on Firebase servers to fill your Firebase database, but you can only make HTTP requests to non-Google addresses if you use a paid plan.
https://firebase.googleblog.com/2017/03/how-to-schedule-cron-jobs-with-cloud.html explains how to invoke periodic tasks with Cloud Functions. It utilizes Google AppEngine for that because Cloud Functions doesn't provide that out of the box.