I am trying to create a Cloud Composer DAG to be triggered via a Pub/Sub message.
There is the following example from Google which triggers a DAG every time a change occurs in a Cloud Storage bucket:
https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
However, on the beginning they say you can trigger DAGs in response to events, such as a change in a Cloud Storage bucket or a message pushed to Cloud Pub/Sub. I have spent a lot of time try to figure out how that can be done, but no result.
Can you please help or giving me some directions? Thanks!
There are 2 ways to trigger a DAG by a Pub/Sub events.
You can place a PubSubPullSensor at the beginning of your DAG. Your DAG will be triggered every time a Pub/Sub message can be pulled by the PubSubPullSensor. And it will execute the rest of the tasks in your DAG.
You can also create a Cloud Function that acts as Pub/Sub trigger. And put the Composer DAG triggering logic inside the Cloud Function trigger. When a message is published to the Pub/Sub topic, the Cloud Function should be able to trigger the Composer DAG.
To extend the public documentation page you already posted, you can configure a Cloud Function to run each time a message is published to a Cloud Pub/Sub topic. There is more information about that in another public documentation page.
To attach a function to a topic, set the --trigger-topic flag when deploying the function:
gcloud functions deploy $FUNCTION_NAME --runtime $RUNTIME --trigger-topic $TOPIC_NAME
Related
Is it possible to run a Function/Cloud Run after a Cloud Scheduler job?
For example, I want to store in Firestore if a job was successful or had any error, in order to do this, I was wondering if I could "connect" the events from Cloud Scheduler to a Cloud Function or Cloud Run.
Is it possible?
These events (image bellow)
EDIT: Maybe using Eventarc?
I am building an appointment booking app, I'm using FCM to handle push notifications,
so I am wondering if there is a way to schedule push notifications
For example;
When the doctor gives an appointment time, the patient should be notified few minutes to the appointment time.
I've read about cron job but I'm not sure on how to schedule it to a dynamic time.
You can either run a scheduled task at a fixed interval (say every minute) and then send all messages for the past/next minute, or you can use Cloud Tasks to schedule the task (which is then a HTTP Cloud Function) either yourself (as Doug explained here) oras shown in this documentation.
This topic has been covered quite a few times before, so I recommend also checking out:
How to create cron jobs dynamically in firebase
How to trigger cloud function based on date stored in cloud firestore?
How to trigger a cloud function based on a timestamp attribute of a firestore document
run some code 1 hour after a document is updated with cloud function and firestore
I am looking to integrate an automated posting service in my firebase application. The users will create a post with a desired posting time from the client application which will be added to my Firestore database.
I would like to be able to create a Cloud Task to actually add the post to the client dashboard at the desired time, which could be weeks/months in the future.
Is a cloud function Firestore trigger that creates a cloud task the best implementation?
I know that Cloud Scheduler/ Pub/Sub /App Engine is normally the flow recommended for functions run on a normal schedule, i.e once daily/weekly. But I am looking to allow my users to specify the exact time they want their post to be sent.
Is my thinking to use Cloud Tasks correct?
Any insight would be appreciated!
I think that best approach is to use the Cloud Functions for Firebase client SDKs that will let you call functions directly from a Firebase app. To call a function from your app in this way, write and deploy an HTTPS Callable function in Cloud Functions, and then add client logic to call the function from your app. Then, if you want to schedule functions to run at specified times, use functions.pubsub.schedule().onRun() This method creates a Pub/Sub topic and uses Cloud Scheduler to trigger events on that topic, ensuring that your function runs on the desired schedule.
I currently working on a sync third party data project, that I decided to implement queue to ensure the message has received to the worker, but the worker need to do only one task at the time, which means the previous task should finished and acknowledged before executing the next one.
so the question is, how can I config the firebase pub/sub trigger to execute task one by one ?.
If I misunderstand on the concept of google pub/sub feel free to tell me :)
What you're trying to do isn't really compatible with the way pubsub functions work. There are other Google Cloud products that can help you limit the rate of invocation of some code, but it won't be implemented with a pubsub type function.
GCP offers a product called Cloud Tasks, which lets you set up a queue that can be configured with a rate limit. The queue can dispatch a task invocation to App Engine, or to an HTTP type Cloud Function (beta). So, if you've deployed an HTTP function, you could use its endpoint URL to configure a Cloud Tasks queue that invokes that function only once at a time.
You can control the number of instances of Cloud Functions with the MAX-INSTANCES flag of gcloud command line tool. See the documentation on Using max instances for full details.
The Firebase SDK for Cloud Functions doesn't wrap this option (yet). But you can deploy the function using the Firebase CLI, and then change the setting in the Cloud console, by following the steps shown in Clearing max instances limits.
So I know that Cloud Functions for Firebase don't natively support cron jobs, and one have to depend on an external source to trigger an HTTP cloud function. But I don't think this will work in my case.
I have an app in which users can create competitions that end in 24 hours, thus each time a competition is created, I need to set up a timer that triggers one time after 24 hours.
Do you guys have any idea on how this problem can be approached?
At this moment the solution that the Firebase guys are providing for solving the cron issue is to use Google App Engine to deploy a server and trigger the functions via Pub/Sub hooks. Here you can find the tutorial.
Basically you have to:
Create a Cloud Function that responds to a PubSub hook.
Deploy an App Engine that will publish an event depending on the cron setup you set.
If you don't want to setup an App Engine you can use an external resource like cron-job.org that can call your HTTP Cloud Function.