Is there a view possible in airflow UI that displays what dags are scheduled on a weekly basis? With the current setup, it's hard to understand how many KRI's are scheduled for a day.
Related
I am using Quartz.Net 3.2.0 with a .net core application and here is my requirement.
Users can alter the schedule from UI, they can select the report they want to send at any specific time or schedule. Now, I have all this info in the backend table, which I am getting through an API.
I know with quartz we can schedule jobs using cron expression at a specific time, but my requirement is, that my jobs are not fixed, these can vary each day. So I need to dynamically refresh the scheduled jobs daily, maybe I can do it at night at one time, and then all jobs will perform at the given time I am getting through API.
But I don't know how to dynamically register the jobs and refresh them daily basis, I am new to quartz.Net.
Please suggest me approach and sample code to do this, thanks in advance.
I have created a data pipeline in Airflow which helps to update the tickets in Freshdesk whenever some values are change in the tickets during business hours. My logic for updatding the ticket is working fine, but I want to make the trigger the dags whenever a event is occured in Freshdesk interface so that I don't need to manually trigger the dags.
There are automation rules in FreshDesk which provides the external webhook in https://companyname.freshdesk.com/api/v2/tickets/{{ticket.id}} where {ticket.id} is essentail to know the id for which the tickets are updated from the data pipeline.
I need to know how to connect the external trigger such as webhook so that we can trigger the data-pipeline automatically in Airflow.
My Airflow Version 1.10
Airflow doesn't have a mechanism that allows triggering DAG based on webhooks from other services.
That use case might be covered by AIP-35 Add Signal Based Scheduling To Airflow but this is currently a draft idea of enhancement to Airflow.
To achieve your use case with current functionality you will need to other application/service to handle the webhook fired by Freshdesk and triggering a new DAG run in Airflow with RestAPI Trigger new DAG run endpoint
You need to be aware of frequency. If the frequency of creating new DAG runs is too high you should consider switching to a batch solution (which Airflow is more suitable for)
I am not sure about Freshdesk, but according to this it sounds as if you're able to hit an external API with its automation mechanism.
If that's the case, then I think this is what you're looking for. That's the documentation for the REST API reference of Apache Airflow.
Airflow allows you to manage it via web API, but you first need to enable it in the configuration. You should set the following configuration to use at least basic auth.
Keep in mind that back then that was an experimental API and newer versions changed the URL schema and the API a bit. It's much more complete nowadays.
I have a unique problem here. I actually need to schedule a task to send emails to eligible users at specific times. I have checked and found out that scheduling cron jobs can solve the problem. However, in my case, I need to schedule these tasks programmatically based on when specific users meet certain conditions.
Here is my scenario
I am running a referral campaign in my application where users who refer up to 5 persons get a discounted subscription
After the user has referred 5 persons, the cost of subscription is slashed as long as he pays within 24 hours. Otherwise the cost returns to the original value
I'm using firebase cloud functions to send eligible users reminder email reminders at certain times to make payment before the 24 hours elapses
Now the problem
I want to send reminder emails at specific times say as soon as he refers 5 persons, when his time is remaining say 3 hours etc.
If I just schedule a function regardless of the user, I may have to set cron jobs at regular intervals say 1 minute to always check if the users in my db are eligible and then send emails to them. This pose serious issue such that cloud functions are ran whether or not there are eligible users.
Another problem is that with the above implementation, I cannot send emails to users at specific/exact times unless I schedule tasks for like every 1 second which will of course sky rocket price as cloud functions will be called around a whooping 86400 times (86400 seconds) a day
My proposed solution
I just need a way to schedule cron tasks dynamically so that I can schedule a job for particular users. This will solve many problems like sending the emails at particular times, prevent cloud functions from running when not necessary, ability to set different email send times for different users etc
I plan using http triggers with a request parameter of the user ID for scheduling tasks so that firebase can use this ID to assign tasks to only specific users, i.e each eligible user will have his own http trigger
Please is there a way to achieve this or any other good solution for my case?
Instead of trying to dynamically creating scheduled function, consider having a single function that runs on a fixed schedule, and then implementing your timing logic inside of that function.
So instead of saying "I need to schedule a function that runs 3 hours from now and send a message to these 5 people", think of it as a task that you can write into a database: "at x:yz send an email to these 5 people", and then have a periodic Cloud Function that checks what tasks it needs to execute.
Also see Delay Google Cloud Function, How to use scheduler for Firebase Cloud Functions with Realtime Database/Analytics triggers?
Alternatively you can use Cloud Scheduler to create a task for the specific action you want to perform, and then have it post to Cloud Functions via PubSub.
As an even newer alternative: Use a separate scheduler service that has an API to create schedules, like Cloud Tasks. Doug wrote a great article about that in How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL).
I have a unique problem here. I actually need to schedule a task to send emails to eligible users at specific times. I have checked and found out that scheduling cron jobs can solve the problem. However, in my case, I need to schedule these tasks programmatically based on when specific users meet certain conditions.
Here is my scenario
I am running a referral campaign in my application where users who refer up to 5 persons get a discounted subscription
After the user has referred 5 persons, the cost of subscription is slashed as long as he pays within 24 hours. Otherwise the cost returns to the original value
I'm using firebase cloud functions to send eligible users reminder email reminders at certain times to make payment before the 24 hours elapses
Now the problem
I want to send reminder emails at specific times say as soon as he refers 5 persons, when his time is remaining say 3 hours etc.
If I just schedule a function regardless of the user, I may have to set cron jobs at regular intervals say 1 minute to always check if the users in my db are eligible and then send emails to them. This pose serious issue such that cloud functions are ran whether or not there are eligible users.
Another problem is that with the above implementation, I cannot send emails to users at specific/exact times unless I schedule tasks for like every 1 second which will of course sky rocket price as cloud functions will be called around a whooping 86400 times (86400 seconds) a day
My proposed solution
I just need a way to schedule cron tasks dynamically so that I can schedule a job for particular users. This will solve many problems like sending the emails at particular times, prevent cloud functions from running when not necessary, ability to set different email send times for different users etc
I plan using http triggers with a request parameter of the user ID for scheduling tasks so that firebase can use this ID to assign tasks to only specific users, i.e each eligible user will have his own http trigger
Please is there a way to achieve this or any other good solution for my case?
Instead of trying to dynamically creating scheduled function, consider having a single function that runs on a fixed schedule, and then implementing your timing logic inside of that function.
So instead of saying "I need to schedule a function that runs 3 hours from now and send a message to these 5 people", think of it as a task that you can write into a database: "at x:yz send an email to these 5 people", and then have a periodic Cloud Function that checks what tasks it needs to execute.
Also see Delay Google Cloud Function, How to use scheduler for Firebase Cloud Functions with Realtime Database/Analytics triggers?
Alternatively you can use Cloud Scheduler to create a task for the specific action you want to perform, and then have it post to Cloud Functions via PubSub.
As an even newer alternative: Use a separate scheduler service that has an API to create schedules, like Cloud Tasks. Doug wrote a great article about that in How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL).
I want to build a task scheduling service on the Google Cloud Platform. The tasks can be as simple as triggering a URL. Tasks can be recurring (once an hour, twice a day, every thursday, ...) and can be created and removed dynamically.
Which services/APIs on the Google Cloud Platform can I use for this?
I have looked into Google App Engine cron jobs but there seems to be no way to programmatically modify them. If possible I would like to avoid running a cron job every minute just to check if there is some task to run.
My framework of choice is ASP.NET Core but if there is a better solution available, e.g. in Java, I'm willing to try it out.
As you have found out, App Engine Cron Service does not have an API for programmatically managing cron tasks. Cron tasks are configured using a file called cron.yaml and this file can be programmatically modified and uploaded to google cron service(details). I'm not sure about exact requirements for your task scheduling service, bu this could be a good enough solution for your problem.
Another option would be to run a Google compute engine instance. As this is basically a virtual server maintained by you, you will have full control over it; allowing you to choose OS, backend/frontend technologies etc. For example you can run a Linux server, use an asp.net core backend to manage crontab tasks.
I would also recommend to take a look on Google Cloud Tasks. You can create queues and push/pull tasks to/from them:
Push queues run tasks by delivering HTTP requests to App Engine worker services. They dispatch these requests at a reliable, steady
rate and guarantee reliable task execution. Because you can control
the rate at which tasks are sent from the queue, you can control the
workers' scaling behavior and hence your costs.
Pull queues do not dispatch tasks at all. They depend on other worker services to "lease" tasks from the queue on their own
initiative. Pull queues give you more power and flexibility over when
and where tasks are processed, but they also require you to do more
process management. When a task is leased the leasing worker declares
a deadline. By the time the deadline arrives the worker must either
complete the task and delete it or the Task Queue service will allow
another worker to lease it.
Source: https://cloud.google.com/appengine/docs/standard/java/taskqueue/