I have created a data pipeline in Airflow which helps to update the tickets in Freshdesk whenever some values are change in the tickets during business hours. My logic for updatding the ticket is working fine, but I want to make the trigger the dags whenever a event is occured in Freshdesk interface so that I don't need to manually trigger the dags.
There are automation rules in FreshDesk which provides the external webhook in https://companyname.freshdesk.com/api/v2/tickets/{{ticket.id}} where {ticket.id} is essentail to know the id for which the tickets are updated from the data pipeline.
I need to know how to connect the external trigger such as webhook so that we can trigger the data-pipeline automatically in Airflow.
My Airflow Version 1.10
Airflow doesn't have a mechanism that allows triggering DAG based on webhooks from other services.
That use case might be covered by AIP-35 Add Signal Based Scheduling To Airflow but this is currently a draft idea of enhancement to Airflow.
To achieve your use case with current functionality you will need to other application/service to handle the webhook fired by Freshdesk and triggering a new DAG run in Airflow with RestAPI Trigger new DAG run endpoint
You need to be aware of frequency. If the frequency of creating new DAG runs is too high you should consider switching to a batch solution (which Airflow is more suitable for)
I am not sure about Freshdesk, but according to this it sounds as if you're able to hit an external API with its automation mechanism.
If that's the case, then I think this is what you're looking for. That's the documentation for the REST API reference of Apache Airflow.
Airflow allows you to manage it via web API, but you first need to enable it in the configuration. You should set the following configuration to use at least basic auth.
Keep in mind that back then that was an experimental API and newer versions changed the URL schema and the API a bit. It's much more complete nowadays.
Related
I'm planning on using WordPress as a headless CMS for a project and using the REST API to retrieve content. I have a few cron tasks that need to run fairly regularly within WordPress and I want to know if API calls are enough to trigger them, or if I'll need to add that functionality myself.
Edit: when I say cron I mean wp_cron
Tasks scheduled through WP_Cron are evaluated to determine if the task should be run each time the init hook is executed [1]. The init hook should be fired on every request for a URL that WordPress is configured to respond to, including REST API calls (although I can't find a definitive source to cite for this).
As such, calls to WordPress REST API should indeed trigger any code scheduled via WP_Cron and which is due to be executed per its configuration.
I have so far been very impressed with the firebase platform for hosting a client-side single page app and for data storage. However, I have one component that I don't know where to host...
I want to have a background process that aperiodically updates the database. The nature of when an update is needed is based on an external source and, although the general timeframe of when updates are available is known, the exact timing is not. My thinking was to have a background task running that has some smarts to determine when an update is needed, and then trigger an update at that time.
I don't know where I would host something like this. I considered running it in a loop in a firebase function, but due to pricing model being based on time, that would get very expensive, and functions are not suited for daemon-type processes. The actual "database update" would be suitable for a function, but not the triggering logic. Also, I have seen functions-cron which does offload the triggering logic, but since my updates are not truly periodic, it doesn't seem exactly appropriate. I haven't looked too much into AppEngine and how that relates to the firebase platform...so basically my question:
What are the options for "reasonably-priced" hosting an always-running background task?
Google App Engine - Standard is something you want to look at more. It is reasonably priced since what you are doing will likely fit into GAE-Std's free daily quota. In GAE-Std, you create a scheduled cron job: GAE will call you task as if it was an incoming web request.
See Firebase doc for integrating with GAE
See GAE doc for cron jobs
I want to build a task scheduling service on the Google Cloud Platform. The tasks can be as simple as triggering a URL. Tasks can be recurring (once an hour, twice a day, every thursday, ...) and can be created and removed dynamically.
Which services/APIs on the Google Cloud Platform can I use for this?
I have looked into Google App Engine cron jobs but there seems to be no way to programmatically modify them. If possible I would like to avoid running a cron job every minute just to check if there is some task to run.
My framework of choice is ASP.NET Core but if there is a better solution available, e.g. in Java, I'm willing to try it out.
As you have found out, App Engine Cron Service does not have an API for programmatically managing cron tasks. Cron tasks are configured using a file called cron.yaml and this file can be programmatically modified and uploaded to google cron service(details). I'm not sure about exact requirements for your task scheduling service, bu this could be a good enough solution for your problem.
Another option would be to run a Google compute engine instance. As this is basically a virtual server maintained by you, you will have full control over it; allowing you to choose OS, backend/frontend technologies etc. For example you can run a Linux server, use an asp.net core backend to manage crontab tasks.
I would also recommend to take a look on Google Cloud Tasks. You can create queues and push/pull tasks to/from them:
Push queues run tasks by delivering HTTP requests to App Engine worker services. They dispatch these requests at a reliable, steady
rate and guarantee reliable task execution. Because you can control
the rate at which tasks are sent from the queue, you can control the
workers' scaling behavior and hence your costs.
Pull queues do not dispatch tasks at all. They depend on other worker services to "lease" tasks from the queue on their own
initiative. Pull queues give you more power and flexibility over when
and where tasks are processed, but they also require you to do more
process management. When a task is leased the leasing worker declares
a deadline. By the time the deadline arrives the worker must either
complete the task and delete it or the Task Queue service will allow
another worker to lease it.
Source: https://cloud.google.com/appengine/docs/standard/java/taskqueue/
I am working with an e-commerce platform, and I have a task to synchronize with some remote accounting software. The task requires syncing orders, products, inventory...etc. With large amounts of data being synced,the process can take awhile. So, I don't think asp.net application would be the best place to handle this. So, the requirements are:
To be able to schedule this process to run overnight
To be able to manually fire off this process and pass into it some variables like order numbers to export.
Possibly get back status info when fired off manually.
Has to work on .net 3.5
Issues: Can't use a windows service because the site is hosted remotely on a shared service, and the host won't allow a service.
Ideas: I'm having a really hard time finding the best way to handle this outside asp.net that fits all requirements, but I do have access to their FTP and thought possibly a console app that hosts a web-service may work, and I can put Quartz scheduler in global file to fire off service from the site.
Anyway, please offer some thoughts and experiences if you have them on which methods have worked for you.
Can't use a windows service because the site is hosted remotely on a shared service, and the host won't allow a service.
That might be a problem. Does this hosting service provide any other kind of scheduling functionality? If not then you may need to consider changing your hosting services.
You're correct in that ASP.NET is not the tool you'd use for scheduling tasks. A web application is a request/response system (and is very much at the mercy of the hosting process, IIS usually for ASP.NET). So you need some way to schedule the task to execute at regular intervals. Windows Services, Windows Task Scheduler, or some other task scheduling tool.
As for the requirement to be able to invoke the process manually, that's a simple matter of separating the invocation of the logic from the logic itself. Picture the following components:
A module which performs the logic, not bound to any UI or any way of invoking it. Basically a Class Library project (or part of one).
A Windows Service or Console Application which references the Class Library and invokes the logic.
A Web Application which references the Class Library and invokes the logic.
Once you've sorted out how to schedule the Console Application, just schedule it and it's all set. If the process returns some information then the Console Application can also perform any notifications necessary to inform people of that information.
The Web Application can then also have an interface somewhere to invoke the process manually. Since the process "can take a while" then of course you won't want the interface to wait for it to complete. This can result in timeouts and leave the system in an unknown state. Instead you'd want to return the UI to the user indicating that the process has started (or been queued) and that they will be notified with the results when it completes. There are a couple of options for this...
You can use a BackgroundWorker to actually invoke the process. When the process completes, send a notification to the user who invoked it.
You can write a record to a database table to "queue" the process and have something like a Windows Service or scheduled Console Application (same scenario as above) which regularly polls that table for queued tasks, performs the task, and sends the notification. (Of course updating the status in the table along the way so it doesn't perform it twice.)
There are pros and cons either way, it's really up to you how you'd like to proceed. Ultimately you're looking at two main things here:
Separate the logic itself from the scheduling/invocation of the logic.
Utilize a scheduling system to schedule tasks. (If your hosting provider doesn't have one, find one that does.)
I'm building a project to send messages to users. The client wants a way to schedule these messages to be sent out at a certain time, for example, he creates the message at 2am but wants it to be sent out at 10am without his intervention, where do I begin with this sort of thing? I'm using ASP.NET MVC3, any help is appreciated.
Update
Darin has suggested Quartz.net, I've finally gotten around to attempting to set it up. But I'm not really understanding how to implement it with my web app.
I'm assuming I should be able to make an httprequest from my service to an action on my webapp, triggered by quartz. But I'm not sure how to communicate between the webapp and this service, such as sending instructions to the quartz server.
So far, I've created a windows service, set up the installers, and added the Quartz.net server 2010 solution to my service project, am I on the right track?
Using a managed Windows Service with Quartz.NET or a console application which you would schedule with the Windows task scheduler seems like a good approaches to achieve that.
Welp, there are scheduled tasks... either make a localhost request at a specific time, or write an executable/service to be called.
A possible alternative if you can't use scheduled tasks (but may be dependent upon the website being available to the Internet) is to make a remote request-maker program or use a website monitoring service. Have either of those make a request to a particular page/querystring on your site periodically. Then, make the web application perform a particular task (send an email) whenever that resource is requested.
A few free monitoring services are limited to one request every hour or half-hour, or you can pay to have it checked more often. The resource's code could be made to record the message-sending event, (thus making them only get sent once, regardless of how often the request is made)