As DAGs get deleted over time they will still appear on the web UI, so I'm interested in running a process to regularly remove inactive DAGs to keep the UI as uncluttered as possible.
I notice that in airflow.models.DagBag there is a method called deactivate_inactive_dags() that appears to be this functionality, but I see it is not called anywhere in code. How is this method intended to be used?
As far as I know, you can't delete DAGs that were previously run. The only way would be to delete the respective records from the airflow database.
Related
We are having google Airflow environment set for our needs.
I have read a lot on the stackoverflow but everyone is saying to restart your webserver, which I think I can not do, as it is managed by google.
All the time when we deploy new DAG into environment, its always like DAG is missing.
What is happening is it - it take few hours before everything work fine after deployment, but until that time its hard for me to understand what is wrong and how to fix that.
Could you please help me get rid of this issue permanently.
Please let me know if any more information required here. Thanks in advance.
Every dag_dir_list_interval, the DagFileProcessorManager process list the scripts in the dags folder, then if the script is new or processed since more than the min_file_process_interval, it creates a DagFileProcessorProcess process for the file to process it and generate the dags.
You can check what do you have as values for these variables, and reduce them to add the dag ASAP to the Metadata.
I have two containers in my meteor app on galaxy servers. I have some background jobs that I want to be executing only on a single container to avoid duplication.
What's the easiest way to achieve this? Is there some procId or the like that I can retrieve during runtime?
If the two servers have their own settings files, you could use a setting to nominate one of the servers as the one that does the background jobs.
There is a package called node-cron that can be used for setting up regular jobs https://www.npmjs.com/package/node-cron
Use a Meteor.startup method to look at the settings file, and if it is the designated server, it can schedule the jobs for itself.
Another technique is to work out the ID of each server, and have database entry containing the id of the nominated server.
On checking further, https://github.com/percolatestudio/meteor-synced-cron supports multiple servers, which should do what you need :)
I was thinking about this more... one solution (which I'm kind of borrowing from meteor-migrations) is to make a simple database collection/entry that holds a startupCodeHasRun flag. Then in a Meteor.startup() block you can check if the flag has been set, if not, set the flag and run the code. This would cause the code to only be run once on only one of your containers that share the same database.
The gotcha is you would have to reset this flag manually before redeploying or else the code would never be run again on redeploy.
Not an ideal solution but could work. This is the same way the above database migrations package works in a multi-container environment. And since it's a one-time database migration, you don't have to worry about redeploy.
I'm getting stock quotes which happen a few times a second. I'm using meteor to display those quotes. I don't think meteor can keep up with the database. What's the best way to keep meteor speedy?
Should I set my query to be non-reactive then somehow manually call when meteor should re-render the templates?
If you are updating your db directly (outside of meteor), than meteor will take 10 seconds to reload and recompute the data changes from the database.
If you are updating your db through a DDP connection, than those changes should reflect to your meteor UI almost instantaneously, certainly keeping up with a few updates per second.
Apart from this, your publish function should not expose all fields of a document in bulk. Instead, you should publish just what is necessary to populate your UI.
I have a project that uses Windows Workflow and I have a database in SQL Server 2008 with Workflow tables.
I want to Edit Record in Project so want to go back to a previous Activity without condition, and update the workflow instance in database.
How can I undo the previous step in Windows Workflow Foundation?
Not really. There is a bit if hack though that might be useful. When you use a workflow instance store and tell the workflow to abandon it's state when an unhandled exception occurs the last saved state in the instance store is unchanged. So you can reload the workflow from that state and continue as if the next activity never executed. The problem is in controlling when the workflow persist. You get to control that for the most part but there are some circumstances where a workflow will always persist.
I have a situation in which i am fetching top row from a temporary table and doing a functionality using that details. The temporary table will be updated with the new uploaded file details. I do have an aspx page which does this functionality. The web page is triggered using a task scheduler at some interval of time. My problem is that when the page is triggered
during a schedule if the functionality is not finished before the next trigger the task scheduler gets stuck and won't work unless a new schedule is created.
I have heard that executing the functionality using asp.net threading can solve my problem. Is that possible for my situation? If so any one please help me with code how i can proceed..
Now i just fetch only one top row to do the process. If i can implement this for top few rows (say top 5 rows) then i can increase the speed of my application. Please suggest a solution to implement this as well..
increase the time scheduler time a little bit to it can get completed before another starts.
Improve the update query, so it can save time.
Threading is good idea. But...
Each scheduler should check if the previous one is already running or not, you can put a status of (running/ completed) somewhere in table.
Log scheduler activity (start, end, elapsed time etc), it will help a lot.