I have an airflow setup.
Some dags run very frequently (~ every 15 min) generating quite a bit of history.
I would like to automatically delete these successfull runs if they are a week old or so.
Any ideas how I can set this?
I've been looking into doing this via the terminal, however this doesn't seem to work.
d=`date -d 'last week' +%Y-%m-%d`
#d=`date -d 'yesterday' +%Y-%m-%d`
echo $d
airflow clear r_calc_daemon -e $d
However the clear command does not seem to clearing up anything.
In the gui there is a delete button for the runs.
But you can imagine I have to push a lot to clean it up like this.
any tips doing this? I guess this is problem a lot of people encounter?
Related
I have a backup-script that runs as a kubernetes cronjob every night and 3 nights ago the firebase auth:export started just looping. I can see that it exports everything fine and on the last row "Exported 57 account(s) successfully." it is normally done and writes the file to our bucket but now it just loops around and starts exporting again.
This is the command i use:
firebase auth:export firebase_auth_$(date +"%Y-%m-%d").json --project ${PROJECT_ID} --token "$(cat /var/secrets/firebase/token.txt)"
I have tried running the CLI locally and run into the same problem, i can't seem to find the issue when i google either. The documentation does not mention any property to loop or stop when all users are exported. Is there any flag/property like that or should this just work?
Turns out that this is a known issue:
https://github.com/firebase/firebase-tools/issues/1643
From the GitHub issue:
partially solved in 7.3.1 and probably a new fix coming due to #1642 in the next version.
So basically it's still a bit broken, even in 7.3.1. Better wait for the next version I guess.
I've tried running this a few different ways, but at the moment I have my crontab file as
MAILTO=email1#gmail.com,email2#gmail.com
40 10 * * * /home/me/my_script.sh
This my_script.sh is a shell script that runs an R script.
#!/bin/sh
/usr/bin/Rscript /home/me/stuff/daily_routine.R >> /home/me/stuff/output/outputstuff
If I just go to the terminal and run the following, though it works:
/home/me/my_script.sh
This R script downloads data from the internet. Could that be it? I've tried to include full paths everywhere, so I'm not sure that it's the PATH variable.
I never configured anything related to email, so I'm not sure how it is supposed to send it. Maybe that's it.
Edit:
It turned out that I had never installed and configured anything to send email. I ended up fixing the issue by installing and configuring ssmtp to send emails by using my gmail account. It took me a lot of fiddling around, so I'm not sure if I have a postable solution.
Also, I am not sure whether I should delete this question or keep it up. All of the cron tutorials I found are misleading--none of them mentioned having to do this.
So I wrote my father a neat little R script that pulls financial indicators on stocks, and outputs the info to a csv...
I would like to have it set up so that the script will run automatically once a day, skipping the weekends if possible. I looked around for awhile online and it seems as though the Mac "Automator" App is what I'm looking for.
However, after reading many guides and posts (like this one https://www.r-bloggers.com/how-to-source-an-r-script-automatically-on-a-mac-using-automator-and-ical/) I cannot get it to work...
In trying to replicate what this man did above I get the error that the first path is a directory; while the latter returns stuff like "cat: Rscript: No such file or directory"
So I was wondering if anyone could recommend either any good free software that will allow me to do what I would like, or how to run an R script from the /bin/bash shell
EDIT:
The suggested solution isn't really answering my problem. The issue is making this as easy as possibly for my dad to run, that way he doesn't have to do anything, specifically use the terminal. Ideally I could just schedule a task that repeats every morning, but the cronR package requires Daemon, and the others are just command line tools
I had a similar experience.
I created an automator calendar alarm
added a Execute AppleScript Action and used the following code:
on run {input, parameters}
try
tell application "R"
activate
with timeout of 90000 seconds
cmd "source(\"Dropbox/RScripts/CV19/liibre_coronabr.R\")"
end timeout
end tell
end try
return input
end run
When you save it, just choose the date and time for it to run and select the option to repeat everyday
That's it!
I have a cron job that launches an Rscript to go fetch some data and stash it away in a database. Sometimes the source isn't ready so it waits. Sometimes the source skips a data point so the script ends up waiting until another instance is started from cron. Now that two instances are running they cause problems with each other. Is there something like the following pseudo code that I could put at the top of my scripts so they stop when they see another instance of themselves is already running:
stopifnot(Sys.running('nameofscript.r'))
One thing I thought to do would be for the script to make a temp file with a fixed name at the start and then delete that temp file at the end. This way the script can check for the existence of the file to know if it's already running. I also see there's something called flock which is probably a better solution than that but I'd prefer if there was a way R can do it without bash commands (I've never really learned any bash). If there isn't a pure R way to do this, can the R script call flock on itself or would it only work if the cron task calls a bash script that then calls the rscript. If it isn't already obvious, I don't really know how to use flock.
Since I started using git I've had my RPROMPT setup to show the current branch. I've recently been using some of the "fancy" scripts to show un/staged file counts and other useful at a glance things. (https://github.com/olivierverdier/zsh-git-prompt/tree/master)
After using this for a week or two, its performance started to bother me.
Are there faster ways of getting this info or are there maybe ways to asynchronously write the RPROMPT? I don't want to wait to type a command while the RPROMPT is computed and would be perfectly happy with it popping in slightly later then my main PROMPT.
No offense to the aforementioned script; it's great. I'm just impatient.