Airflow : How to schedule a dag at different times? - airflow

I want to schedule a dag daily, but at different times in the day, for instance :
monday at 16H
tuesday at 9H
wednesday at 10H
...
monday at 15H
tuesday at 12H
etc...
How can I achieve this ?

If you want the dag to be scheduled randomly once a day, write python helper code.
In your dag code right before you define your dag, put a seeded random (that doesn't change with date) to create a pseudo rand.
In this example, I've converted the full date to numerical date but you can use whatever method you prefer.
Something Like the below code should work.
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
dag=Dag('TestDag,schedule_interval=randomCronString,default_args=args, catchup=False)
Edit
Airflow runs the dag every 5 seconds but by using a seeded random, you're forcing the random to only change when the seed changes (in this case when the day flips over), keep in mind though, in most systems airflow is on UTC.
If you run the following code block over and over
import random
import datetime
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
You get the following results
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *
random.seed(int(datetime.date.today().strftime('%Y%m%d')))
randomCronString="* {} * * *".format(random.randint(0,24))
print(randomCronString)
* 1 * * *

How about adding a delay-step (with randomized sleep interval), and schedule it before actually doing something?
E.g.
def _delay_dag():
import random
import time
delay_by = random.randint(0,60*60*6) #e.g. to delay up to 6h from schedule time
print(f'Delaying start by {delay_by} seconds..')
time.sleep(delay_by)
return True
def _do_something(ti, *kwargs):
pass
with DAG('my_DAG_with_random_start',
schedule_interval='#daily',
tags=['test','random_start'],
default_args=default_args) as dag:
delay_dag = PythonOperator(
task_id = 'delay_dag',
python_callable=_delay_dag
)
do_something = PythonOperator(
task_id = 'do_something',
provide_context=True,
python_callable=_do_something
)
delay_dag >> do_something

Related

Compute sequence of summed fractions in R

I want to compute the following sequence using R, without loops, i.e. for cycles.
1 + (2/3) + ((2/3)*(4/5)) + ((2/3)*(4/5)*(6/7)) + ... + ((2/3)*(4/5)*...(20/21))
So far, I tried different approaches with a sequence as well as a while function, but could not came up with a suitable solution. Help would be highly appreciated.
We may use cumprod
v1 <- seq(2, 20, by = 2)
v2 <- seq(3, 21, by = 2)
1 + sum(cumprod(v1/v2))
[1] 4.945724
-manual calculation
1 + (2/3) + ((2/3)*(4/5)) + ((2/3)*(4/5)*(6/7)) + ((2/3)*(4/5)*(6/7) * (8/9)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11) * (12/13)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11) * (12/13) * (14/15)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11) * (12/13) * (14/15) * (16/17)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11) * (12/13) * (14/15) * (16/17) * (18/19)) + ((2/3)*(4/5)*(6/7) * (8/9) * (10/11) * (12/13) * (14/15) * (16/17) * (18/19) * (20/21))
[1] 4.945724

I need this diamond shape pattern in R

*
* *
* * *
* * * *
* * * * *
* * * *
* * *
* *
*
This is my code so far, but it throws an error:
Error in " " * (rows - i - 1) : non-numeric argument to binary operator
Calls: pyramid -> print
Execution halted
#R version 3
pyramid<-function(rows){for (i in rows){print(" "*(rows-i-1)+"*"*(i+1))}
for(j in (rows-1|0|-1)){print(" "*(rows-j)+"*"*(j))}}
rows<-5
pyramid(rows)
You can find plenty of (pseudo-code) examples on the net. That should've been your first approach towards solving your problem. SO is not a free code writing service, and you'll get a much more positive response if you demonstrate a genuine attempt at solving the problem yourself.
That aside, here is a "crude" R implementation of a code example I found here. The code can and probably should be "R-ified", and I encourage you to spend some time doing so. I promise that you'll learn a lot. For example, it should be possible to replace most (all?) explicit for loops by making use of vectorised functions.
diamond <- function(max) {
# Upper triangle
space <- max - 1
for (i in 0:(max - 1)) {
for (j in 0:space) cat(" ")
for (j in 0:i) cat("* ")
cat("\n")
space <- space - 1
}
# Lower triangle
space = 1;
for (i in (max - 1):1) {
for (j in 0:space) cat(" ")
for (j in 0:(i - 1)) cat("* ")
cat("\n")
space <- space + 1
}
}
diamond(5)
# *
# * *
# * * *
# * * * *
#* * * * *
# * * * *
# * * *
# * *
# *

R: Turn content of string into body of function

I wrote a function, which returns a function based on certain properties.
The problem is that I had to use "paste", which makes the returned object a string. Here an example of the returned object:
the_problem <- "beta['v_p'] * 0.1 * ((3.99 * exp(-0.144 * time)) +
(4.78 * exp(-0.0111 * time))) + 0.1 * beta['ktrans_1'] * (3.99 * (exp(-
beta['kep_1']* time) - exp(- 0.144 * time)) * (0.144 -
beta['kep_1'])**(-1) + (4.78 * (exp(- beta['kep_1'] * time)- exp(-
0.0111 * time)) * (0.0111 - beta['kep_1'])**(-1)))"
I would like to manipulate the object "the_problem" to make it usable as function. Something like:
dcemri_func <- function(beta){
return(get.rid.of.string(the_problem))}
I already tried "as.function", but this does not work.
Any ideas?
Thanks and best whishes,
Chris
You could try using eval(parse(text = ..)), i.e.:
the_problem <- paste0('function(beta) ', the_problem)
dcemri_func <- eval(parse(text = the_problem))

Evaluating one variable with a list

I have the following code:
nth <- expression(((1/p)*a0/2)+sum(((1/p)*a*cos(i*pi*x/p)))+sum((1/p)*b*sin(i*pi*x/p)))
nth <- as.expression(gsub('pi',pi,nth))
nth <- as.expression(gsub('p',p,nth))
nth <- as.expression(gsub('a0',a0,nth))
nth <- as.expression(gsub('a',a,nth))
nth <- as.expression(gsub('b',b,nth))
this results to the expression:
"((1/1) * 1.26424111790395/2) + sum(((1/1) * 0.251688909862584 * cos(i * 3.14159265358979 * x/1))) + sum((1/1) * -1.03501509824516e-16 * sin(i * 3.14159265358979 * x/1))"
What I want to do next is to evaluate i with a list (eg. i = 1:3) without evaluating x. So what I want to get is something like:
"((1/1) * 1.26424111790395/2) + sum(((1/1) * 0.251688909862584 * cos(1 * 3.14159265358979 * x/1)), ((1/1) * 0.251688909862584 * cos(2 * 3.14159265358979 * x/1)), ((1/1) * 0.251688909862584 * cos(3 * 3.14159265358979 * x/1))) + sum(((1/1) * 0.251688909862584 * sin(1 * 3.14159265358979 * x/1)), ((1/1) * 0.251688909862584 * sin(2 * 3.14159265358979 * x/1)), ((1/1) * 0.251688909862584 * sin(3 * 3.14159265358979 * x/1)))"
How can I do this? Thanks.
Why not try this. You will see that I wrapped the values you needed in a loop and put all output in a new "finished" dataframe that will be updated as you roll through the loop. You can specify how many i's there are and change the expression as you wish:
# Define the initial variables that might be changed here
var_1 <- 3.14159265358979 # This referred to pi in your initial expression
var_2 <- 1 # This referred to p in your initial expression
var_3 <- 1.26424111790395 # This refers to a0 in your initial expression
var_4 <- 0.251688909862584 # This refers to a in your initial expression
var_5 <- -1.03501509824516e-16 # This refers to b in your initial expression
n <- 3 # This is the number of equations that will be run through
# Create an empty dataframe to hold the outputted expressions
finished = c() # Empty data frame
# Create an array holding values from 1 to the number of n's that will be run through
cycle <- c(1:n)
# Convert cycle to a matrix
cycle <- as.matrix(cycle)
# The variable we will be changing is i ... Create the initial loop
for (i in 1:3 ) {
nth <- expression(((1/p)*a0/2)+sum(((1/p)*a*cos(i*pi*x/p)))+sum((1/p)*b*sin(i*pi*x/p))) # Write the expression to be changed
# Substitute in all the relevant values. Note that this is made to be more explicity
nth <- as.expression(gsub('pi',var_1,nth))
nth <- as.expression(gsub('p',var_2,nth))
nth <- as.expression(gsub('a0',var_3,nth))
nth <- as.expression(gsub('a',var_4,nth))
nth <- as.expression(gsub('b',var_5,nth))
# I will also, for each value, substitue in relevant value from the cycle array
# This will change the i values for you
i_index <- cycle[i,1]
i_index <- as.character(i_index)
nth <- as.expression(gsub('i',i_index,nth)) # Append the nth equation
# I will then bind this solution into the finished data frame to hold all solutions
finished[i] = nth
}
This is the output that was generated after running the code:
expression("((1/1) * 1.26424111790395/2) + sum(((1/1) * 0.251688909862584 * cos(1 * 3.14159265358979 * x/1))) + sum((1/1) * -1.03501509824516e-16 * s1n(1 * 3.14159265358979 * x/1))",
"((1/1) * 1.26424111790395/2) + sum(((1/1) * 0.251688909862584 * cos(2 * 3.14159265358979 * x/1))) + sum((1/1) * -1.03501509824516e-16 * s2n(2 * 3.14159265358979 * x/1))",
"((1/1) * 1.26424111790395/2) + sum(((1/1) * 0.251688909862584 * cos(3 * 3.14159265358979 * x/1))) + sum((1/1) * -1.03501509824516e-16 * s3n(3 * 3.14159265358979 * x/1))")
This should get you going. You were definitely on the right track with gsub.
Notice that j in the original expression is being replaced with 1, 2, and 3, respectively. I went with j because gsub with i interferes with sin and pi. Hope it helps...
> expres <- "(((1/p)*a0/2)+sum(((1/p)*a*cos(j*pi*x/p)))+sum((1/p)*b*sin(j*pi*x/p)))"
> noquote(sapply(1:3, function(j){
GS <- gsub("j", as.numeric(j), expres)
paste0("expression(", GS, ")")
}))
[1] expression((((1/p)*a0/2)+sum(((1/p)*a*cos(1*pi*x/p)))+sum((1/p)*b*sin(1*pi*x/p))))
[2] expression((((1/p)*a0/2)+sum(((1/p)*a*cos(2*pi*x/p)))+sum((1/p)*b*sin(2*pi*x/p))))
[3] expression((((1/p)*a0/2)+sum(((1/p)*a*cos(3*pi*x/p)))+sum((1/p)*b*sin(3*pi*x/p))))
Another possible solution is to use substitute
g = function(i){
env = list(pi=pi, p=1, a0=1.26424111790395, a=0.251688909862584, b=-1.03501509824516e-16, i=i)
as.character(as.expression(substitute(((1/p)*a0/2)+sum(((1/p)*a*cos(i*pi*x/p)))+sum((1/p)*b*sin(i*pi*x/p)), env)))
}
paste(lapply(1:3, g), collapse=", ")

How can I generate ascii "graphical output" from R?

I believe R can generate stem-and-leaf for ascii histograms, and scatter plots using this code from Matt Shotwell.
Can it also generate ASCII based line graphs, like this from GNUPlot?
You should look at the fairly recent txtplot package. Currently, it includes scatterplot, line plot, density plot, acf, and bar chart.
From the online help,
> txtplot(cars[,1], cars[,2])
+----+------------+------------+-----------+------------+--+
120 + * +
| |
100 + +
| * * |
80 + * * +
| * * * |
60 + * * +
| * * * * * |
40 + * * * * * +
| * * * * * * * |
20 + * * * * * * * +
| * * * * |
| * * * |
0 +----+------------+------------+-----------+------------+--+
5 10 15 20 25
I know there is support for basic interaction between R and GnuPlot in the TeachingDemos package. Perhaps that can achieve what you want.

Resources