Run multiple tasks in parallel - airflow

I know it is possible to run 1 set of tasks in parallel. For example, something like this:
begin >> [A, B, C, D,E] >> end
would run A, B, C, D, E all in parallel. However, I want to do something like this such that after begin, there are two workflows running in parallel. Something like:
A -> B -> C
begin -> -> end
D -> E -> F
What would be the correct syntax to achieve this?

Every split is like a starting point.
start_dag = start_task()
end_dag = end_task()
a = DummyOperator(task_id="a")
b = DummyOperator(task_id="b")
c = DummyOperator(task_id="c")
d = DummyOperator(task_id="d")
e = DummyOperator(task_id="e")
f = DummyOperator(task_id="f")
(start_dag >> [a, d])
(a >> b >> c >> end_dag)
(d >> e >> f >> end_dag)

Related

F# extended Euclidian algorithm with Continuations question

I am new to F# and haven't done functional programming since I was an undergraduate, but I've been trying teach myself. I wrote a naive recursive Extended Euclidean implementation, which works just fine, and am now trying again but with continuations.
I walked through the code by hand twice with a small example and got the correct answer, but when I run it through the interpreter I am not getting the same result, so I'm clearly misunderstanding something I am trying to do.
I ran eea 7 3 by hand, and the (correct) result I computed was (1, 1, -2)
But when I run it in the interpreter I get
eea 7 3;;
val it : int * int * int = (1, 0, 1)
Here's my implementation:
let eea a b =
let rec contEEA a b f =
match b with
| 0 -> f () (a,1,0)
| _ ->
contEEA b (a%b) (fun () t ->
let (d,x',y') = t
(d, y', x'-(y'*(a/b)))
)
contEEA a b (fun () t -> t)
For reference the naive approach, straight from a textbook, is
let rec eea_gcd a b =
match b with
| 0 -> (a, 1, 0)
| _ ->
let d, x', y' = eea_gcd b (a % b)
(d, y', x'-(y'*(a/b)))
Your continuation-based version is always doing exactly one iteration (the last one). When you make the recursive call, your continuation just straight up returns the result instead of "returning" it to the previous call by passing to the previous continuation.
So the call sequence goes like this:
eea 7 3
contEEA 7 3 (fun () t -> t)
b <> 0 ==> second case matches
contEEA 3 1 (fun () t -> ... (d, y', ...))
b <> 0 ==> second case matches
contEEA 1 0 (fun () t -> ... (d, y', ...))
b = 0 ==> first case matches
The continuation is called f () (1, 1, 0)
The continuation calculates result (1, 0, 1 - (0*(3/1)) = (1, 0, 1) and immediately returns it
What you want to do instead is when the first continuation calculates the result of (1, 0, 1) it should pass it to the previous continuation, so that it may carry on the calculations from there, ultimately passing the result to the very first continuation fun () t -> t, which returns it back to the consumer.
To do that, replace this line:
(d, y', x'-(y'*(a/b)))
With this:
f (d, y', x'-(y'*(a/b)))
Also, a few notes on some other aspects.
The first parameter of the continuation (the unit, ()) is not necessary, since it's never actually used (and how can it be?). You can lose it.
After removing the unit parameter, the first continuation becomes fun t -> t, which has a special name id (aka "the identity function")
Rather than destructure the triple with a let, you can do it right in the parameter declaration. Parameters can be patterns!
Applying all of the above, as well as the actual problem fix, here's a better version:
let eea a b =
let rec contEEA a b f =
match b with
| 0 -> f (a,1,0)
| _ ->
contEEA b (a%b) (fun (d,x',y') ->
f (d, y', x'-(y'*(a/b)))
)
contEEA a b id

Remove line-crossing in Graphviz; layout different in jupyter and webgraphviz

I'm looking for a way to remove an unacceptable line crossing in a simple dot script. There is no line-crossing when the code is executed in http://www.webgraphviz.com/, but there is a line-crossing when the exact same code is executed in a Jupyter notebook. I'd be grateful for a suggestion of how to remove the line-crossing in Jupyter, that is, to make the layout in Jupyter look like the layout in webgraphviz.
Here is the dot code:
Digraph X {
rankdir=LR;
{rank=same; a b}
{rank=same; d c}
z -> a
a -> b
b -> c
c -> d
d -> a
}
In http://www.webgraphviz.com/, it displays as follows:
But, in Jupyter, it displays as follows, with an unacceptable line-crossing:
The jupyter code is the following:
import graphviz
graphviz.Source('''Digraph X {
rankdir=LR;
{rank=same; a b}
{rank=same; d c}
z -> a
a -> b
b -> c
c -> d
d -> a
}''')
This took some trial and error and piecing together clues from https://graphviz.gitlab.io/_pages/doc/info/attrs.html
import graphviz
graphviz.Source('''Digraph X {
rankdir=RL;
{rank=sink; z}
{rank=same; c d}
{rank=same; a b}
z -> a
a -> b
b -> c [constraint=false]
c -> d [constraint=false]
d -> a
}''')

Ocaml - This expression has type 'a list but an expression was expected of type 'a The type variable 'a occurs inside 'a list

Why is the following code giving me that error?
Note that the is_sorted function returns either true or false
and make_move function returns a list of lists. e.g [[0,1,3,2],[1,0,2,3]]
let rec solve_helper b pos n r fn =
if n = 0 then b :: r :: fn (*fn is the final array with all paths*)
else match pos with
[] -> fn
|(h::t) -> if is_sorted h = true then h
else h :: r (* ERROR HERE: r is the temp array that contains 1 path*)
solve_helper b (make_moves h) (n-1) r
solve_helper b t (n-1) r (*tail recursion*)
;;
let solve_board b n = solver_helper b (make_moves b) n [] []
;;
new code:
let rec solve_helper b pos n r fn =
if n = 0 then r :: fn (*fn is the final array with all paths*)
else match pos with
[] -> fn
|(h::t) -> if is_sorted h = true then
let j = h :: r in
r :: fn
else
let u = h :: r in
let k = solve_helper b (make_moves h) (n - 1) r fn in
solve_helper b t (n - 1) r fn(*tail recursion*)
;;
let solve_board b n = solve_helper b (make_moves b) n [] []
;;
These lines of your code:
else h :: r (* ERROR HERE: r is the temp array that contains 1 path*)
solve_helper b (make_moves h) (n-1) r
solve_helper b t (n-1) r (*tail recursion*)
do not make sense as far as I can tell. They represent a call to a function named r with 10 arguments (two of which are the function r itself).
Possibly you need to edit your code to show exactly what the compiler is seeing.
If your code actually looks like this, you need to rethink this part. It reads like imperative code (a series of things to do) rather than functional code (an expression consisting of functions applied to arguments).

Ocaml nested if without else

Is it possible to have nested if without else statements. I wrote the following useless program to demonstrate nested ifs. How do I fix this so it's correct in terms of syntax. lines 5 and 6 gives errors.
let rec move_helper b sz r = match b with
[] -> r
|(h :: t) ->
if h = 0 then
if h - 1 = sz then h - 1 ::r
if h + 1 = sz then h + 1 ::r
else move_helper t sz r
;;
let move_pos b =
move_helper b 3 r
;;
let g = move_pos [0;8;7;6;5;4;3;2;1]
You can't have if without else unless the result of the expression is of type unit. This isn't the case for your code, so it's not possible.
Here's an example where the result is unit:
let f x =
if x land 1 <> 0 then print_string "1";
if x land 2 <> 0 then print_string "2";
if x land 4 <> 0 then print_string "4"
You must understand that if ... then is an expression like any other. If no else is present, it must be understood as if ... then ... else () and thus has type unit. To emphasize the fact that it is an expression, suppose you have two functions f and g of type, say, int → int. You can write
(if test then f else g) 1
You must also understand that x :: r does not change r at all, it constructs a new list putting x in front of r (the tail of this list is shared with the list r). In your case, the logic is not clear: what is the result when h=0 but the two if fail?
let rec move_helper b sz r = match b with
| [] -> r
| h :: t ->
if h = 0 then
if h - 1 = sz then (h - 1) :: r
else if h + 1 = sz then (h + 1) :: r
else (* What do you want to return here? *)
else move_helper t sz r
When you have a if, always put an else. Because when you don't put an else, Java will not know if the case is true or false.

Complex iterations in haskell

I have this complex iterations program I wrote in TI Basic to perform a basic iteration on a complex number and then give the magnitude of the result:
INPUT “SEED?”, C
INPUT “ITERATIONS?”, N
C→Z
For (I,1,N)
Z^2 + C → Z
DISP Z
DISP “MAGNITUDE”, sqrt ((real(Z)^2 + imag(Z)^2))
PAUSE
END
What I would like to do is make a Haskell version of this to wow my teacher in an assignment. I am still only learning and got this far:
fractal ::(RealFloat a) =>
(Complex a) -> (Integer a) -> [Complex a]
fractal c n | n == a = z : fractal (z^2 + c)
| otherwise = error "Finished"
What I don't know how to do is how to make it only iterate n times, so I wanted to have it count up a and then compare it to n to see if it had finished.
How would I go about this?
Newacct's answer shows the way:
fractal c n = take n $ iterate (\z -> z^2 + c) c
Iterate generates the infinite list of repeated applications.
Ex:
iterate (2*) 1 == [1, 2, 4, 8, 16, 32, ...]
Regarding the IO, you'll have to do some monadic computations.
import Data.Complex
import Control.Monad
fractal c n = take n $ iterate (\z -> z^2 + c) c
main :: IO ()
main = do
-- Print and read (you could even omit the type signatures here)
putStr "Seed: "
c <- readLn :: IO (Complex Double)
putStr "Number of iterations: "
n <- readLn :: IO Int
-- Working with each element the result list
forM_ (fractal c n) $ \current -> do
putStrLn $ show current
putStrLn $ "Magnitude: " ++ (show $ magnitude current)
Since Complex is convertible from and to strings by default, you can use readLn to read them from the console (format is Re :+ Im).
Edit: Just for fun, one could desugar the monadic syntax and type signatures which would compress the whole programm to this:
main =
(putStr "Seed: ") >> readLn >>= \c ->
(putStr "Number of iterations: ") >> readLn >>= \n ->
forM_ (take n $ iterate (\z -> z^2 + c) c) $ \current ->
putStrLn $ show current ++ "\nMagnitude: " ++ (show $ magnitude current)
Edit #2: Some Links related to plotting and Mandelbrot's sets.
Fractal plotter
Plotting with
Graphics.UI
Simplest solution
(ASCII-ART)
Well you can always generate an infinite list of results of repeated applications and take the first n of them using take. And the iterate function is useful for generating an infinite list of results of repeated applications.
If you'd like a list of values:
fractalList c n = fractalListHelper c c n
where
fractalListHelper z c 0 = []
fractalListHelper z c n = z : fractalListHelper (z^2 + c) c (n-1)
If you only care about the last result:
fractal c n = fractalHelper c c n
where
fractalHelper z c 0 = z
fractalHelper z c n = fractalHelper (z^2 + c) c (n-1)
Basically, in both cases you need a helper function to the counting and accumulation. Now I'm sure there's a better/less verbose way to do this, but I'm pretty much a Haskell newbie myself.
Edit: just for kicks, a foldr one-liner:
fractalFold c n = foldr (\c z -> z^2 + c) c (take n (repeat c))
(although, the (take n (repeat c)) thing seems kind of unnecessary, there has to be an even better way)

Resources