I'm new to elixir, I'm trying to find something similar to Python's ContextManager.
Problem:
I have a bunch of functions and I want to add latency metric around them.
Now we have:
def method_1 do
...
end
def method_2 do
...
end
... more methods
I'd like to have:
def method_1 do
start = System.monotonic_time()
...
end = System.monotonic_time()
emit_metric(end-start)
end
def method_2 do
start = System.monotonic_time()
...
end = System.monotonic_time()
emit_metric(end-start)
end
... more methods
Now code duplication is a problem
start = System.monotonic_time()
...
end = System.monotonic_time()
emit_metric(end-start)
So what is a better way to avoid code duplication in this case? I like the context manager idea in python. But now sure how I can achieve something similar in Elixir, thanks for the help in advance!
In Erlang/Elixir this is done through higher-order functions, take a look at BEAM telemetry. It is an Erlang and Elixir library/standard for collecting metrics and instrumenting your code - it is widely adopted by Pheonix, Ecto, cowboy and other libraries. Specifically, you'd be interested in :telemetry.span/3 function as it emits start time and duration measurements by default:
def some_function(args) do
:telemetry.span([:my_app, :my_function], %{metadata: "Some data"}, fn ->
result = do_some_work(args)
{result, %{more_metadata: "Some data here"}}
end)
end
def do_some_work(args) # actual work goes here
And then, in some other are of your code you listen to those events and log them/send them to APM:
:telemetry.attach_many(
"test-telemetry",
[[:my_app, :my_function, :start],
[:my_app, :my_function, :stop],
[:my_app, :my_function, :exception]],
fn event, measurements, metadata, config ->
# Handle the actual event.
end)
nil
)
I think the closest thing to python context manager would be to use higher order functions, i.e. functions taking a function as argument.
So you could have something like:
def measure(fun) do
start = System.monotonic_time()
result = fun.()
stop = System.monotonic_time()
emit_metric(stop - start)
result
end
And you could use it like:
measure(fn ->
do_stuff()
...
end)
Note: there are other similar instances where you would use a context manager in python that would be done in a similar way, on the top of my head: Django has a context manager for transactions but Ecto uses a higher order function for the same thing.
PS: to measure elapsed time, you probably want to use :timer.tc/1 though:
def measure(fun) do
{elapsed, result} = :timer.tc(fun)
emit_metric(elapsed)
result
end
There is actually a really nifty library called Decorator in which macros can be used to "wrap" your functions to do all sorts of things.
In your case, you could write a decorator module (thanks to #maciej-szlosarczyk for the telemetry example):
defmodule MyApp.Measurements do
use Decorator.Define, measure: 0
def measure(body, context) do
meta = Map.take(context, [:name, :module, :arity])
quote do
# Pass the metadata information about module/name/arity as metadata to be accessed later
:telemetry.span([:my_app, :measurements, :function_call], unquote(meta), fn ->
{unquote(body), %{}}
end)
end
end
end
You can set up a telemetry listener in your Application.start definition:
:telemetry.attach_many(
"my-app-measurements",
[[:my_app, :measurements, :function_call, :start],
[:my_app, :measurements, :function_call, :stop],
[:my_app, :measurements, :function_call, :exception]],
&MyApp.MeasurementHandler.handle_telemetry/4)
nil
)
Then in any module with a function call you'd like to measure, you can "decorate" the functions like so:
defmodule MyApp.Domain.DoCoolStuff do
use MyApp.Measurements
#decorate measure()
def awesome_function(a, b, c) do
# regular function logic
end
end
Although this example uses telemetry, you could just as easily print out the time difference within your decorator definition.
What’s the right way to export all the methods of thing from the submodules here (this doesn't work):
module Foo
module Bar
thing(x::String) = 1
end
import .Bar: thing
module Baz
thing(x::Int) = 1
end
import .Baz: thing
export thing
end
You have to make them the same function. thing can't mean two different things in the same namespace.
For instance:
module Foo
function thing end
module Bar
import ..thing
thing(x::String) = 1
end
module Baz
import ..thing
thing(x::Int) = 1
end
export thing
end
I have a custom DAG (meant to be subclassed), let's name it MyDAG. In the __enter__ method I want to add (or not) an operator based on the subclassing DAG. I'm not interested in using the BranchPythonOperator.
class MyDAG(DAG):
def __enter__(self, context):
start = DummyOperator(taks_id=start)
end = DummyOperator(task_id=end)
op = self.get_additional_operator()
if op:
start >> op
else:
start >> end
retrun self
def get_additional_operator(self):
# None if the subclass doesn't add any operator. A reference to another operator otherwise
if get_additional_operator is returning a reference, I'm obtaining this shape (two branches):
* start --> op
* end
otherwise, if it's returning None, I'm obtaining this (one branch):
* start --> end
What I want is not having end at all in the subclass inherting from MyDAG if get_additional_operator doesn't return None, something like this:
* start --> op
Instead of the two branches I'm obtaining above.
Airflow is somehow parsing every operator declared in the __enter__ method of a subclass of MyDAG. From that assumption, in order not to have an operator it only suffices to declare the operator in the right place. code below:
class MyDAG(DAG):
def __enter__(self, context):
start = DummyOperator(taks_id=start)
op = self.get_additional_operator()
if op:
start >> op
else:
end = DummyOperator(task_id=end)
start >> end
retrun self
def get_additional_operator(self):
# None if the subclass doesn't add any operator. A reference to another operator otherwise
The declaration of the end operator is made in the else section. I think it's only parsed when the else is evaluated to true.
In ruby 1.9, one could do this:
def foo
val = "a string"
def val.big
self.upcase
end
val
end
and one could call foo, with "a string" returned, or foo.big and get "A STRING" returned.
However this is no longer possible in Ruby 2.1, an error "cannot define singleton method" is thrown. How would the methods be implemented in 2.1 for foo and foo.big?
(Obviously this is a simplified example, what I need to do is implement a singleton method on the value returned by another method).
UPDATE: it appears that this IS possible in ruby 2.1, but NOT in the context of an ActiveRecord class!
SECOND UPDATE: When val is a string, this does work. My the problem surfaces when one tries to define a singleton class on a number like this:
def foo
val = 42
def val.smaller
self - 11
end
val
end
this throws the error "TypeError: can't define singleton"
I'd like to start that I've not been able to recreate my problem in a stripped down version of the code. The code below works as intended, so this post is perhaps not well posed. The extended code, which is too long to post here, fails. I'll describe what I'm trying to do as maybe it'll help someone else out there.
I create three types: Bar, Baz, and Qux, which contains the method foo on Bar and Baz. I create a qux and query it's foo
qux = Wubble.Qux()
qux.foo
I get the following two methods, as expected:
foo(bar::Bar)
foo(baz::Baz)
Then when I try to actually use qux.foo with a bar or a baz, it gives me an error 'foo' has no method matching foo(::Bar).
Sadly, I can't recreate this error with stripped down code and the real code is unattractively long. What are the various ways of getting this error in this scenario that I've missed? It may be related to method extension and function shadowing like in this post, but I couldn't work a fix.
module Wibble
type Bar
data::Number
function Bar(num::Number=0)
this = new(num)
return this
end
end
end
module Wobble
type Baz
data::String
function Baz(vec::String="a")
this = new(vec)
return this
end
end
end
module Wubble
using Wibble
using Wobble
typealias Bar Wibble.Bar
typealias Baz Wobble.Baz
type Qux
foo::Function
function Methods(this::Qux)
function foo(bar::Bar)
println("foo with bar says ", bar.data)
end
function foo(baz::Baz)
println("foo with baz says ", baz.data)
end
this.foo = foo
return this
end
function Qux()
this = new()
this = Methods(this)
return this
end
end
end
I'm not really sure what's going wrong, but a couple of points which might help
You almost never want to have Function field in a type: this is a common idiomatic mistake made by people coming from "dot-based" OOP languages. Julia methods are generic, and don't belong to a particular type. There is no advantage to doing this, and not only is it more confusing (you have a lot of nested levels to write something that could be written in 2 lines), but it can make it harder for the compiler to reason about types, impacting performance.
You should use import Wibble.Bar instead of a typealias. If you use this, you don't need using.
Outer constructors are easier to use for specifying default arguments.
So in short, my version would be:
module Wibble
type Bar
data::Number
end
Bar() = Bar(0)
end
module Wobble
type Baz
data::String
end
Baz() = Baz("a")
end
module Wubble
import Wibble.Bar
import Wobble.Baz
qux(bar::Bar) = println("foo with bar says ", bar.data)
qux(baz::Baz) = println("foo with baz says ", baz.data)
end