I am trying out a simple python program with decorator function. It is strange that the program only execute the print statement from the decorator function, but not the calling function main_print?
decor.py
def decorator(some_func):
def wrapper():
print('execute wrapper function from the decorator function')
return wrapper
#decorator
def main_print():
print('executing main_print')
main_print()
Output shows:
$ python3 decor.py
print('execute wrapper function from the decorator function')
I think you have to call the function inside the decorator eg
decor.py
def decorator(some_func):
def wrapper():
print('execute wrapper function from the decorator function')
some_func() # call the passed function
return wrapper
#decorator
def main_print():
print('executing main_print')
main_print()
Related
I am trying to implement basic ETL job, using Airflow, but stucked in one point:
I have 3 functions. And I want to define global variables for each of them like:
function a():
return a_result
function b():
use a
return b_result
function c():
use a and b
And then use these functions in python_callable.
Defining as usual global a_result is not working. Any solutions?
As I wrote in my comment,
When you return something in your python_callable, you can access the returned value if you pass the task context to the next operator. https://airflow.apache.org/concepts.html?highlight=xcom
The following is semi-pseudo code that illustrates the idea
# inside a PythonOperator called 'pushing_task'
def push_function():
return value
# inside another PythonOperator where provide_context=True
def pull_function(**context):
value = context['task_instance'].xcom_pull(task_ids='pushing_task')
pushing_task = PythonOperator('pushing_task',
push_function, ...)
pulling_task = PythonOperator('pulling_task',
pull_function,
provide_context=True ...)
Is there a way to print all of a function's variables? I want to build a custom debugging decorator, and can't seem to find what I'm looking for. I'm assuming there is some dunder method for this? So for a function:
def debugger(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
print(func.__funcVariables__) #Some dunder method that Prints all variables contained in func
return func
return wrapper
#debugger
def my_func():
x = 'foo'
y = 'bar'
I would want 'foo' and 'bar' printed to the console from the decorator. How can I achieve this?
It sounds like you're looking for function.__code__.co_varnames, which is a tuple of the names of the functions arguments and local variables. This is documented with the rest of the code introspection tools in the documentation for the inspect module
def debugger(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
print(func.__code__.co_varnames)
return func
return wrapper
#debugger
def my_func():
x = 'foo'
y = 'bar'
Imagine having a function, which handles a heavy computational job, that we wish to execute asynchronously in a Tornado application context. Moreover, we would like to lazily evaluate the function, by storing its results to the disk, and not rerunning the function twice for the same arguments.
Without caching the result (memoization) one would do the following:
def complex_computation(arguments):
...
return result
#gen.coroutine
def complex_computation_caller(arguments):
...
result = complex_computation(arguments)
raise gen.Return(result)
Assume to achieve function memoization, we choose Memory class from joblib. By simply decorating the function with #mem.cache the function can easily be memoized:
#mem.cache
def complex_computation(arguments):
...
return result
where mem can be something like mem = Memory(cachedir=get_cache_dir()).
Now consider combining the two, where we execute the computationally complex function on an executor:
class TaskRunner(object):
def __init__(self, loop=None, number_of_workers=1):
self.executor = futures.ThreadPoolExecutor(number_of_workers)
self.loop = loop or IOLoop.instance()
#run_on_executor
def run(self, func, *args, **kwargs):
return func(*args, **kwargs)
mem = Memory(cachedir=get_cache_dir())
_runner = TaskRunner(1)
#mem.cache
def complex_computation(arguments):
...
return result
#gen.coroutine
def complex_computation_caller(arguments):
result = yield _runner.run(complex_computation, arguments)
...
raise gen.Return(result)
So the first question is whether the aforementioned approach is technically correct?
Now let's consider the following scenario:
#gen.coroutine
def first_coroutine(arguments):
...
result = yield second_coroutine(arguments)
raise gen.Return(result)
#gen.coroutine
def second_coroutine(arguments):
...
result = yield third_coroutine(arguments)
raise gen.Return(result)
The second question is how one can memoize second_coroutine? Is it correct to do something like:
#gen.coroutine
def first_coroutine(arguments):
...
mem = Memory(cachedir=get_cache_dir())
mem_second_coroutine = mem(second_coroutine)
result = yield mem_second_coroutine(arguments)
raise gen.Return(result)
#gen.coroutine
def second_coroutine(arguments):
...
result = yield third_coroutine(arguments)
raise gen.Return(result)
[UPDATE I] Caching and reusing a function result in Tornado discusses using functools.lru_cache or repoze.lru.lru_cache as a solution for second question.
The Future objects returned by Tornado coroutines are reusable, so it generally works to use in-memory caches such as functools.lru_cache, as explained in this question. Just be sure to put the caching decorator before #gen.coroutine.
On-disk caching (which seems to be implied by the cachedir argument to Memory) is trickier, since Future objects cannot generally be written to disk. Your TaskRunner example should work, but it's doing something fundamentally different from the others because complex_calculation is not a coroutine. Your last example will not work, because it's trying to put the Future object in the cache.
Instead, if you want to cache things with a decorator, you'll need a decorator that wraps the inner coroutine with a second coroutine. Something like this:
def cached_coroutine(f):
#gen.coroutine
def wrapped(*args):
if args in cache:
return cache[args]
result = yield f(*args)
cache[args] = f
return result
return wrapped
I'm using pytest.mark to give my tests kwargs. However, if I use the same mark on both the class and a test within the class, the class's mark overrides the mark on the function when the same kwargs are used for both.
import pytest
animal = pytest.mark.animal
#animal(species='croc') # Mark the class with a kwarg
class TestClass(object):
#animal(species='hippo') # Mark the function with new kwarg
def test_function(self):
pass
#pytest.fixture(autouse=True) # Use a fixture to inspect my function
def animal_inspector(request):
print request.function.animal.kwargs # Show how the function object got marked
# prints {'species': 'croc'} but the function was marked with 'hippo'
Where'd my hippo go and how can I get him back?
There are unfortunately various pytest bugs related to this, I'm guessing you're running into one of them. The ones I found are related to subclassing which you don't do there though.
So I've been digging around in the pytest code and figured out why this is happening. The marks on the functions are applied to the function at import time but the class and module level marks don't get applied on the function level until test collection. Function marks happen first and add their kwargs to the function. Then class marks overwrite any same kwargs and module marks further overwrite any matching kwargs.
My solution was to simply create my own modified MarkDecorator that filters kwargs before they are added to the marks. Basically, whatever kwarg values get set first (which seems to always be by a function decorator) will always be the value on the mark. Ideally I think this functionality should be added in the MarkInfo class but since my code wasn't creating instances of that I went with what I was creating instances of: MarkDecorator. Note that I only change two lines from the source code (the bits about keys_to_add).
from _pytest.mark import istestfunc, MarkInfo
import inspect
class TestMarker(object): # Modified MarkDecorator class
def __init__(self, name, args=None, kwargs=None):
self.name = name
self.args = args or ()
self.kwargs = kwargs or {}
#property
def markname(self):
return self.name # for backward-compat (2.4.1 had this attr)
def __repr__(self):
d = self.__dict__.copy()
name = d.pop('name')
return "<MarkDecorator %r %r>" % (name, d)
def __call__(self, *args, **kwargs):
""" if passed a single callable argument: decorate it with mark info.
otherwise add *args/**kwargs in-place to mark information. """
if args and not kwargs:
func = args[0]
is_class = inspect.isclass(func)
if len(args) == 1 and (istestfunc(func) or is_class):
if is_class:
if hasattr(func, 'pytestmark'):
mark_list = func.pytestmark
if not isinstance(mark_list, list):
mark_list = [mark_list]
mark_list = mark_list + [self]
func.pytestmark = mark_list
else:
func.pytestmark = [self]
else:
holder = getattr(func, self.name, None)
if holder is None:
holder = MarkInfo(
self.name, self.args, self.kwargs
)
setattr(func, self.name, holder)
else:
# Don't set kwargs that already exist on the mark
keys_to_add = {key: value for key, value in self.kwargs.items() if key not in holder.kwargs}
holder.add(self.args, keys_to_add)
return func
kw = self.kwargs.copy()
kw.update(kwargs)
args = self.args + args
return self.__class__(self.name, args=args, kwargs=kw)
# Create my Mark instance. Note my modified mark class must be imported to be used
animal = TestMarker(name='animal')
# Apply it to class and function
#animal(species='croc') # Mark the class with a kwarg
class TestClass(object):
#animal(species='hippo') # Mark the function with new kwarg
def test_function(self):
pass
# Now prints {'species': 'hippo'} Yay!
I'm using Python because it's generally easy to read, but this is not a Python-specific question.
Take the following Python function strip_argument:
def strip_argument(func_with_no_args):
return lambda unused: func_with_no_args()
In use, I can pass a no-argument function to strip_argument, and it will return a function that accepts one argument that is never used. For example:
# some API I want to use
def set_click_event_listener(listener):
"""Args:
listener: function which will be passed the view that was clicked.
"""
# ...implementation...
# my code
def my_click_listener():
# I don't care about the view, so I don't want to make that an arg.
print "some view was clicked"
set_click_event_listener(strip_argument(my_click_listener))
Is there a standard name for the function strip_argument? I'm interested in any languages that have a function like this in the standard library.
Most functional programming languages offer a const function, that's a function that will always ignore it's first parameter and return it's second. If you pass a function to const that's exactly the behavior you described.
In Haskell you can use it like that:
f x = x + 1
g = const f
g 2 3 == 4 --2 is ignored and 3 is incremented
I have done a quick search for such a function in python but haven't found anything. It seems the standard is to use a lambda function as you did.