First-Class Functions
Functions in Python are first-class objects. Programming language theorists define a “first-class object” as a program entity that can be:
- Created at runtime
- Assigned to a variable or element in a data structure
- Passed as an argument to a function
- Returned as the result of a function
Treating a Function Like an Object
Example 5-1. Create and test a function, then read its __doc__ and check its type
>>> def factorial(n):
... '''returns n!'''
... return 1 if n < 2 else n * factorial(n-1)
...
>>> factorial(42)
1405006117752879898543142606244511569936384000000000
>>> factorial.__doc__
'returns n!'
>>> type(factorial)
<class 'function'>
Example 5-2 shows the “first class” nature of a function object. We can assign it a variable fact and call it through that name. We can also pass factorial as an argument to map.
>>> fact = factorial
>>> fact
<function factorial at 0x...>
>>> fact(5)
120
>>> map(factorial, range(11))
<map object at 0x...>
>>> list(map(fact, range(11)))
[1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800]
Higher-Order Functions
A function that takes a function as argument or returns a function as the result is a higher-order function. One example is map. Another is the builtin function sorted.
>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=len)
['fig', 'apple', 'cherry', 'banana', 'raspberry', 'strawberry']
>>> def reverse(word):
... return word[::-1]
>>> reverse('testing')
'gnitset'
>>> sorted(fruits, key=reverse)
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
Anonymous Functions
The lambda keyword creates an anonymous function within a Python expression.
>>> fruits = ['strawberry', 'fig', 'apple', 'cherry', 'raspberry', 'banana']
>>> sorted(fruits, key=lambda word: word[::-1])
['banana', 'apple', 'fig', 'raspberry', 'strawberry', 'cherry']
The Seven Flavors of Callable Objects
The call operator (i.e., ()) may be applied to other objects beyond user-defined functions. To determine whether an object is callable, use the callable() built-in function. The Python Data Model documentation lists seven callable types:
- User-defined functions: Created with def statements or lambda expressions.
- Built-in functions
- Built-in methods
- Methods: Functions defined in the body of a class.
- Classes: When invoked, a class runs its __new__ method to create an instance, then __init__ to initialize it, and finally the instance is returned to the caller. Because there is no new operator in Python, calling a class is like calling a function.
- Class instances: If a class defines a __call__ method, then its instances may be invoked as functions.
- Generator functions: Functions or methods that use the yield keyword. When called, generator functions return a generator object.
Given the variety of existing callable types in Python, the safest way to determine whether an object is callable is to use the callable() built-in.
>>> abs, str, 13
(<built-in function abs>, <class 'str'>, 13)
>>> [callable(obj) for obj in (abs, str, 13)]
[True, True, False]
User-Defined Callable Types
Not only are Python functions real objects, but arbitrary Python objects may also be made to behave like functions. Implementing a __call__ instance method is all it takes.
Example 5-8. bingocall.py: A BingoCage does one thing: picks items from a shuffled list.
import random
class BingoCage:
def __init__(self, items):
self._items = list(items)
random.shuffle(self._items)
def pick(self):
try:
return self._items.pop()
except IndexError:
raise LookupError('pick from empty BingoCage')
# Shortcut to bingo.pick(): bingo().
def __call__(self):
return self.pick()
>>> bingo = BingoCage(range(3))
>>> bingo.pick()
1
>>> bingo()
0
>>> callable(bingo)
True
Function Introspection
Function objects have many attributes beyond __doc__. See what the dir function reveals about our factorial.
>>> dir(factorial)
['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
Like the instances of a plain user-defined class, a function uses the __dict__ attribute to store user attributes assigned to it.
def upper_case_name(obj):
return ("%s %s" % (obj.first_name, obj.last_name)).upper()
upper_case_name.short_description = 'Customer name'
From Positional to Keyword-Only Parameters
One of the best features of Python functions is the extremely flexible parameter handling mechanism, enhanced with keyword-only arguments in Python 3.
Example 5-10. tag generates HTML; a keyword-only argument cls is used to pass “class” attributes as a workaround because class is a keyword in Python
def tag(name, *content, cls=None, **attrs):
"""Generate one or more HTML tags"""
if cls is not None:
attrs['class'] = cls
if attrs:
attr_str = ''.join(' %s="%s"' % (attr, value)
for attr, value
in sorted(attrs.items()))
else:
attr_str = ''
if content:
return '\n'.join('<%s%s>%s</%s>' %
(name, attr_str, c, name) for c in content)
else:
return '<%s%s />' % (name, attr_str)
>>> tag('br')
'<br />'
>>> tag('p', 'hello')
'<p>hello</p>'
>>> print(tag('p', 'hello', 'world'))
<p>hello</p>
<p>world</p>
>>> tag('p', 'hello', id=33)
'<p id="33">hello</p>'
>>> print(tag('p', 'hello', 'world', cls='sidebar'))
<p class="sidebar">hello</p>
<p class="sidebar">world</p>
>>> tag(content='testing', name="img")
'<img content="testing" />'
>>> my_tag = {'name': 'img', 'title': 'Sunset Boulevard',
... 'src': 'sunset.jpg', 'cls': 'framed'}
>>> tag(**my_tag)
'<img class="framed" src="sunset.jpg" title="Sunset Boulevard" />'
Function Annotations
Python 3 provides syntax to attach metadata to the parameters of a function declaration and its return value.
Example 5-19. Annotated clip function
def clip(text:str, max_len:'int > 0'=80) -> str:
"""Return text clipped at the last space before or after max_len
"""
end = None
if len(text) > max_len:
space_before = text.rfind(' ', 0, max_len)
if space_before >= 0:
end = space_before
else:
space_after = text.rfind(' ', max_len)
if space_after >= 0:
end = space_after
if end is None: # no spaces were found
end = len(text)
return text[:end].rstrip()
>>> from clip_annot import clip
>>> clip.__annotations__
{'text': <class 'str'>, 'max_len': 'int > 0', 'return': <class 'str'>}
Packages for Functional Programming
The operator Module
Often in functional programming it is convenient to use an arithmetic operator as a function.
Example 5-21. Factorial implemented with reduce and an anonymous function
from functools import reduce
def fact(n):
return reduce(lambda a, b: a*b, range(1, n+1))
To save you the trouble of writing trivial anonymous functions like lambda a, b: a*b, the operator module provides function equivalents for dozens of arithmetic operators.
Example 5-22. Factorial implemented with reduce and operator.mul
from functools import reduce
from operator import mul
def fact(n):
return reduce(mul, range(1, n+1))
Of the remaining operator functions, methodcaller is the last we will cover. The function it creates calls a method by name on the object given as argument.
Example 5-25. Demo of methodcaller: second test shows the binding of extra arguments
>>> from operator import methodcaller
>>> s = 'The time has come'
>>> upcase = methodcaller('upper')
>>> upcase(s)
'THE TIME HAS COME'
>>> hiphenate = methodcaller('replace', ' ', '-')
>>> hiphenate(s)
'The-time-has-come'
Freezing Arguments with functools.partial
The functools module brings together a handful of higher-order functions. The best known of them is probably reduce. Of the remaining functions in functools, the most
useful is partial and its variation, partialmethod.
functools.partial is a higher-order function that allows partial application of a function. Given a function, a partial application produces a new callable with some of the arguments of the original function fixed.
Example 5-26. Using partial to use a two-argument function where a one-argument callable is required
>>> from operator import mul
>>> from functools import partial
>>> triple = partial(mul, 3)
>>> triple(7)
21
>>> list(map(triple, range(1, 10)))
[3, 6, 9, 12, 15, 18, 21, 24, 27]
A more useful example involves the unicode.normalize function.
Example 5-27. Building a convenient Unicode normalizing function with partial
>>> import unicodedata, functools
>>> nfc = functools.partial(unicodedata.normalize, 'NFC')
>>> s1 = 'café'
>>> s2 = 'cafe\u0301'
>>> s1, s2
('café', 'café')
>>> s1 == s2
False
>>> nfc(s1) == nfc(s2)
True
Design Patterns with First-Class Functions
Case Study: Refactoring Strategy
Classic Strategy
The UML class diagram for the Strategy pattern is depicted in Figure. Its participants are:
- Context: Provides a service by delegating some computation to interchangeable components that implement alternative algorithms. In the ecommerce example, the context is an Order, which is configured to apply a promotional discount according to one of several algorithms.
- Strategy: The interface common to the components that implement the different algorithms. In our example, this role is played by an abstract class called Promotion.
- Concrete Strategy: One of the concrete subclasses of Strategy. FidelityPromo, BulkPromo, and Large OrderPromo are the three concrete strategies implemented.
from abc import ABC, abstractmethod
from collections import namedtuple
Customer = namedtuple('Customer', 'name fidelity')
class LineItem:
def __init__(self, product, quantity, price):
self.product = product
self.quantity = quantity
self.price = price
def total(self):
return self.price * self.quantity
class Order: # the Context
def __init__(self, customer, cart, promotion=None):
self.customer = customer
self.cart = list(cart)
self.promotion = promotion
def total(self):
if not hasattr(self, '__total'):
self.__total = sum(item.total() for item in self.cart)
return self.__total
def due(self):
if self.promotion is None:
discount = 0
else:
discount = self.promotion.discount(self)
return self.total() - discount
def __repr__(self):
fmt = '<Order total: {:.2f} due: {:.2f}>'
return fmt.format(self.total(), self.due())
class Promotion(ABC): # the Strategy: an Abstract Base Class
@abstractmethod
def discount(self, order):
"""Return discount as a positive dollar amount"""
class FidelityPromo(Promotion): # first Concrete Strategy
"""5% discount for customers with 1000 or more fidelity points"""
def discount(self, order):
return order.total() * .05 if order.customer.fidelity >= 1000 else 0
class BulkItemPromo(Promotion): # second Concrete Strategy
"""10% discount for each LineItem with 20 or more units"""
def discount(self, order):
discount = 0
for item in order.cart:
if item.quantity >= 20:
discount += item.total() * .1
return discount
class LargeOrderPromo(Promotion): # third Concrete Strategy
"""7% discount for orders with 10 or more distinct items"""
def discount(self, order):
distinct_items = {item.product for item in order.cart}
if len(distinct_items) >= 10:
return order.total() * .07
return 0
Note that in this Example, I coded Promotion as an abstract base class (ABC), to be able to use the @abstractmethod decorator, thus making the pattern more explicit.
>>> joe = Customer('John Doe', 0)
>>> ann = Customer('Ann Smith', 1100)
>>> cart = [LineItem('banana', 4, .5),
... LineItem('apple', 10, 1.5),
... LineItem('watermellon', 5, 5.0)]
>>> Order(joe, cart, FidelityPromo())
<Order total: 42.00 due: 42.00>
>>> Order(ann, cart, FidelityPromo())
<Order total: 42.00 due: 39.90>
>>> banana_cart = [LineItem('banana', 30, .5),
... LineItem('apple', 10, 1.5)]
>>> Order(joe, banana_cart, BulkItemPromo())
<Order total: 30.00 due: 28.50>
>>> long_order = [LineItem(str(item_code), 1, 1.0)
... for item_code in range(10)]
>>> Order(joe, long_order, LargeOrderPromo())
<Order total: 10.00 due: 9.30>
>>> Order(joe, cart, LargeOrderPromo())
<Order total: 42.00 due: 42.00>
Function-Oriented Strategy
Each concrete strategy in this Example is a class with a single method, discount. Furthermore, the strategy instances have no state (no instance attributes). You could say they look a lot like plain functions, and you would be right.
from collections import namedtuple
Customer = namedtuple('Customer', 'name fidelity')
class LineItem:
def __init__(self, product, quantity, price):
self.product = product
self.quantity = quantity
self.price = price
def total(self):
return self.price * self.quantity
class Order: # the Context
def __init__(self, customer, cart, promotion=None):
self.customer = customer
self.cart = list(cart)
self.promotion = promotion
def total(self):
if not hasattr(self, '__total'):
self.__total = sum(item.total() for item in self.cart)
return self.__total
def due(self):
if self.promotion is None:
discount = 0
else:
discount = self.promotion(self)
return self.total() - discount
def __repr__(self):
fmt = '<Order total: {:.2f} due: {:.2f}>'
return fmt.format(self.total(), self.due())
def fidelity_promo(order):
"""5% discount for customers with 1000 or more fidelity points"""
return order.total() * .05 if order.customer.fidelity >= 1000 else 0
def bulk_item_promo(order):
"""10% discount for each LineItem with 20 or more units"""
discount = 0
for item in order.cart:
if item.quantity >= 20:
discount += item.total() * .1
return discount
def large_order_promo(order):
"""7% discount for orders with 10 or more distinct items"""
distinct_items = {item.product for item in order.cart}
if len(distinct_items) >= 10:
return order.total() * .07
return 0
>>> joe = Customer('John Doe', 0)
>>> ann = Customer('Ann Smith', 1100)
>>> cart = [LineItem('banana', 4, .5),
... LineItem('apple', 10, 1.5),
... LineItem('watermellon', 5, 5.0)]
>>> Order(joe, cart, fidelity_promo)
<Order total: 42.00 due: 42.00>
>>> Order(ann, cart, fidelity_promo)
<Order total: 42.00 due: 39.90>
>>> banana_cart = [LineItem('banana', 30, .5),
... LineItem('apple', 10, 1.5)]
>>> Order(joe, banana_cart, bulk_item_promo)
<Order total: 30.00 due: 28.50>
>>> long_order = [LineItem(str(item_code), 1, 1.0)
... for item_code in range(10)]
>>> Order(joe, long_order, large_order_promo)
<Order total: 10.00 due: 9.30>
>>> Order(joe, cart, large_order_promo)
<Order total: 42.00 due: 42.00>
Choosing the Best Strategy: Simple Approach
The implementation of best_promo is very simple.
promos = [fidelity_promo, bulk_item_promo, large_order_promo]
def best_promo(order):
"""Select best discount available
"""
return max(promo(order) for promo in promos)
>>> Order(joe, long_order, best_promo)
<Order total: 10.00 due: 9.30>
>>> Order(joe, banana_cart, best_promo)
<Order total: 30.00 due: 28.50>
>>> Order(ann, cart, best_promo)
<Order total: 42.00 due: 39.90>
Finding Strategies in a Module
Modules in Python are also first-class objects, and the standard library provides several functions to handle them. The built-in globals is described as follows in the Python docs.
- globals(): Return a dictionary representing the current global symbol table. This is always the dictionary of the current module.
Example 6-7. The promos list is built by introspection of the module global namespace
promos = [globals()[name] for name in globals()
if name.endswith('_promo')
and name != 'best_promo']
def best_promo(order):
"""Select best discount available
"""
return max(promo(order) for promo in promos)
Another way of collecting the available promotions would be to create a module and put all the strategy functions there, except for best_promo.
In Example 6-8, the only significant change is that the list of strategy functions is built by introspection of a separate module called promotions. Note that Example 6-8 depends
on importing the promotions module as well as inspect, which provides highlevel introspection functions.
Example 6-8. The promos list is built by introspection of a new promotions module
promos = [func for name, func in
inspect.getmembers(promotions, inspect.isfunction)]
def best_promo(order):
"""Select best discount available
"""
return max(promo(order) for promo in promos)
The function inspect.getmembers returns the attributes of an object—in this case, the promotions module—optionally filtered by a predicate (a boolean function). We use inspect.isfunction to get only the functions from the module.
Command
Command is another design pattern that can be simplified by the use of functions passed as arguments.
Example 6-9. Each instance of MacroCommand has an internal list of commands.
class MacroCommand:
"""A command that executes a list of commands"""
def __init__(self, commands):
self.commands = list(commands)
def __call__(self):
for command in self.commands:
command()
Further Reading
As far as I know, Learning Python Design Patterns, by Gennadiy Zlobin (Packt), is the only book entirely devoted to patterns in Python.
Expert Python Programming by Tarek Ziadé (Packt) is one of the best intermediate-level Python books in the market, and its final chapter, “Useful Design Patterns,” presents seven of the classic patterns from a Pythonic perspective.
Function Decorators and Closures
Function decorators let us “mark” functions in the source code to enhance their behavior in some way. This is powerful stuff, but mastering it requires understanding closures.
One of the newest reserved keywords in Python is nonlocal, introduced in Python 3.0.
Decorators 101
A decorator is a callable that takes another function as argument.
In other words, assuming an existing decorator named decorate, this code:
@decorate
def target():
print('running target()')
Has the same effect as writing this:
def target():
print('running target()')
target = decorate(target)
Example 7-1. A decorator usually replaces a function with a different one
>>> def deco(func):
... def inner():
... print('running inner()')
... return inner
...
>>> @deco
... def target():
... print('running target()')
...
>>> target()
running inner()
>>> target
<function deco.<locals>.inner at 0x10063b598>
To summarize: the first crucial fact about decorators is that they have the power to replace the decorated function with a different one. The second crucial fact is that they
are executed immediately when a module is loaded.
When Python Executes Decorators
A key feature of decorators is that they run right after the decorated function is defined. That is usually at import time (i.e., when a module is loaded by Python).
registry = [] # registry will hold references to functions decorated by @register.
def register(func): # register takes a function as argument.
print('running register(%s)' % func)
registry.append(func)
return func
@register
def f1():
print('running f1()')
@register
def f2():
print('running f2()')
def f3():
print('running f3()')
def main(): # main displays the registry, then calls f1(), f2(), and f3().
print('running main()')
print('registry ->', registry)
f1()
f2()
f3()
if __name__=='__main__':
main()
The output of running registration.py as a script looks like this:
$ python3 registration.py
running register(<function f1 at 0x100631bf8>)
running register(<function f2 at 0x100631c80>)
running main()
registry -> [<function f1 at 0x100631bf8>, <function f2 at 0x100631c80>]
running f1()
running f2()
running f3()
Note that register runs (twice) before any other function in the module.
If registration.py is imported (and not run as a script), the output is this:
>>> import registration
running register(<function f1 at 0x10063b1e0>)
running register(<function f2 at 0x10063b268>)
The main point of Example 7-2 is to emphasize that function decorators are executed as soon as the module is imported, but the decorated functions only run when they are explicitly invoked. This highlights the difference between what Pythonistas call import time and runtime.
Considering how decorators are commonly employed in real code, Example 7-2 is unusual in two ways:
- The decorator function is defined in the same module as the decorated functions. A real decorator is usually defined in one module and applied to functions in other modules.
- The register decorator returns the same function passed as argument. In practice, most decorators define an inner function and return it.
Decorator-Enhanced Strategy Pattern
Example 7-3. The promos list is filled by the promotion decorator
promos = []
def promotion(promo_func):
promos.append(promo_func)
return promo_func
@promotion
def fidelity(order):
"""5% discount for customers with 1000 or more fidelity points"""
return order.total() * .05 if order.customer.fidelity >= 1000 else 0
@promotion
def bulk_item(order):
"""10% discount for each LineItem with 20 or more units"""
discount = 0
for item in order.cart:
if item.quantity >= 20:
discount += item.total() * .1
return discount
@promotion
def large_order(order):
"""7% discount for orders with 10 or more distinct items"""
distinct_items = {item.product for item in order.cart}
if len(distinct_items) >= 10:
return order.total() * .07
return 0
def best_promo(order):
"""Select best discount available
"""
return max(promo(order) for promo in promos)
Variable Scope Rules
Example 7-5. Variable b is local, because it is assigned a value in the body of the function
>>> def f1(a):
... print(a)
... print(b)
...
>>> b = 6
>>> f1(3)
3
6
>>> b = 6
>>> def f2(a):
... print(a)
... print(b)
... b = 9
...
>>> f2(3)
3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in f2
UnboundLocalError: local variable 'b' referenced before assignment
When Python compiles the body of the function, it decides that b is a local variable because it is assigned within the function. The generated bytecode reflects this decision and will try to fetch b from the local environment.
If we want the interpreter to treat b as a global variable in spite of the assignment within the function, we use the global declaration:
>>> def f3(a):
... global b
... print(a)
... print(b)
... b = 9
...
>>> f3(3)
3
6
>>> b
9
Closures
A closure is a function with an extended scope that encompasses nonglobal variables referenced in the body of the function but not defined there.
It does not matter whether the function is anonymous or not; what matters is that it can access nonglobal variables that are defined outside of its body.
For starters, Example 7-8 is a class-based implementation. Example 7-8. average_oo.py: A class to calculate a running average
class Averager():
def __init__(self):
self.series = []
def __call__(self, new_value):
self.series.append(new_value)
total = sum(self.series)
return total/len(self.series)
>>> avg = Averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0
Now, Example 7-9 is a functional implementation. Example 7-9. average.py: A higher-order function to calculate a running average.
def make_averager():
series = []
def averager(new_value):
series.append(new_value)
total = sum(series)
return total/len(series)
return averager
When invoked, make_averager returns an averager function object. Each time an averager is called, it appends the passed argument to the series, and computes the current average.
>>> avg = make_averager()
>>> avg(10)
10.0
>>> avg(11)
10.5
>>> avg(12)
11.0
Note the similarities of the examples: we call Averager() or make_averager() to get a callable object avg that will update the historical series and calculate the current mean.
Note that series is a local variable of make_averager because the initialization series = [ ] happens in the body of that function. But when avg(10) is called, make_averager has already returned, and its local scope is long gone.
Within averager, series is a free variable. This is a technical term meaning a variable that is not bound in the local scope.
Example 7-11. Inspecting the function created by make_averager in Example 7-9.
>>> avg.__code__.co_varnames
('new_value', 'total')
>>> avg.__code__.co_freevars
('series',)
To summarize: a closure is a function that retains the bindings of the free variables that exist when the function is defined, so that they can be used later when the function is invoked and the defining scope is no longer available.
The nonlocal Declaration
Our previous implementation of make_averager was not efficient. A better implementation would just store the total and the number of items so far, and compute the mean from these two numbers.
Example 7-13 is a broken implementation, just to make a point. Example 7-13. A broken higher-order function to calculate a running average without keeping all history
def make_averager():
count = 0
total = 0
def averager(new_value):
count += 1
total += new_value
return total / count
return averager
>>> avg = make_averager()
>>> avg(10)
Traceback (most recent call last):
...
UnboundLocalError: local variable 'count' referenced before assignment
The problem is that the statement count += 1 actually means the same as count = count + 1, when count is a number or any immutable type. So we are actually assigning to count in the body of averager, and that makes it a local variable. The same problem affects the total variable.
We did not have this problem in Example 7-9 because we never assigned to the series name; we only called series.append and invoked sum and len on it. So we took advantage of the fact that lists are mutable.
But with immutable types like numbers, strings, tuples, etc., all you can do is read, but never update. If you try to rebind them, as in count = count + 1, then you are implicitly creating a local variable count. It is no longer a free variable, and therefore it is not saved in the closure.
To work around this, the nonlocal declaration was introduced in Python 3. It lets you flag a variable as a free variable even when it is assigned a new value within the function.
Example 7-14. Calculate a running average without keeping all history (fixed with the use of nonlocal)
def make_averager():
count = 0
total = 0
def averager(new_value):
nonlocal count, total
count += 1
total += new_value
return total / count
return averager
Implementing a Simple Decorator
Example 7-15. A simple decorator to output the running time of functions
import time
def clock(func):
def clocked(*args):
t0 = time.time()
result = func(*args)
elapsed = time.time() - t0
name = func.__name__
arg_str = ', '.join(repr(arg) for arg in args)
print('[%0.8fs] %s(%s) -> %r' % (elapsed, name, arg_str, result))
return result
return clocked
Example 7-16. Using the clock decorator
import time
from clockdeco import clock
@clock
def snooze(seconds):
time.sleep(seconds)
@clock
def factorial(n):
return 1 if n < 2 else n*factorial(n-1)
if __name__=='__main__':
print('*' * 40, 'Calling snooze(.123)')
snooze(.123)
print('*' * 40, 'Calling factorial(6)')
print('6! =', factorial(6))
The output of running Example 7-16 looks like this:
$ python3 clockdeco_demo.py
**************************************** Calling snooze(123)
[0.12405610s] snooze(.123) -> None
**************************************** Calling factorial(6)
[0.00000191s] factorial(1) -> 1
[0.00004911s] factorial(2) -> 2
[0.00008488s] factorial(3) -> 6
[0.00013208s] factorial(4) -> 24
[0.00019193s] factorial(5) -> 120
[0.00026107s] factorial(6) -> 720
6! = 720
This is the typical behavior of a decorator: it replaces the decorated function with a new function that accepts the same arguments and (usually) returns whatever the decorated function was supposed to return, while also doing some extra processing.
The clock decorator implemented in Example 7-15 has a few shortcomings: it does not support keyword arguments, and it masks the __name__ and __doc__ of the decorated function.
Example 7-17 uses the functools.wraps decorator to copy the relevant attributes from func to clocked. Also, in this new version, keyword arguments are correctly handled.
Example 7-17. An improved clock decorator
import time
import functools
def clock(func):
@functools.wraps(func)
def clocked(*args, **kwargs):
t0 = time.time()
result = func(*args, **kwargs)
elapsed = time.time() - t0
name = func.__name__
arg_lst = []
if args:
arg_lst.append(', '.join(repr(arg) for arg in args))
if kwargs:
pairs = ['%s=%r' % (k, w) for k, w in sorted(kwargs.items())]
arg_lst.append(', '.join(pairs))
arg_str = ', '.join(arg_lst)
print('[%0.8fs] %s(%s) -> %r ' % (elapsed, name, arg_str, result))
return result
return clocked
Decorators in the Standard Library
Python has three built-in functions that are designed to decorate methods: property, classmethod, and staticmethod.
Another frequently seen decorator is functools.wraps, a helper for building wellbehaved decorators.
Two of the most interesting decorators in the standard library are lru_cache and the brand-new singledispatch (added in Python 3.4).
Memoization with functools.lru_cache
A very practical decorator is functools.lru_cache. It implements memoization: an optimization technique that works by saving the results of previous invocations of an expensive function, avoiding repeat computations on previously used arguments.
A good demonstration is to apply lru_cache to the painfully slow recursive function to generate the nth number in the Fibonacci sequence.
Example 7-18. The very costly recursive way to compute the nth number in the Fibonacci series.
from clockdeco import clock
@clock
def fibonacci(n):
if n < 2:
return n
return fibonacci(n-2) + fibonacci(n-1)
if __name__=='__main__':
print(fibonacci(6))
Here is the result of running fibo_demo.py.
$ python3 fibo_demo.py
[0.00000095s] fibonacci(0) -> 0
[0.00000095s] fibonacci(1) -> 1
[0.00007892s] fibonacci(2) -> 1
[0.00000095s] fibonacci(1) -> 1
[0.00000095s] fibonacci(0) -> 0
[0.00000095s] fibonacci(1) -> 1
[0.00003815s] fibonacci(2) -> 1
[0.00007391s] fibonacci(3) -> 2
[0.00018883s] fibonacci(4) -> 3
[0.00000000s] fibonacci(1) -> 1
[0.00000095s] fibonacci(0) -> 0
[0.00000119s] fibonacci(1) -> 1
[0.00004911s] fibonacci(2) -> 1
[0.00009704s] fibonacci(3) -> 2
[0.00000000s] fibonacci(0) -> 0
[0.00000000s] fibonacci(1) -> 1
[0.00002694s] fibonacci(2) -> 1
[0.00000095s] fibonacci(1) -> 1
[0.00000095s] fibonacci(0) -> 0
[0.00000095s] fibonacci(1) -> 1
[0.00005102s] fibonacci(2) -> 1
[0.00008917s] fibonacci(3) -> 2
[0.00015593s] fibonacci(4) -> 3
[0.00029993s] fibonacci(5) -> 5
[0.00052810s] fibonacci(6) -> 8
8
The waste is obvious: fibonacci(1) is called eight times, fibonacci(2) five times, etc. But if we just add two lines to use lru_cache, performance is much improved.
Example 7-19. Faster implementation using caching
import functools
from clockdeco import clock
@functools.lru_cache()
@clock # This is an example of stacked decorators: @lru_cache() is applied on the function returned by @clock.
def fibonacci(n):
if n < 2:
return n
return fibonacci(n-2) + fibonacci(n-1)
if __name__=='__main__':
print(fibonacci(6))
Execution time is halved, and the function is called only once for each value of n:
$ python3 fibo_demo_lru.py
[0.00000119s] fibonacci(0) -> 0
[0.00000119s] fibonacci(1) -> 1
[0.00010800s] fibonacci(2) -> 1
[0.00000787s] fibonacci(3) -> 2
[0.00016093s] fibonacci(4) -> 3
[0.00001216s] fibonacci(5) -> 5
[0.00025296s] fibonacci(6) -> 8
Besides making silly recursive algorithms viable, lru_cache really shines in applications that need to fetch information from the Web.
Generic Functions with Single Dispatch
Imaging we want to be able to generate HTML displays for different types of Python objects.
import html
def htmlize(obj):
content = html.escape(repr(obj))
return '<pre>{}</pre>'.format(content)
That will work for any Python type, but now we want to extend it to generate custom displays for some types: str, int and list.
Because we don’t have method or function overloading in Python, we can’t create variations of htmlize with different signatures for each data type we want to handle differently.
A common solution in Python would be to turn htmlize into a dispatch function, with a chain of if/elif/elif calling specialized functions like htmlize_str, htmlize_int, etc. This is not extensible by users of our module, and is unwieldy: over time, the htmlize dispatcher would become too big, and the coupling between it and the specialized functions would be very tight.
The new functools.singledispatch decorator in Python 3.4 allows each module to contribute to the overall solution, and lets you easily provide a specialized function even for classes that you can’t edit. If you decorate a plain function with @singledispatch, it becomes a generic function: a group of functions to perform the same operation in different ways, depending on the type of the first argument.
Example 7-21. singledispatch creates a custom htmlize.register to bundle several functions into a generic function
from functools import singledispatch
from collections import abc
import numbers
import html
@singledispatch # @singledispatch marks the base function that handles the object type.
def htmlize(obj):
content = html.escape(repr(obj))
return '<pre>{}</pre>'.format(content)
@htmlize.register(str) # Each specialized function is decorated with @«base_function».register(«type»)
def _(text): # The name of the specialized functions is irrelevant; _ is a good choice to make this clear.
content = html.escape(text).replace('\n', '<br>\n')
return '<p>{0}</p>'.format(content)
@htmlize.register(numbers.Integral) # numbers.Integral is a virtual superclass of int
def _(n):
return '<pre>{0} (0x{0:x})</pre>'.format(n)
@htmlize.register(tuple) # You can stack several register decorators to support different types with the same function.
@htmlize.register(abc.MutableSequence)
def _(seq):
inner = '</li>\n<li>'.join(htmlize(item) for item in seq)
return '<ul>\n<li>' + inner + '</li>\n</ul>'
When possible, register the specialized functions to handle ABCs (abstract classes) such as numbers.Integral and abc.MutableSequence instead of concrete implementations like int and list. This allows your code to support a greater variety of compatible types.
Using ABCs for type checking allows your code to support existing or future classes that are either actual or virtual subclasses of those ABCs.
singledispatch is a well-thought-out addition to the standard library, and it offers more features than we can describe here. The best documentation for it is PEP 443 — Single-dispatch generic functions.
Stacked Decorators
When two decorators @d1 and @d2 are applied to a function f in that order, the result is the same as f = d1(d2(f)).
Parameterized Decorators
A Parameterized Registration Decorator
Conceptually, the new register function is not a decorator but a decorator factory. When called, it returns the actual decorator that will be applied to the target function.
Example 7-23. To accept parameters, the new register decorator must be called as a function.
registry = set() # registry is now a set, so adding and removing functions is faster.
def register(active=True): # register takes an optional keyword argument.
def decorate(func): # The decorate inner function is the actual decorator; note how it takes a function as argument.
print('running register(active=%s)->decorate(%s)'
% (active, func))
if active:
registry.add(func)
else:
registry.discard(func)
return func
return decorate
@register(active=False)
def f1():
print('running f1()')
@register()
def f2():
print('running f2()')
def f3():
print('running f3()')
>>> import registration_param
running register(active=False)->decorate(<function f1 at 0x10063c1e0>)
running register(active=True)->decorate(<function f2 at 0x10063c268>)
>>> registration_param.registry
[<function f2 at 0x10063c268>]
The Parameterized Clock Decorator
import time
DEFAULT_FMT = '[{elapsed:0.8f}s] {name}({args}) -> {result}'
def clock(fmt=DEFAULT_FMT): # clock is our parameterized decorator factory.
def decorate(func): # decorate is the actual decorator.
def clocked(*_args):
t0 = time.time()
_result = func(*_args) # _result is the actual result of the decorated function.
elapsed = time.time() - t0
name = func.__name__
args = ', '.join(repr(arg) for arg in _args) # _args holds the actual arguments of clocked, while args is str used for display.
result = repr(_result)
print(fmt.format(**locals()))
return _result
return clocked
return decorate
if __name__ == '__main__':
@clock()
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)
Example 7-26. clockdeco_param_demo1.py
import time
from clockdeco_param import clock
@clock('{name}: {elapsed}s')
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)
Output of Example 7-26:
$ python3 clockdeco_param_demo1.py
snooze: 0.12414693832397461s
snooze: 0.1241159439086914s
snooze: 0.12412118911743164s
Example 7-27. clockdeco_param_demo2.py
import time
from clockdeco_param import clock
@clock('{name}({args}) dt={elapsed:0.3f}s')
def snooze(seconds):
time.sleep(seconds)
for i in range(3):
snooze(.123)
Output of Example 7-27:
$ python3 clockdeco_param_demo2.py
snooze(0.123) dt=0.124s
snooze(0.123) dt=0.124s
snooze(0.123) dt=0.124s
Graham Dumpleton and Lennart Regebro—one of this book’s technical reviewers—argue that decorators are best coded as classes implementing __call__, and not as functions like the examples in this chapter.
Further Reading
Chapter 9, “Metaprogramming,” of the Python Cookbook, Third Edition by David Beazley and Brian K. Jones (O’Reilly), has several recipes from elementary decorators to very sophisticated ones.
Graham Dumpleton has a series of in-depth blog posts about techniques for implementing well-behaved decorators.
