Chapter 3: Algorithms

Python includes several modules for implementing algorithms elegantly and concisely using whatever style is most appropriate for the task. It supports purely procedural, object-oriented, and functional styles. All three styles are frequently mixed within different parts of the same program.

functools includes functions for creating function decorators, enabling aspect-oriented programming and code reuse beyond what a traditional object-oriented approach supports. It also provides a class decorator for implementing all rich comparison APIs using a shortcut and partial objects for creating references to functions with their arguments included.

The itertools module includes functions for creating and working with iterators and generators used in functional programming. The operator module eliminates the need for many trivial lambda functions when using a functional programming style by providing function-based interfaces to built-in operations, such as arithmetic or item lookup.

contextlib makes resource management easier, more reliable, and more concise for all programming styles. Combining context managers and the with statement reduces the number of try:finally blocks and indentation levels needed, while ensuring that files, sockets, database transactions, and other resources are closed and released at the right time.

3.1 functools—Tools for Manipulating Functions

Purpose Functions that operate on other functions.

Python Version 2.5 and later

The functools module provides tools for adapting or extending functions and other callable objects, without completely rewriting them.

3.1.1 Decorators

The primary tool supplied by the functools module is the class partial, which can be used to “wrap” a callable object with default arguments. The resulting object is itself callable and can be treated as though it is the original function. It takes all the same arguments as the original, and it can be invoked with extra positional or named arguments as well. A partial can be used instead of a lambda to provide default arguments to a function, while leaving some arguments unspecified.

Partial Objects

This example shows two simple partial objects for the function myfunc(). The output of show_details() includes the func, args, and keywords attributes of the partial object.

import functools 

def myfunc(a, b=2): 
  """Docstring for myfunc().""" 
  print ’ called myfunc with:’, (a, b) 
  return 

def show_details(name, f, is_partial=False): 
    """Show details of a callable object.""" 
    print ’%s:’ % name 

    print ’ object:’,f 

    if not is_partial: 
       print ’ __name__:’, f.__name__ 
    if is_partial:
       print ’ func:’, f.func 
       print ’ args:’, f.args 
       print ’ keywords:’, f.keywords 
     return 
show_details(’myfunc’, myfunc) 
myfunc(’a’, 3)
print 

# Set a different default value for ’b’, but require 
# the caller to provide ’a’.
p1 = functools.partial(myfunc, b=4) 
show_details(’partial with named default’, p1, True) 
p1(’passing a’) 
p1(’override b’, b=5)
print 

# Set default values for both ’a’ and ’b’. 
p2 = functools.partial(myfunc, ’default a’, b=99) 
show_details(’partial with defaults’, p2, True) 
p2() 
p2(b=’override b’)
print 

print ’Insufficient arguments:’ 
p1() 

At the end of the example, the first partial created is invoked without passing a value for a, causing an exception.

$ python functools_partial.py 

myfunc:
 object: <function myfunc at 0x100d9bf50>
 __name__: myfunc
 called myfunc with: (’a’, 3) 
partial with named default:
 object: <functools.partial object at 0x100d993c0>
 func: <function myfunc at 0x100d9bf50>
 args: ()
 keywords: {’b’: 4}
 called myfunc with: (’passing a’, 4)
 called myfunc with: (’override b’, 5) 
partial with defaults:
 object: <functools.partial object at 0x100d99418>
 func: <function myfunc at 0x100d9bf50>
 args: (’default a’,)
 keywords: {’b’: 99}
 called myfunc with: (’default a’, 99)
 called myfunc with: (’default a’, ’override b’) 
Insufficient arguments:
Traceback (most recent call last): 
File "functools_partial.py", line 51, in <module>
 p1() 
TypeError: myfunc() takes at least 1 argument (1 given) 

Acquiring Function Properties

The partial object does not have __name__ or __doc__ attributes by default, and without those attributes, decorated functions are more difficult to debug. Using update_wrapper() copies or adds attributes from the original function to the partial object.

import functools 

def myfunc(a, b=2): 
    """Docstring for myfunc()."""
    print ’ called myfunc with:’, (a, b)
    return 

def show_details(name, f): 
   """Show details of a callable object."""
   print ’%s:’ % name
   print ’ object:’,f
   print ’ __name__:’,
   try: 
      print f.__name__
   except AttributeError: 
      print ’(no __name__)’
   print ’ __doc__’, repr(f.__doc__)
   print
   return 

show_details(’myfunc’, myfunc) 

p1 = functools.partial(myfunc, b=4) 
show_details(’raw wrapper’, p1) 

print ’Updating wrapper:’
print ’ assign:’, functools.WRAPPER_ASSIGNMENTS
print ’ update:’, functools.WRAPPER_UPDATES
print 

functools.update_wrapper(p1, myfunc) 
show_details(’updated wrapper’, p1) 

The attributes added to the wrapper are defined in WRAPPER_ASSIGNMENTS, while WRAPPER_UPDATES lists values to be modified.

$ python functools_update_wrapper.py 

myfunc:
 object: <function myfunc at 0x100da2050>
 __name__: myfunc
 __doc__ ’Docstring for myfunc().’ 

raw wrapper:
 object: <functools.partial object at 0x100d993c0>
 __name__: (no __name__)
 __doc__ ’partial(func, *args, **keywords) -new function with parti 
al application\n of the given arguments and keywords.\n’ 

Updating wrapper:
 assign: (’__module__’, ’__name__’, ’__doc__’)
 update: (’__dict__’,) 

updated wrapper:
 object: <functools.partial object at 0x100d993c0>
 __name__: myfunc
 __doc__ ’Docstring for myfunc().’ 

Other Callables

Partials work with any callable object, not just with stand-alone functions.

import functools 

class MyClass(object): 
   """Demonstration class for functools""" 

   def method1(self, a, b=2): 
      """Docstring for method1().""" 
      print ’ called method1 with:’, (self, a, b)
      return 

   def method2(self, c, d=5): 
      """Docstring for method2""" 
       print ’ called method2 with:’, (self, c, d)
       return 
       wrapped_method2 = functools.partial(method2, ’wrapped c’)
       functools.update_wrapper(wrapped_method2, method2) 
  
     def __call__(self, e, f=6): 
       """Docstring for MyClass.__call__"""
        print ’ called object with:’, (self, e, f)
        return 

def show_details(name, f): 
    """Show details of a callable object."""
     print ’%s:’ % name
     print ’ object:’,f
     print ’ __name__:’,
     try: 
        print f.__name__
     except AttributeError: 
        print ’(no __name__)’
     print ’ __doc__’, repr(f.__doc__)
     return 

o = MyClass() 

show_details(’method1 straight’, o.method1)
o.method1(’no default for a’, b=3) 
print 

p1 = functools.partial(o.method1, b=4) 
functools.update_wrapper(p1, o.method1) 
show_details(’method1 wrapper’, p1) 
p1(’a goes here’) 
print 

show_details(’method2’, o.method2)
o.method2(’no default for c’, d=6) 
print 

show_details(’wrapped method2’, o.wrapped_method2) 
o.wrapped_method2(’no default for c’, d=6) 
print 

show_details(’instance’, o) 
o(’no default for e’) 
print 
p2 = functools.partial(o, f=7) 
show_details(’instance wrapper’, p2) 
p2(’e goes here’) 

This example creates partials from an instance and methods of an instance.

$ python functools_method.py 

method1 straight: 
  object: <bound method MyClass.method1 of <__main__.MyClass object
at 0x100da3550>> 
  __name__: method1 
  __doc__ ’Docstring for method1().’ 
  called method1 with: (<__main__.MyClass object at 0x100da3550>, ’n 
o default for a’, 3) 

method1 wrapper: 
  object: <functools.partial object at 0x100d99470> 
  __name__: method1 
  __doc__ ’Docstring for method1().’ 
  called method1 with: (<__main__.MyClass object at 0x100da3550>, ’a 
 goes here’, 4) 

method2: 
  object: <bound method MyClass.method2 of <__main__.MyClass object
at 0x100da3550>> 
  __name__: method2 
  __doc__ ’Docstring for method2’ 
  called method2 with: (<__main__.MyClass object at 0x100da3550>, ’n 
o default for c’, 6) 

wrapped method2: 
  object: <functools.partial object at 0x100d993c0> 
  __name__: method2 
  __doc__ ’Docstring for method2’ 
  called method2 with: (’wrapped c’, ’no default for c’, 6) 

instance: 
  object: <__main__.MyClass object at 0x100da3550> 
  __name__: (no __name__) 
  __doc__ ’Demonstration class for functools’ 
  called object with: (<__main__.MyClass object at 0x100da3550>, ’no 
    default for e’, 6) 
  instance wrapper:
    object: <functools.partial object at 0x100d994c8>
    __name__: (no __name__) 
    __doc__ ’partial(func, *args, **keywords) -new function with part 
  ial application\n of the given arguments and keywords.\n’
    called object with: (<__main__.MyClass object at 0x100da3550>, ’e
  goes here’, 7) 

Acquiring Function Properties for Decorators

Updating the properties of a wrapped callable is especially useful when used in a decorator, since the transformed function ends up with properties of the original “bare” function.

import functools 

def	show_details(name, f): 
      """Show details of a callable object.""" 
      print ’%s:’ % name 
      print ’ object:’,f 
      print ’ __name__:’, 
      try: 
           print f.__name__ 
      except AttributeError: 
           print ’(no __name__)’ 
      print ’ __doc__’, repr(f.__doc__) 
      print 
      return 

def	simple_decorator(f): 
      @functools.wraps(f)
      def decorated(a=’decorated defaults’, b=1): 
          print ’ decorated:’, (a, b) 
          print ’’,
          f(a, b=b) 
          return 
      return decorated 

def	myfunc(a, b=2): 
     "myfunc() is not complicated" 
      print ’ myfunc:’, (a,b) 
      return 

# The raw function 
show_details(’myfunc’, myfunc) 
myfunc(’unwrapped, default b’) 
myfunc(’unwrapped, passing b’, 3) 
print 

# Wrap explicitly 
wrapped_myfunc = simple_decorator(myfunc) 
show_details(’wrapped_myfunc’, wrapped_myfunc) 
wrapped_myfunc() 
wrapped_myfunc(’args to wrapped’, 4) 
print 

# Wrap with decorator syntax 
@simple_decorator 
def decorated_myfunc(a, b): 
    myfunc(a, b) 
    return 

show_details(’decorated_myfunc’, decorated_myfunc) 
decorated_myfunc() 
decorated_myfunc(’args to decorated’, 4) 

functools provides a decorator, wraps(), that applies update_wrapper() to the decorated function.

$ python functools_wraps.py 


myfunc:
 object: <function myfunc at 0x100da3488>
 __name__: myfunc
 __doc__ ’myfunc() is not complicated’ 

 myfunc: (’unwrapped, default b’, 2)
 myfunc: (’unwrapped, passing b’, 3) 

wrapped_myfunc:
 object: <function myfunc at 0x100da3500>
 __name__: myfunc
 __doc__ ’myfunc() is not complicated’ 

decorated: (’decorated defaults’, 1)
   myfunc: (’decorated defaults’, 1) 
decorated: (’args to wrapped’, 4)
   myfunc: (’args to wrapped’, 4) 

decorated_myfunc:
 object: <function decorated_myfunc at 0x100da35f0>
 __name__: decorated_myfunc
 __doc__ None 

 decorated: (’decorated defaults’, 1)
    myfunc: (’decorated defaults’, 1)
 decorated: (’args to decorated’, 4)
    myfunc: (’args to decorated’, 4) 

3.1.2 Comparison

Under Python 2, classes can define a __cmp__() method that returns -1, 0, or 1 based on whether the object is less than, equal to, or greater than the item being compared. Python 2.1 introduces the rich comparison methods API (__lt__(), __le__(), __eq__(), __ne__(), __gt__(), and __ge__()), which perform a single comparison operation and return a Boolean value. Python 3 deprecated __cmp__() in favor of these new methods, so functools provides tools to make it easier to write Python 2 classes that comply with the new comparison requirements in Python 3.

Rich Comparison

The rich comparison API is designed to allow classes with complex comparisons to implement each test in the most efficient way possible. However, for classes where comparison is relatively simple, there is no point in manually creating each of the rich comparison methods. The total_ordering() class decorator takes a class that provides some of the methods and adds the rest of them.

import functools 
import inspect 
from pprint import pprint 

@functools.total_ordering 
class MyObject(object):
    def __init__(self, val):
        self.val = val 
    def __eq__(self, other): 
        print ’ testing __eq__(%s, %s)’ % (self.val, other.val) 
        return self.val == other.val 
    def __gt__(self, other):
        print ’ testing __gt__(%s, %s)’ % (self.val, other.val) 
        return self.val > other.val 

print ’Methods:\n’ 
pprint(inspect.getmembers(MyObject, inspect.ismethod)) 

a = MyObject(1) 
b = MyObject(2) 

print ’\nComparisons:’ 
for	expr in [ ’a < b’, ’a <= b’, ’a == b’, ’a >= b’, ’a > b’ ]: 
      print ’\n%-6s:’ % expr 
      result = eval(expr) 
      print ’ result of %s: <%s’ % (expr, result) 

The class must provide implementation of __eq__() and one other rich comparison method. The decorator adds implementations of the rest of the methods that work by using the comparisons provided.

$	python functools_total_ordering.py 

Methods: 


[(’__eq__’, <unbound method MyObject.__eq__>), 
 (’__ge__’, <unbound method MyObject.__ge__>), 
 (’__gt__’, <unbound method MyObject.__gt__>), 
 (’__init__’, <unbound method MyObject.__init__>), 
 (’__le__’, <unbound method MyObject.__le__>), 
 (’__lt__’, <unbound method MyObject.__lt__>)] 

Comparisons: 

a < b: 
 testing __gt__(2, 1) 
 result of a < b: True 

a <= b:
 testing __gt__(1, 2)
 result of a <= b: True 

a == b: 
 testing __eq__(1, 2)
 result of a == b: False 

a >= b:
 testing __gt__(2, 1)
 result of a >= b: False 

a > b:
 testing __gt__(1, 2)
 result of a > b: False 

Collation Order

Since old-style comparison functions are deprecated in Python 3, the cmp argument to functions like sort() is also no longer supported. Python 2 programs that use comparison functions can use cmp_to_key() to convert them to a function that returns a collation key, which is used to determine the position in the final sequence.

import functools
 
class MyObject(object):
   def __init__(self, val):
       self.val = val 
   def __str__(self): 
      return ’MyObject(%s)’ % self.val 

def compare_obj(a, b): 
    """Old-style comparison function.
    """ 
    print ’comparing %s and %s’ % (a, b) 
    return cmp(a.val, b.val) 

#	Make a key function using cmp_to_key() 
get_key = functools.cmp_to_key(compare_obj) 

def get_key_wrapper(o): 
   """Wrapper function for get_key to allow for print statements.
   """ 
   new_key = get_key(o) 
   print ’key_wrapper(%s) -> %s’ % (o, new_key)
   return new_key 
objs = [ MyObject(x) for x in xrange(5, 0, -1) ] 

for	o in sorted(objs, key=get_key_wrapper):
 print o 

Normally, cmp_to_key() would be used directly, but in this example, an extra wrapper function is introduced to print out more information as the key function is being called.

The output shows that sorted() starts by calling get_key_wrapper() for each item in the sequence to produce a key. The keys returned by cmp_to_key() are instances of a class defined in functools that implements the rich comparison API using the old-style comparison function passed in. After all keys are created, the sequence is sorted by comparing the keys.

$ python functools_cmp_to_key.py 

key_wrapper(MyObject(5)) -> <functools.K object at 0x100da2a50> 
key_wrapper(MyObject(4)) -> <functools.K object at 0x100da2a90> 
key_wrapper(MyObject(3)) -> <functools.K object at 0x100da2ad0> 
key_wrapper(MyObject(2)) -> <functools.K object at 0x100da2b10> 
key_wrapper(MyObject(1)) -> <functools.K object at 0x100da2b50> 
comparing MyObject(4) and MyObject(5) 
comparing MyObject(3) and MyObject(4) 
comparing MyObject(2) and MyObject(3) 
comparing MyObject(1) and MyObject(2) 
MyObject(1) 
MyObject(2) 
MyObject(3) 
MyObject(4) 
MyObject(5) 

See Also:

functools (https://docs.python.org/library/functools.html) The standard library documentation for this module.

Rich comparison methods (https://docs.python.org/reference/datamodel.html# object.__lt__) Description of the rich comparison methods from the Python Reference Guide.

inspect (page 1200) Introspection API for live objects.

© Copyright Pearson Education. All rights reserved.

	Excerpt from Python Standard Library by Example, The.
By Doug Hellmann Published by Addison-Wesley Professional ISBN-10: 0-321-76734-9 ISBN-13: 978-0-321-76734-9

This excerpt is from the book, ‘The Python Standard Library by Example’ by Doug Hellmann, published by Pearson/Addison-Wesley Professional, June 2011, ISBN 0321767349, Copyright 2011 Pearson Education, Inc. For more info please visit www.informit.com/title/0321767349