Python speleology

Python Speleology
Getting deep in python features

The Zen of Python

○ Beautiful is better than ugly.
○ Explicit is better than implicit.
○ Simple is better than complex.
○ Complex is better than complicated.
○ Flat is better than nested.
○ Sparse is better than dense.
○ Readability counts.
○ Special cases aren't special enough to break the rules.
○ Although practicality beats purity.
○ Errors should never pass silently.
○ Unless explicitly silenced.
○ In the face of ambiguity, refuse the temptation to guess.
○ There should be one -and preferably only one- obvious way to do it.
○ Although that way may not be obvious at first unless you're Dutch.
○ Now is better than never.
○ Although never is often better than right now.
○ If the implementation is hard to explain, it's a bad idea.
○ If the implementation is easy to explain, it may be a good idea.
○ Namespaces are one honking great idea -let's do more of those!

Types and objects: To be or not to be

a = 256
b = 256
print (a is b)

a = 257
b = 257

print (a is b)


a = 256
b = 256
print (a is b)

a = 257
b = 257

print (a is b)

True
False


a = 256 a = 256
b = 256 b = 256

print (a is b) print id(a), id(b)

a = 257 a = 257
b = 257 b = 257


True 22036112 22036112
False 22363568 22363640


a = 256 a = 256
b = 256 b = 256


a = 257 a = 257
b = 257 b = 257


True
False

Types and objects: functions are objects

def x(f):
return f

def y(f):
return x

print x(y)(8)(0)


def x(f):
return f

def y(f):
return x

print x(y)(8)(0)

0


def x(i):
if x.enabled:
return i
else:
return "disabled"

x.enabled = True
print x(8)


def x(i):
if x.enabled:
8
return i
else:
return "disabled"

x.enabled = True
print x(8)

What happened if not set x.enabled?

Types and objects: star operator

args = (3 , 6)

print range(*args)

args = { "name": "example" }

def f(name):
print name

f(**args)

def f(*args, **kwargs):
print args
print kwargs

f(1,2,3, name="example")

Types and objects: star operator

args = (3 , 6)
[3, 4, 5]
example
print range(*args) (1, 2, 3)
{'name': 'example'}
args = { "name": "example" }

def f(name):
print name

f(**args)

def f(*args, **kwargs):
print args
print kwargs

f(1,2,3, name="example")

Reflection: get & set

class Example(object):
num = 10

x = Example

dir(x)
hasattr(x, "num") == True
getattr(x, "num", 0) == 10
setattr(x, "num", 20)

Reflection: local & global

globals()
Return a dictionary representing the current global symbol table. This is always the dictionary of the
current module (inside a function or method, this is the module where it is defined, not the module from
which it is called).

locals()
Update and return a dictionary representing the current local symbol table. Free
variables are returned by locals() when it is called in function blocks, but not in
class blocks.

Reflection: __magic__

__name__
This is the name of the function. This only have a meaningful value is the function is defined with “def”.

__class__
This is a reference to the class a method belongs to.

__code__
This is a reference to the code object used in the implementation of python.

Reflection: inspector

from inspect import getcomments

# This is a comment
def f(x):
print x

print getcomments(f)


from inspect import getcomments

# This is a comment
def f(x):
print x

print getcomments(f)

# This is a comment


from inspect import getsource

# This is a comment
def f(x):
print x

print getsource(f)


from inspect import getsource

# This is a comment
def f(x):
print x

print getsource(f)

def f(x):
print x

Reflection: let's more tricky

def f(x):
print x

print f.__code__.co_code


def f(x):
print x


'dx01x00GHdx00x00S'


def f(x):
print x


'dx01x00GHdx00x00S'

YEEES!!! THE BYTECODE!!!

Context manager: in the beginning...

item = Item()

try:
item.open()
item.do()
finally:
item.close()

Context manager: nowadays...

with Item() as item:
item.do

class Item(object):
def __enter__(self):
self.open()
return self

def __exit__(self,exc_type,exc_value,exc_t):
self.close()

...

Context manager: the real world

with file("/tmp/test", "w") as f:
f.write("hello world")

with lock():
# do some concurrent

with sudo("root"):
# do some as root

Decorations: bold & italic

def makebold(fn):
def wrapped():
return "" + fn() + ""
return wrapped

def makeitalic(fn):
def wrapped():
return "" + fn() + ""
return wrapped

@makebold
@makeitalic
def hello():
return "hello world"

print hello()

Decorations: bold & italic

def makebold(fn):
def wrapped(): hello world
return "" + fn() + ""
return wrapped

def makeitalic(fn):
def wrapped():
return "" + fn() + ""
return wrapped

@makebold
@makeitalic
def hello():

print hello()

Decorations: complex decor

def makestyle(arg):
def decorator(f):
def wrapper(*args, **kw):
return "<" + arg + ">" + f() + "</" + arg + ">"
return wrapper
return decorator

@makestyle("b")
def hello():

print hello()

Decorations: syntax sugar

def makebold(fn):
def wrapped():
return "" + fn() + ""
return wrapped

def makestyle(arg):
def decorator(f):
def wrapper(*args, **kw):
return "<" + arg + ">" + f() + "</" + arg + ">"
return wrapper
return decorator

makebold(hello)

makestyle("b")(hello)

Iterations: comprehesions

squares = []
for x in range(10):
squares.append(x**2)

squares = [x**2 for x in range(10)]

[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]

{ (k,v) for k,v in [(1,2)] }


squares = []
for x in range(10):



{ (k,v) for k,v in [(1,2)] }

SET NOT DICT!


squares = []
for x in range(10):



{ (k,v) for k,v in [(1,2)] }

SET NOT DICT!

{ k:v for k,v in [(1,2)] }

Iterations: co-routine

def countdown(n):
print "Counting down from", n
while n > 0:
yield n
n -= 1
print "Done counting down"

for i in countdown(10):
print i


def countdown(n):
while n > 0:
yield n
n -= 1

for i in countdown(10):
print i

Counting down from 10
10
9
8
7
6
5
4
3
2
1
Done counting down

def countdown(n):
while n > 0:
yield n
n -= 1

print countdown(10)

<generator object at 0x4035874c>

Overloading: binary operations

__add__(self, other) x+y
__sub__(self, other) x-y
__mul__(self, other) x*y
__div__(self, other) x/y
__pow__(self, other) x ** y


__radd__(self, other) y+x
__rsub__(self, other) y-x
__rmul__(self, other) y*x
__rdiv__(self, other) y/x
__rpow__(self, other) y ** x


__radd__(self, other) 1+x
__rsub__(self, other) 1-x
__rmul__(self, other) 1*x
__rdiv__(self, other) 1/x
__rpow__(self, other) 1 ** x


__iadd__(self, other) x+=y
__isub__(self, other) x-=y
__imul__(self, other) x*=y
__idiv__(self, other) x/=y
__ipow__(self, other) x**=y

Overloading: unary operations

__neg__(self) -x
__pos__(self) +x
__abs__(self) abs(x)
__invert__(self) ~x

Overloading: conversion operations

__int__(self) int(x)
__float__(self) float(x)
__complex__(self) complex(x)
__str__(self) str(x)
__nonzero__(self) bool(x)
__unicode__(self) unicode(x)

Overloading: comparison operations

__eq__(self, other) x == y
__lt__(self, other) x<y
__le__(self, other) x <= y
__gt__(self, other) x>y
__ge__(self, other) x >= y
__ne__(self, other) x != y

Overloading: containers

__contains__(self, other) y in x
__getitem__(self, other) x[y]
__setitem__(self, other,value) x[y] = z
__delitem__(self, other) del x[y]
__len__(self) len(x)
__reversed__(self) reversed(x)
__iter__(self) iter(x)

The Class Factory: class & objects

attribute = "this is a class attribute"

def __init__(self):
self.attribute = "this is an obj attr override class one"
self.another = "this is another obj attr, no class"

print Example.attribute
print Example().attribute
print Example().another
print Example.another

The Class Factory: class & objects

attribute = "this is a class attribute"

def __init__(self):
self.attribute = "this is an obj attr override class one"
self.another = "this is another obj attr, no class"

print Example.attribute
print Example().attribute
this is a class attribute
print Example().another this is an object attribute and override class one
this is another object attribute, no class
Traceback (most recent call last):
Line 11, in <module>
AttributeError: type object 'Example' has no attribute 'another'

The Class Factory: set & get

def __init__(self):
self._name = x

@property
def name(self):
return self._name

@name.setter
def name(self, value):
self._name = value

The Class Factory: @property abuse

def __init__(self, host, port):
self.host = host
self.port = port

@property
def connect(self):
lib.connect(self.host, self.port)

The Class Factory: @property abuse

def __init__(self, host, port):
self.host = host
self.port = port

@property
def connect(self):
lib.connect(self.host, self.port)

NEVER TYPE METHODS AS
PROPERTIES

The Class Factory: methods and more methods

@staticmethod

Nothing more than a function defined inside a class.
It is callable without instantiating the class first. It’s
definition is immutable via inheritance.

@classmethod

Also callable without instantiating the class, but its
definition follows Sub class, not Parent class, via
inheritance. That’s because the first argument for
@classmethod function must always be cls (class).


@staticmethod

Nothing more than a function defined inside a class.
It is callable without instantiating the class first. It’s
definition is immutable via inheritance.

@staticmethod
def static_method():
print "I do not receive nothing :("

@classmethod

Also callable without instantiating the class, but its
definition follows Sub class, not Parent class, via
inheritance. That’s because the first argument for
@classmethod function must always be cls (class).

@classmethod
def class_method(cls):
print "I'm a class %s" % str(cls)


def __init__(self, name):
self.name = name

@classmethod
def class_method(cls, name):
return cls(name)

x = Example.class_method("example")

print x

self.name = name

@classmethod
def class_method(cls, name):
return cls(name)

x = Example.class_method("example")

print x

<__main__.Example object at 0x40358b2c>

The Class Factory: children and parents

self.name = name

def do_something(self):
raise NotImplementedError()

class ChildExample(Example):
print self.name

The Class Factory: children and parents

class ExampleA(object):
self.name = name


class ExampleB(object):
def do_otherthing(self):

class ChildExample(ExampleB, ExampleA):
print self.name

def do_otherthing(self):
print self.name

The Class Factory: τὰ μετὰ τὰ κλάση

pass
x = Example()
y = Example

print x
print y

pass
x = Example()
y = Example

print x
print y

<__main__.Example object at 0x4035894c>
<class '__main__.Example'>


def example():
pass
return Example

x = example()
y = x()

print x
print y

def example():
pass
return Example

x = example()
y = x()

print x
print y

<__main__.Example object at 0x403589ec>


pass
print type(Example)
print type(Example())

pass
print type(Example)
print type(Example())

<type 'type'>


pass

x = Example
y = type('Example', (object,), {})

print x
print y
print (x == y)


pass

x = Example
y = type('Example', (object,), {})

print x
print y
print (x == y)

False

The Class Factory: __magic__

__new__(cls, *args, **kwargs)
Is the first method to get called in an object's instantiation, is a
@classmethod, and must return a new instance of type cls.

__init__(self, *args, **kwargs)
Is the initializer for the instance. It gets passed whatever the
primary constructor was called with.

__del__(self)
Is the destructor of the instance, will be invoked before clean
the reference to the instance.

The Class Factory: __metaclass__

def upper_attr(f_class_name, f_class_parents, f_class_attr):
attrs = ((name, value)
for name, value in f_class_attr.items()
if not name.startswith('__'))
uppercase_attr = dict((name.upper(), value)
for name, value in attrs)
return type(f_class_name, f_class_parents, uppercase_attr)

class Foo(object):
__metaclass__ = upper_attr
bar = 'bip'

print hasattr(Foo, 'bar')
print hasattr(Foo, 'BAR')

The Class Factory: __metaclass__

def upper_attr(f_class_name, f_class_parents, f_class_attr):
attrs = ((name, value)
for name, value in f_class_attr.items()
if not name.startswith('__'))
uppercase_attr = dict((name.upper(), value)
for name, value in attrs)
return type(f_class_name, f_class_parents, uppercase_attr)

class Foo(object):
__metaclass__ = upper_attr
bar = 'bip'

print hasattr(Foo, 'bar') False
print hasattr(Foo, 'BAR') True

Monkey patching: first try

class Person(object):
def speak(self):
print "hello"

def monkey(foo):
print "uh uh uh"

Person.speak = monkey

x = Person()
x.speak()

Monkey patching: be evil }:-)

import os

def monkey(*args, **kwargs):
print "no no no"

os.system = monkey

os.system("ls /tmp")

Monkey patching: be evil }:-)

import os

def monkey(*args, **kwargs):
print "no no no"

os.system = monkey

os.system("ls /tmp")

no no no

Profiling: hand made

import time

class Timer(object):
def __init__(self, verbose=False):
self.verbose = verbose

def __enter__(self):
self.start = time.time()
return self

def __exit__(self, *args):
self.end = time.time()
self.secs = self.end - self.start
self.msecs = self.secs * 1000 # millisecs
if self.verbose:
print 'elapsed time: %f ms' % self.msecs

Profiling: hand made

from redis import Redis
import time rdb = Redis()
with Timer() as t:
class Timer(object): rdb.lpush("foo", "bar")
def __init__(self, verbose=False): print "=> elasped lpush: %s s" % t.
secs
self.verbose = verbose
with Timer as t:
rdb.lpop("foo")
def __enter__(self): print "=> elasped lpop: %s s" % t.secs
self.start = time.time()
return self

def __exit__(self, *args):
self.end = time.time()
self.secs = self.end - self.start
self.msecs = self.secs * 1000 # millisecs
if self.verbose:
print 'elapsed time: %f ms' % self.msecs

Profiling: profile & cProfile

try:
import cProfile as profile
except ImportError:
import profile

def fib(n):
if n == 0:
return 0
elif n == 1:
return 1
else:
return fib(n-1) + fib(n-2)

def fib_seq(n):
seq = [ ]
if n > 0:
seq.extend(fib_seq(n-1))
seq.append(fib(n))
return seq

profile.run('print fib_seq(6); print')

Profiling: profile & cProfile

try:
import cProfile as profile
except ImportError:
import profile
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765]

57356 function calls (66 primitive calls) in 0.746 CPU seconds
def fib(n):
if n == 0: standard name
Ordered by:
return 0
ncalls tottime percall cumtime percall filename:lineno(function)
21 0.000 0.000 0.000 0.000 :0(append)
elif n == 1:
20 0.000 0.000 0.000 0.000 :0(extend)
return 1
1 0.001 0.001 0.001 0.001 :0(setprofile)
else: 0.000 0.000 0.744 0.744 <string>:1(<module>)
1
1 0.000 0.000 0.746 0.746 profile:0(print fib_seq(20); print)
return fib(n-1) + 0.000
0 0.000 fib(n-2) profile:0(profiler)
57291/21 0.743 0.000 0.743 0.035 profile_fibonacci_raw.py:13(fib)
21/1 0.001 0.000 0.744 0.744 profile_fibonacci_raw.py:22(fib_seq)
def fib_seq(n):
seq = [ ]
if n > 0:
seq.extend(fib_seq(n-1))
seq.append(fib(n))
return seq

profile.run('print fib_seq(6); print')

Documentation: Zen

Don't create documentation for your code.
Code your documentation.

def elements(n):
"""Return a list of n numbers from 0 to n-
1.
"""
return range(0,n)

Documentation: everything is an object

def elements(n):
1.
"""
return range(0,n)

print elements.__doc__

Documentation: everything is an object

def elements(n):
1.
"""
return range(0,n)

print elements.__doc__

Return a list of n numbers from 0 to n-1

Documentation: style is important

def elements(n):
"""Return a list of n numbers from 0 to n-1.
:type n: int
:param n: the limit upper for the elements to be created (not
included).

:return: a class:`list` with the items.
"""
return range(0,n)

"""This is the documentation of the class.
Usually you do not need it :)
"""

def __init__(self, param):
"""This is the documentation of the instance type.
"""

Documentation: do the work

$ sphinx-quickstart

Documentation: Bonus... abuse the doc

class Command(object):
"""Undocumented command
"""

class ListCommand(Command):
"""List command.
"""

def show_help(command):
cmd = getattr(globals()[__name__], "%sCommand" % command,
None)
if cmd is None:
return "Invalid command"
else:
return cmd.__doc__

print show_help("List")

Future: pypy

Just in time compiler FAST!

With sandboxing

Best concurrency support

Future: python3

def hello(name: str, age: int) -> str:
return name

print hello.__anotations__

Future: python3

def hello(name: str, age: int) -> str:
return name

print hello.__anotations__

{'return':<class'int'>,'name':<class'str'>,'age':<class'int'>}

Future: python3

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
if n < 2:
return n
return fib(n-1)+fib(n-2)

Applauses & questions
Not necessarily in that order.

Python speleology

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Python speleology (20)

Recently uploaded (20)

Python speleology