Intro python-object-protocol
0 likes804 views
This document provides an overview of objects and protocols in CPython. Some key points: - Everything in Python is an object, with a common header including a reference count and pointer to its type. Types are defined by TypeObjects which store metadata like function pointers. - Protocols like Number, Sequence, Mapping are defined by slots on TypeObjects like tp_as_number and tp_as_sequence. These allow objects to support common operations even if they are different types, through duck typing. - Magic methods fill in protocol slots, so e.g. a class with a __len__ method will support the len() operation by filling tp_as_sequence->sq_length. This provides a
![CPython Overview
‣ First implemented in Dec.1989 by GvR, the BDFL
‣ Serving as the reference implementation.
‣ IronPython (clr)
‣ Jython (jvm)
‣ Brython (v8, spider) [no kidding]
‣ Written in ANSI C.
‣ flexible language binding
‣ embedding (libpython), e.g., openwrt, etc.
4](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-4-320.jpg)
![CPython Overview
‣ execution lifetime
7
PY
Parser
PY[CO]
VM
LOAD 1
LOAD 2
ADD
LOAD X
STORE X
x = 1 + 2
1 1
2
3
3
x
STACK
takeaway: py/pyc/c inter-relatoin](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-7-320.jpg)
![Object Structure
Will PyLongObject overflow?
10
The answer: chunk-chunk
digit[n]
…
digit[3]
digit[2]
ob_type
ob_size
digit ob_digit[1]
ob_refcnt
PyLongObject
typedef PY_UINT32_T digit;
result = PyObject_MALLOC(offsetof(PyLongObject, ob_digit) +
size*sizeof(digit));
n = 2 ** 64 # more bits than a word
assert type(n) is int and n > 0](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-10-320.jpg)
![Object Structure
Why my multi-dimensional array won’t work?
11
The answer: indirection, mutability
allocated
ob_type
ob_item
ob_refcnt
ob_size
PyListObject
PyObject*
PyObject*
…
PyObject*
PyObject*
allocated
ob_type
ob_item
ob_refcnt
ob_size
PyObject*
PyObject*
… 42
None
m, n = 4, 2
arr = [ [ None ] * n ] * m
arr[1][1] = 42
# [[None, 42], [None, 42], [None, 42], [None, 42]]
PyList_SetItem](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-11-320.jpg)
![More Example
Why can we multiply a list? Is it slow?
19
arr = [None] * 3
# [None, None, None]
Exercise:
arr = [None] + [None]
# [None, None]](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-19-320.jpg)
![Next: Heterogeneous
Have you ever felt insecure towards negative indexing of
PyListObject?
22
The answer: RTFSC
words = "the quick brown fox jumps over the old lazy dog".split()
assert words[-1] == "dog"
words.insert(-100, "hayabusa")
assert words[-100] == ??](https://image.slidesharecdn.com/intro-python-object-protocol-150504154824-conversion-gate02/85/Intro-python-object-protocol-22-320.jpg)
More Related Content
What's hot (20)
Viewers also liked (20)
Similar to Intro python-object-protocol (20)
Recently uploaded (20)
Intro python-object-protocol
- 1. a smalltalk on Object and Protocol in CPython shiyao.ma <[email protected]> May. 4th
- 2. Why this ‣ Python is great for carrying out research experiment. this should lay the foundation why I discuss Python. ‣ Life is short. You need Python. this should lay the foundation why people like Python. 2 life is neither a short nor a long, just a (signed) int, 31bits at most, I say.
- 3. Takeaway ‣ Understand the inter-relation among {.py, .pyc .c} file. ‣ Understand that everything in Python is an object. ‣ Understand how functions on TypeObject affect InstanceObject. 3
- 4. CPython Overview ‣ First implemented in Dec.1989 by GvR, the BDFL ‣ Serving as the reference implementation. ‣ IronPython (clr) ‣ Jython (jvm) ‣ Brython (v8, spider) [no kidding] ‣ Written in ANSI C. ‣ flexible language binding ‣ embedding (libpython), e.g., openwrt, etc. 4
- 5. CPython Overview ‣ Code maintained by Mercurial. ‣ source: https://hg.python.org/cpython/ ‣ Build toolchain is autoconf (on *nix) ./configure --with-pydebug && make -j2 5
- 7. CPython Overview ‣ execution lifetime 7 PY Parser PY[CO] VM LOAD 1 LOAD 2 ADD LOAD X STORE X x = 1 + 2 1 1 2 3 3 x STACK takeaway: py/pyc/c inter-relatoin
- 8. 8 object and protocol: the objects
- 9. Object object: memory of C structure with common header 9 PyListObject PyDictObject PyTupleObject PySyntaxErrorObject PyImportErrorObject … takeaway: everything is object ob_type ob_refcnt PyObject ob_type ob_size ob_refcnt PyVarObject
- 10. Object Structure Will PyLongObject overflow? 10 The answer: chunk-chunk digit[n] … digit[3] digit[2] ob_type ob_size digit ob_digit[1] ob_refcnt PyLongObject typedef PY_UINT32_T digit; result = PyObject_MALLOC(offsetof(PyLongObject, ob_digit) + size*sizeof(digit)); n = 2 ** 64 # more bits than a word assert type(n) is int and n > 0
- 11. Object Structure Why my multi-dimensional array won’t work? 11 The answer: indirection, mutability allocated ob_type ob_item ob_refcnt ob_size PyListObject PyObject* PyObject* … PyObject* PyObject* allocated ob_type ob_item ob_refcnt ob_size PyObject* PyObject* … 42 None m, n = 4, 2 arr = [ [ None ] * n ] * m arr[1][1] = 42 # [[None, 42], [None, 42], [None, 42], [None, 42]] PyList_SetItem
- 12. Object Structure what is the ob_type? 12 The answer: flexible type system class Machine(type): pass # Toy = Machine(foo, bar, hoge) class Toy(metaclass=Machine): pass toy = Toy() # Toy, Machine, type, type print(type(toy), type(Toy), type(Machine), type(type)) ob_type ob_refcnt … toy ob_type ob_refcnt … ob_type ob_refcnt … Toy Machine ob_type ob_refcnt … Type
- 13. Object Structure what is the ob_type? 13 # ob_type2 # 10fd69490 - 10fd69490 - 10fd69490 print("%x - %x - %x" % (id(42 .__class__), id(233 .__class__), id(int))) assert dict().__class__ is dict # dynamically create a class named "MagicKlass" klass=“MagicKlass" klass=type(klass, (object,), {"quack": lambda _: print("quack")}); duck = klass() # quack duck.quack() assert duck.__class__ is klass
- 14. Object Structure what is the ob_type? 14 ob_type … … … ob_refcnt PyObject … *tp_as_mapping *tp_as_sequence *tp_as_number … ob_type tp_getattr … tp_print ob_refcnt PyTypeObject … nb_subtract … nb_add
- 15. 15 object and protocol: the protocol
- 17. AOL ‣ Abstract Object Layer 17 … *tp_as_mapping *tp_as_sequence *tp_as_number … ob_type tp_getattr … tp_print ob_refcnt PyTypeObject … nb_subtract … nb_add When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck. Object Protocol Number Protocol Sequence Protocol Iterator Protocol Buffer Protocol int PyObject_Print(PyObject *o, FILE *fp, int flags) int PyObject_HasAttr(PyObject *o, PyObject *attr_name) int PyObject_DelAttr(PyObject *o, PyObject *attr_name) … PyObject* PyNumber_Add(PyObject *o1, PyObject *o2) PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2) PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) … PyObject* PySequence_Concat(PyObject *o1, PyObject *o2) PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count) PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i) … int PyIter_Check(PyObject *o) PyObject* PyIter_Next(PyObject *o) int PyObject_GetBuffer(PyObject *exporter, Py_buffer *view, int flags) void PyBuffer_Release(Py_buffer *view) int PyBuffer_IsContiguous(Py_buffer *view, char order) … Mapping Protocol int PyMapping_HasKey(PyObject *o, PyObject *key) PyObject* PyMapping_GetItemString(PyObject *o, const char *key) int PyMapping_SetItemString(PyObject *o, const char *key, PyObject *v) …
- 18. Example ‣ Number Protocol (PyNumber_Add) 18 // v + w? PyObject * PyNumber_Add(PyObject *v, PyObject *w) { // this just an example! // try on v result = v->ob_type->tp_as_number.nb_add(v, w) // if fail or if w->ob_type is a subclass of v->ob_type result = w->ob_type->tp_as_number.nb_add(w, v) // return result } … *tp_as_mapping *tp_as_sequence *tp_as_number … ob_type tp_getattr … tp_print ob_refcnt PyTypeObject … nb_subtract … nb_add takeaway: typeobject stores meta information
- 19. More Example Why can we multiply a list? Is it slow? 19 arr = [None] * 3 # [None, None, None] Exercise: arr = [None] + [None] # [None, None]
- 20. Magic Methods access slots of tp_as_number, and its friends 20 Note tp_as_mapping->mp_length and tp_as_sequence->sq_length map to the same slot __len__ If your C based MyType implements both, what’s MyType.__len__ and len(MyType()) ? # access magic method of dict and list dict.__getitem__ # tp_as_mapping->mp_subscript dict.__len__ # tp_as_mapping->mp_length list.__getitem__ # tp_as_sequence->sq_item list.__len__ # tp_as_sequence->sq_length
- 21. Magic Methods backfill as_number and its friends 21 class A(): def __len__(self): return 42 class B(): pass # 42 print(len(A())) # TypeError: object of type 'B' has no len() print(len(B())) Py_ssize_t PyObject_Size(PyObject *o) { PySequenceMethods *m; if (o == NULL) { null_error(); return -1; } m = o->ob_type->tp_as_sequence; if (m && m->sq_length) return m->sq_length(o); return PyMapping_Size(o); }Which field does A.__len__ fill?
- 22. Next: Heterogeneous Have you ever felt insecure towards negative indexing of PyListObject? 22 The answer: RTFSC words = "the quick brown fox jumps over the old lazy dog".split() assert words[-1] == "dog" words.insert(-100, "hayabusa") assert words[-100] == ??
- 23. Thanks 23