CLN: make _Indexer.obj a weak ref

[jreback]

This will allow pandas objects to be collected on the first generation of the gc rather than wait for it to break cycles. Practically I am not sure this will have much of a user change.

dask/distributed#956 (comment)

The idea is to change this code here

from

class _NDFrameIndexer(object):
    _valid_types = None
    _exception = KeyError
    axis = None

    def __init__(self, obj, name):
        self.obj = obj
        self.ndim = obj.ndim
        self.name = name

to

class _NDFrameIndexer(object):
    _valid_types = None
    _exception = KeyError
    axis = None

    def __init__(self, obj, name):
        self.obj = weakref.ref(obj)
        self.ndim = obj.ndim
        self.name = name

and corresponding self.obj to self.obj()

it 'works' in that gc collection happens immedately upon object deletion (IOW del df). but a few fails on caching / chaining. In particular tests like: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/indexing/test_chaining_and_caching.py#L31 I think were relying upon the reference NOT being collected (so that they can check it).

So this would require some internal reworking to remove / fix this. I suspect we will still achieve the same user effects (meaning of detection of chaining etc).

[jreback]

cc @pitrou
cc @mrocklin

[pitrou]

I don't really understand the Pandas internals so I'd be hard-pressed to submit a solution here. Note you can use weakref.proxy instead of weakref.ref if you want to avoid writing self.obj() everywhere, though.

[jreback]

xref #10221

[pitrou]

It seems it will be difficult to solve this while keeping both the caching and the requirement that an indexer still works after the user loses visible references to the dataframe.

One idea is to remove the caching and avoid a large performance loss by using a C (actually Cython, probably) constructor for _NDFrameIndexer. What do you think, @jreback ?

[pitrou]

Note about current performance:

>>> df = pd.DataFrame(data={'a': [1,2], 'b': [3,4]})
>>> %timeit df.iloc
323 ns ± 0.949 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
>>> %timeit _iLocIndexer(df, 'iloc')
1.07 µs ± 2.69 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

[jreback]

you could move this to cython, though I think you would have to move the entire hierarchy (e.g. most of core/indexing.py), so not sure this would be easy.

[pitrou]

Actually, it seems only minimal changes are required:
https://gist.github.com/pitrou/1556024bef72ac26c7848f6f1d7c3fa3

Performance with this patch:

>>> df = pd.DataFrame(data={'a': [1,2], 'b': [3,4]})
>>> %timeit df.iloc
204 ns ± 1.37 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)