Aggregator for questions regarding _libs

[jbrockmendel]

I don't see any entry for the cython code in _libs over on the codecov page. Is that measured somewhere else?

[jreback]

I don't think coverage is setup up to mark this. I suppose it could be added, though generally codecov is pretty poor.

[jbrockmendel]

I'm vaguely aware of long-term plans to overhaul things in pandas2. Are there medium-term plans for _libs? In particular, I'd be interested in:

• Implementing test coverage
• Identifying and removing unused code (e.g. I think offsets.pyx)
• Making it more accessible to those of us are only comfortable with python (more import, less include and extern)
• Avoiding from xyz import *
• Splitting up 5k line modules

[jreback]

@jbrockmendel there is an issue for removing offsets.pyx already
if you want to propose a change pls do so before actually implementing it.

• Splitting up 5k line modules, not sure we have any of those (the ones that are are all templated code and auto-generated).
• from xyz import * is pretty standard in cython, but sure
• Making it more accessible to those of us are only comfortable with python (more import, less include and extern), well this is cython, so this is not worth doing at all.

[jbrockmendel]

Splitting up 5k line modules, not sure we have any of those

You're right, looks like tslib is the only one. It clocks in at 5740. lib gets close if you count the included files.

from xyz import * is pretty standard in cython, but sure

This combined with include makes it really tough to figure out where names come from (especially when * is imported from more than one place). A little bit of verbosity in these imports will make this a lot more accessible to newcomers (assuming that's desired).

Making it more accessible to those of us are only comfortable with python (more import, less include and extern), well this is cython, so this is not worth doing at all.

To be clear, I'm not suggesting de-cythoning anything. Just refactoring enough that modules can be imported independently (a comment in _libs/__init__.py suggests this desirable at some point). At is it, a bunch of these pyx files import tslib in many cases just to get at NaT.

[jbrockmendel]

Ref: #11214

[jbrockmendel]

For kicks I tried implementing datetime_helper.total_seconds in native cython and was surprised that it was appreciably faster than the existing version. Are these measurements misleading somehow? Is the C version preferred for some other reason?

Update: a comment in the C version says // Python 2.6 compat. Is that relevant?

cdef extern from "datetime_helper.h":
    double total_seconds(object)

from numpy cimport float64_t
cpdef float64_t ptotal_seconds(object obj):
    return total_seconds(obj)

cpdef float64_t total_seconds2(object td):
    cdef int microseconds, seconds, days, days_in_seconds

    microseconds = getattr(td, "microseconds", 0)
    seconds = getattr(td, "seconds", 0)
    days = getattr(td, "days", 0)
    days_in_seconds = days * 24 * 3600;
    return (microseconds + (seconds + days_in_seconds) * 1000000.0) / 1000000.0

import timeit
from datetime import timedelta
dt = timedelta(4)

def func0():
    return total_seconds(dt)

def func1():
    return ptotal_seconds(dt)

def func2():
    return total_seconds2(dt)

times = {}
times['total_seconds'] = timeit.timeit(func0)
times['ptotal_seconds'] = timeit.timeit(func1)
times['total_seconds2'] = timeit.timeit(func2)

Results:

{'ptotal_seconds': 0.21397995948791504,
 'total_seconds': 0.21534299850463867,
 'total_seconds2': 0.07924604415893555}

I get roughly the same results if I run a loop inside the file instead of calling timeit.timeit. The fact that the wrapped ptotal_seconds is slightly faster than the not-wrapped total_seconds seems fishy.

[jbrockmendel]

It looks like pandas_datetime_struct is identical to npy_datetimestruct. Is there a compelling reason to have a separate implementation?

It also looks like every single use of pandas_datetimestruct_to_datetime uses PANDAS_FR_ns. For pandas_datetime_to_datetimestruct there is one use in _get_datetime64_nanos that does not use PANDAS_FR_ns. Are the general cases used somewhere else I'm not seeing?

[jreback]

pandas was originally written before cython was really that compat so a fair amount of c was written. not sue why the perf differences are there, but usually things like this come down to certain ways given type conversions and some not. find a real world benchmark where this actually matters.

In [2]: import datetime

In [3]: dt = datetime.timedelta(4)

In [4]: dt.total_seconds()
Out[4]: 345600.0

In [5]: %timeit dt.total_seconds()
182 ns +- 2.61 ns per loop (mean +- std. dev. of 7 runs, 10000000 loops each)

In [6]: from pandas._libs.tslib import tot_seconds

In [7]: %timeit tot_seconds(dt)
381 ns +- 3.46 ns per loop (mean +- std. dev. of 7 runs, 1000000 loops each)

this makes almost no difference.

[jreback]

we don't really need an explicit issue for this.