PERF: generators in dataframe.from_records

[westurner]

itertools.islice would be faster than n*comparisons here:

https://github.com/pydata/pandas/blob/88c4c5526b38194f00a00c207ee5388dfd508092/pandas/core/frame.py#L759

Not sure whether to amortize an import itertoosl or not.

[westurner]

See: #4911 #4910

@jreback

Not sure whether to amortize an import itertoosl or not.

[jreback]

you can just import at the top of the file

[jreback]

can u show a perf comparison before and with the change?

[westurner]

I ran a few tests w/ %timeit. Like as a slow test or w/ vbench? [1]

[1] https://github.com/wesm/pandas/blob/master/vb_suite/frame_methods.py

[westurner]

Enumerate [1] is faster than incrementing within a for loop (I don't know why); and then itertools.islice made more sense because it accepts None since 2.5 [2][3]

[1] http://docs.python.org/2/library/functions.html#enumerate
[2] http://docs.python.org/2/library/itertools.html#itertools.islice
[3] http://hg.python.org/cpython/file/v2.7.6/Modules/itertoolsmodule.c#l1126

[jreback]

maybe can do a vbench
but just looking for an example of the timings and how much this matters

make a resonance sized rec array and convert before the change and then with it

show a sample session

[westurner]

Here's this: https://gist.github.com/westurner/9754768 for running timeit.timeit with (what I believe are) the relevant portions of code affected by this patch. (Note that com.is_iterator(data) is not True with a numpy.ndarray)

Summary:

• f0 -- Pandas 0.13.1 status quo
• f1 -- what's currently in the patch
• r2 -- f2 would ameliorate the 0.5% performance regression for reading all rows introduced by the proposed f1
• f1 and f2 are 42% faster when specifying nrows

a vbench in vb_suite/frame_methods.py could look something like:

#-----------------------------------------------------------------------------
# from_records issue-6700

setup = common_setup + """
data = ((n, n*20, n*100) for n in xrange(100000))
"""
frame_from_records = Benchmark('df = DataFrame.from_records(data)', setup,
                    name = 'frame_from_records_generator',
                    start_date=datetime(2013,10,04))  # issue-4911

[jreback]

this looks fine,

• add the vbench you suggested (and post results).
• add a release note in enhancements, referencing this PR number.