Scrape HTML tables into Dataframes

from ML: https://groups.google.com/forum/?fromgroups=#!topic/pydata/q7VVD8YeSLk

User provides HTML string for whatever source he likes, or url.
optionally specify table id, or regex to match against contained cell
content to quickly single out .+ tables, when multiple exist on the page.

Pseudo:

DataFrame.from_html('http://foo.com/tickers?sym=GOOG',match="high")

Aside: Perhaps not widely known, but excel and co can import tables directly
from online webpages, a cheap "no code" way to get the data into a form
directly readable by pandas.

[cpcloud]

Is the goal to have a parser written just for pandas so that are no dependencies? Beautiful Soup + lxml does this quite effectively with a little coaxing. This would of course add dependencies which I'm guessing is frowned upon to avoid feature bloat.

yep on all three. except the first.

[jreback]

why is it a problem to add an optional dependency for bs or lxml for this parsing? we do this for excel

if he user wants it they can install it

I think so too. lxml I think, it's faster and recent versions have as good
support for css selectors as bs4.

[cpcloud]

Bs4 can use lxml under the hood and is much easier to use than lxml.

I've used both, and as I said recent lxml is very usable. But it really doesn't matter what you use.
If you'd like to claim this, would happily move it to 0.12..

[cpcloud]

Although I guess that doesn't really matter if you're just exposing this API and want to minimize deps.

[changhiskhan]

Also on SO: http://stackoverflow.com/questions/16009778/how-to-convert-a-html-table-into-pandas-dataframe

[jreback]

just do something like: from_html(data, method='lxml', ......) and support whatever methods you want, raise if you can't deal with it, or maybe flavor is better

lxml is compiled and has some lib/.so deps, bs4's ability to abstract the underlying
parser (including using a pure python solution) would make things smoother for windows
users. I think that clinches it.

[cpcloud]

Ok, so bs4 it is.

[cpcloud]

Are there any type inference utility functions hidden in the guts of pandas?

[jreback]

df.convert_objects(convert_numeric=True,convert_dates='coerce') will convert anything datelike and anything number like into the correct dtypes (and set the rest to NaN/NaT. This might be too aggressive, so you could start with convert_dates=True (I think you want convert_numeric=True always) for this type of thing

[cpcloud]

Hm, df.convert_objects doesn't seem to work on the most human readable of dates such as "March 17, 2005". E.g.,

from pandas import DataFrame
df = DataFrame(['March 25, 2005', 'March 27, 2001'])
df.convert_objects(convert_dates=True) # still strings

This could be hacked around by using dateutil.parser.parse but that's slow for a frame with many columns.

[jreback]

that's a 'soft' conversion (to force it use, 'force') (equiv to to_pydatetime on the column)

In [3]: df.convert_objects(convert_dates='coerce')
Out[3]: 
                    0
0 2005-03-25 00:00:00
1 2001-03-27 00:00:00

In [4]: df.convert_objects(convert_dates='coerce').dtypes
Out[4]: 
0    datetime64[ns]
dtype: object

fyi...obv 0.11 needed (which assume you are using)