Skip to content

DataFrame.to_parquet raises fails when dataframe is empty #27339

Closed
@CJStadler

Description

@CJStadler

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame()
pd.to_parquet('empty.parquet')

Problem description

DataFrame.to_parquet raises ValueError: parquet must have string column names when the dataframe has no columns. pyarrow and fastparquet both support writing empty dataframes so I don't think this restriction is necessary (see below script).

import fastparquet as fp
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd

df = pd.DataFrame()
fn = 'empty.parquet'

# Test fastparquet
fp.write(fn, df)

# Test pyarrow
table = pa.Table.from_pandas(df)
pq.write_table(table, fn)

# This will fail
df.to_parquet(fn)

I'm happy to open a PR if it's agreed that this is an issue.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Darwin
OS-release: 18.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: None
pip: 19.1.1
setuptools: 41.0.1
Cython: None
numpy: 1.16.4
scipy: None
pyarrow: 0.14.0
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions