Cannot print DataFrame with unicode columns in IPython

[craustin]

from pandas import DataFrame
df = DataFrame({u'c/\u03c3':[1,2,3]})

Try typing 'df' in IPython:
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03c3' in position 5: ordinal not in range(128)

[craustin]

We've seen this bug in DataFrame.to_string as well.

[wesm]

@takluyver do you know if there any workarounds for Python 2.x? Basically __repr__ can't return unicode in Python 2.x. this works, though:

In [7]: print df.to_string()
   c/σ
0  1  
1  2  
2  3  

[craustin]

Sorry for the premature post, this is the code that produces the original issue I intended to report. Let me know if you want a new issue.

from pandas import DataFrame
df = DataFrame({u'c/\u03c3':[1,2,3]})
df.to_string(formatters={u'c/\u03c3': lambda x: '%s' % x})

[takluyver]

If you've got unicode on Python 2, you can encode it just before you return from __repr__. Then you run into the question of picking an encoding. On modern Linux & OS X, the terminal encoding is usually UTF-8. On Windows, it's the current code page. IPython has code to try to detect the terminal encoding, but there's no guarantee that you're running in a terminal, so there might not be a 'correct' choice.

The best options are: 1) check sys.stdin.encoding, being aware that stdin can be None, and encoding can be None, or 2) avoid the whole question, and .encode('unicode-escape') or .encode('ascii', 'replace').

[wesm]

Wow, thanks @takluyver as I don't think I would have had the patience to figure that all out on my own. The referenced commit with changes and unit tests makes all the above work fine and typing df into the console works now (prints the \sigma). I need to make sure that it works alright on Windows too whenever I have a chance but I suspect so. Also may be able to close #340 once I check it out

[lodagro]

Somehow this does not work for me, see log of failing unittests at the bottom.

I tried it interactively also:

In [1]: cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:from pandas import DataFrame
:df = DataFrame({u'c/\u03c3':[1,2,3]})
:--

In [2]: print df.to_string()
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
...
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 2: ordinal not in range(256)

In [3]: df
Out[3]: ---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
...
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 2: ordinal not in range(256)

In [4]: print u'c/\u03c3'
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
/projects/hardware/users/wovermei/sandbox/pandas/sandbox/<ipython-input-4-45529fc7c4b5> in <module>()
----> 1 print u'c/\u03c3'

UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 2: ordinal not in range(256)

In [5]: import sys

In [6]: sys.stdin.encoding
Out[6]: 'ISO-8859-1'

ERROR: test_to_string_repr_unicode (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/tests/test_frame.py", line 1823, in test_to_string_repr_unicode
    df.to_string(col_space=10, buf=buf)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/frame.py", line 969, in to_string
    formatter.to_string()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 195, in to_string
    fmt_values = self._format_col(c)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 248, in _format_col
    formatter)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 429, in _format_fixed_width
    formatted = [formatter(x) for x in values]
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 226, in formatter
    col_width=col_width)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 603, in _format
    return _just_help('%s' % _stringify(s))
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 512, in _stringify
    return '%s' % console_encode(col)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 823, in console_encode
    return value.encode(sys.stdin.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 0: ordinal not in range(256)

======================================================================
ERROR: test_to_string_unicode_columns (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/tests/test_frame.py", line 1832, in test_to_string_unicode_columns
    df.to_string(buf=buf)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/frame.py", line 969, in to_string
    formatter.to_string()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 189, in to_string
    str_columns = self._get_formatted_column_labels()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 370, in _get_formatted_column_labels
    fmt_columns = self.columns.format()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/index.py", line 269, in format
    result.extend(_stringify(x) for x in self)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/index.py", line 269, in <genexpr>
    result.extend(_stringify(x) for x in self)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 512, in _stringify
    return '%s' % console_encode(col)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 823, in console_encode
    return value.encode(sys.stdin.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 0: ordinal not in range(256)

======================================================================
ERROR: test_to_string_with_formatters_unicode (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/tests/test_frame.py", line 1841, in test_to_string_with_formatters_unicode
    result = df.to_string(formatters={u'c/\u03c3': lambda x: '%s' % x})
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/frame.py", line 969, in to_string
    formatter.to_string()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 189, in to_string
    str_columns = self._get_formatted_column_labels()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 370, in _get_formatted_column_labels
    fmt_columns = self.columns.format()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/index.py", line 269, in format
    result.extend(_stringify(x) for x in self)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/index.py", line 269, in <genexpr>
    result.extend(_stringify(x) for x in self)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 512, in _stringify
    return '%s' % console_encode(col)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 823, in console_encode
    return value.encode(sys.stdin.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 2: ordinal not in range(256)

======================================================================
ERROR: test_repr_unicode (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/tests/test_series.py", line 752, in test_repr_unicode
    repr(s)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/series.py", line 558, in __repr__
    name=True)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/series.py", line 596, in _get_repr
    return formatter.to_string()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 118, in to_string
    fmt_values = self._get_formatted_values()
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 105, in _get_formatted_values
    fmt_values.append(' %s' % self.formatter(v))
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/format.py", line 58, in formatter
    col_width=col_width)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 603, in _format
    return _just_help('%s' % _stringify(s))
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 512, in _stringify
    return '%s' % console_encode(col)
  File ".../lib/python2.7/site-packages/pandas-0.7.0.dev_b4ca18b-py2.7-linux-x86_64.egg/pandas/core/common.py", line 823, in console_encode
    return value.encode(sys.stdin.encoding)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u03c3' in position 0: ordinal not in range(256)

[takluyver]

I guess that gives the lie to my assertion that all modern Linux uses UTF-8 as the terminal encoding. What distro is that, @lodagro ?

It should be an easy fix - see my comments on b4ca18b.

[lodagro]

Linux version 2.6.18-238.5.1.el5 (mockbuild@x86-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Mon Feb 21 05:52:39 EST 2011

[takluyver]

RHEL 5.1?

[lodagro]

no 5.1 available, but 5.6, same issue.

LSB Version:    :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Release:        5.6
Codename:       Tikanga

[lodagro]

I changed my linux LANG environment variable. pandas testing runs fine now, print on screen is not as expected though (\sigma is not shown).

[1115][i] more /etc/sysconfig/i18n 
SYSFONT="latarcyrheb-sun16"
LANG="en_US"
LC_COLLATE="C"

[1116][i] echo $LANG  
en_US

[1117][i] export LANG=en_US.UTF-8 

[1118][i] echo $LANG 
en_US.UTF-8

In [1]: import sys

In [2]: sys.stdin.encoding
Out[2]: 'UTF-8'

In [3]: cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:from pandas import DataFrame
:df = DataFrame({u'c/\u03c3':[1,2,3]})
:--

In [4]: df
Out[4]:
   c/Ï
0  1
1  2
2  3

[takluyver]

Python is now able to encode it (any character can be encoded to UTF-8), but I guess your terminal is still expecting ISO-8859-1 encoded text, so it doesn't display properly. If what your terminal expects doesn't match the encoding Python thinks it should be using, output will inevitably get messed up.