BUG: Weird console formatting defaults

[wesm]

Working on book updates and was surprised to see pandas do this:

before

 In [99]: frame
 Out[99]: 
-   Ohio  Texas  California
-a     0      1           2
-c     3      4           5
-d     6      7           8

after

+   Ohio     ...      California
+a     0     ...               2
+c     3     ...               5
+d     6     ...               8
+[3 rows x 3 columns]

Why is the middle column being hidden in a small 3x3 data frame?

[wesm]

This is on git master

[wesm]

Update: this seems to only be happening inside a Docker container since it's failing to detect the console dimensions. I guess the "headless IPython" use case is pretty unusual

[wesm]

The workaround is to set pd.options.display.max_colwidth = 70 or something similar

[wesm]

Here's another one:

-In [163]: info[:5]
-Out[163]: 
-                          description                   group    id  \
-0                     Cheese, caraway  Dairy and Egg Products  1008   
-1                     Cheese, cheddar  Dairy and Egg Products  1009   
-2                        Cheese, edam  Dairy and Egg Products  1018   
-3                        Cheese, feta  Dairy and Egg Products  1019   
-4  Cheese, mozzarella, part skim milk  Dairy and Egg Products  1028   
-  manufacturer  
-0               
-1               
-2               
-3               
-4               
+In [164]: info[:5]
+Out[164]: 
+                          description     ...      manufacturer
+0                     Cheese, caraway     ...                  
+1                     Cheese, cheddar     ...                  
+2                        Cheese, edam     ...                  
+3                        Cheese, feta     ...                  
+4  Cheese, mozzarella, part skim milk     ...                  

What is the option to do the old behavior (print the overflowing columns on the next line)?

[wesm]

Appears I can make more of the problems going away with this (in case anyone stumbles on this thread later):

export COLUMNS=80
export LINES=50
stty cols $COLUMNS rows $LINES

[wesm]

Another example of odd formatting even though the console width is set to 80

    .....:                  columns='smoker', margins=True)
-Out[132]: 
-                 size                       tip_pct                    
-smoker             No       Yes       All        No       Yes       All
-time   day                                                             
-Dinner Fri   2.000000  2.222222  2.166667  0.139622  0.165347  0.158916
-       Sat   2.555556  2.476190  2.517241  0.158048  0.147906  0.153152
-       Sun   2.929825  2.578947  2.842105  0.160113  0.187250  0.166897
-       Thur  2.000000       NaN  2.000000  0.159744       NaN  0.159744
-Lunch  Fri   3.000000  1.833333  2.000000  0.187735  0.188937  0.188765
-       Thur  2.500000  2.352941  2.459016  0.160311  0.163863  0.161301
-All          2.668874  2.408602  2.569672  0.159328  0.163196  0.160803</programlisting>
+Out[133]: 
+                 size              ...      tip_pct          
+smoker             No       Yes    ...          Yes       All
+time   day                         ...                       
+Dinner Fri   2.000000  2.222222    ...     0.165347  0.158916
+       Sat   2.555556  2.476190    ...     0.147906  0.153152
+       Sun   2.929825  2.578947    ...     0.187250  0.166897
+       Thur  2.000000       NaN    ...          NaN  0.159744
+Lunch  Fri   3.000000  1.833333    ...     0.188937  0.188765
+       Thur  2.500000  2.352941    ...     0.163863  0.161301
+All          2.668874  2.408602    ...     0.163196  0.160803
+[7 rows x 6 columns]</programlisting>

[wesm]

Another example of a dataset that looks fine in 0.20.x and not in 0.23.x

-In [74]: data
-Out[74]: 
-         user_id  movie_id  rating  timestamp gender  age  occupation    zip  \
-0              1      1193       5  978300760      F    1          10  48067   
-1              2      1193       5  978298413      M   56          16  70072   
-2             12      1193       4  978220179      M   25          12  32793   
-3             15      1193       4  978199279      M   25           7  22903   
-4             17      1193       5  978158471      M   50           1  95350   
-...          ...       ...     ...        ...    ...  ...         ...    ...   
-1000204     5949      2198       5  958846401      M   18          17  47901   
-1000205     5675      2703       3  976029116      M   35          14  30030   
-1000206     5780      2845       1  958153068      M   18          17  92886   
-1000207     5851      3607       5  957756608      F   18          20  55410   
-1000208     5938      2909       4  957273353      M   25           1  35401   
-                                               title                genres  
-0             One Flew Over the Cuckoo's Nest (1975)                 Drama  
-1             One Flew Over the Cuckoo's Nest (1975)                 Drama  
-2             One Flew Over the Cuckoo's Nest (1975)                 Drama  
-3             One Flew Over the Cuckoo's Nest (1975)                 Drama  
-4             One Flew Over the Cuckoo's Nest (1975)                 Drama  
-...                                              ...                   ...  
-1000204                           Modulations (1998)           Documentary  
-1000205                        Broken Vessels (1998)                 Drama  
-1000206                            White Boys (1999)                 Drama  
-1000207                     One Little Indian (1973)  Comedy|Drama|Western  
-1000208  Five Wives, Three Secretaries and Me (1998)           Documentary  
-[1000209 rows x 10 columns]
+    <programlisting language="python" format="linespecific">In [74]: data = pd.merge(pd.merge(ratings, users), movies)
 
-In [75]: data.iloc[0]
+In [75]: data
 Out[75]: 
+         user_id          ...                         genres
+0              1          ...                          Drama
+1              2          ...                          Drama
+2             12          ...                          Drama
+3             15          ...                          Drama
+4             17          ...                          Drama
+...          ...          ...                            ...
+1000204     5949          ...                    Documentary
+1000205     5675          ...                          Drama
+1000206     5780          ...                          Drama
+1000207     5851          ...           Comedy|Drama|Western
+1000208     5938          ...                    Documentary
+[1000209 rows x 10 columns]

[gfyoung]

cc @jreback

[wesm]

I can try to dig into this, but if there's a temporary workaround it would be helpful to know. This is the last item blocking me releasing accumulated errata fixes to be published

[WillAyd]

Does doing pd.options.display.max_columns = 20 not give you what you are looking for, namely the behavior before #17023

[jorisvandenbossche]

Yes, doing that (pd.options.display.max_columns = 20) should restore the old behaviour. See also the whatsnew entry here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#better-pretty-printing-of-dataframes-in-a-terminal

That said, I personally have had several times that the new truncating behaviour was somewhat annoying.

The workaround is to set pd.options.display.max_colwidth = 70 or something similar

I don't think this should influence that

[wesm]

FWIW I think the new behavior is worse. The old behavior is more consistent with what other projects (like R) do IIUC

[wesm]

I also think the assertion that it is "better" or that the old behavior "was relatively difficult to read" is truly in the eye of the beholder / very subjective