Pandas Series: maximum recursion error when replacing empty lists

[FXLab91]

I have a pandas Series containing lists. I wanted to replace empty lists with NaN. My first approach was using .replace, but that unexpectedly gave me a maximum recursion error.

I am using pandas 0.21.1 on Python 3.4, but other users had the same problem on 0.23

import numpy as np
import pandas as 
ts = pd.Series([[1], [2, 3], [], [4]])
ts.replace([], np.nan)

RuntimeError: maximum recursion depth exceeded in comparison

I achieved the desired output using:

ts[ts.apply(len) == 0] = np.nan

but I don't understand the maximum recursion error

[WillAyd]

Thanks for the report - investigation and PRs are always welcome!

[jschendel]

xref #20380: another RecursionError issue with DataFrame.replace

[h-vetinari]

Lists in replace are very sensitive (I'm guessing because they get interpreted somehow), see also #19266

[minggli]

happy to look at this issue and try to fix.

[minggli]

Hi @WillAyd,

Quick question, is list as single element to scalar replacement a supported use case for .replace? the documentation doesn't include this use case it appears.

Have done some investigation and it appears supporting this use case can break a lot of other use cases. I feel replacing list as one data element in a series or frame is not a supported use case from the look of it.

However, it should not raise recursion error. I think it's best to handle this issue as error handling.

[WillAyd]

@minggli if it breaks a lot of stuff then your suggestion of handling as an error is probably better. We don't have first class support for containers within pandas objects, and theoretically won't until we can get those established as Extension Arrays.

cc @jreback if he has any other thoughts but like I said I'd be OK with error handling for now