object arrays converted to string arrays of 'S' dtype have default length. (Trac #1892)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/1892 on 2011-07-04 by trac user ehiggs, assigned to unknown.

I'm sure this must have been reported before, but I can't find it in the bug tracker. The bug is that calling a vectorized function on a string type truncates the string to '|S8' regardless of the original length.

Example:

In [1]: import numpy

In [2]: def __f(x):

...: return x

...:

In [3]: f = numpy.vectorize(__f)

In [4]: s = '2011-07-04'

In [5]: y = f(s)

In [6]: y

Out[6]:
array('2011-07-04',
dtype='|S10')

In [8]: z = f(numpy.array([s,s,s,s]))

In [9]: z
Out[9]:
array(['2011-07-', '2011-07-', '2011-07-', '2011-07-'],
dtype='|S8')

If this has been raised, please merge this in so I can track progress on the original.

[numpy-gitbot]

trac user ehiggs wrote on 2011-07-04

This is also a problem in 1.4.1:

import numpy
numpy.version
'1.4.1'

numpy.vectorize(lambda x: x[:10], otypes=[numpy.dtype('|S10')])(numpy.array(['a'*20]))
array(['aaaaaaaa'],
dtype='|S8')

[numpy-gitbot]

trac user ehiggs wrote on 2011-07-04

Now with code block formatting:

>>> import numpy
>>> numpy.__version__
    '1.4.1'

>>> numpy.vectorize(lambda x: x[:10], otypes=[numpy.dtype('|S10')])(numpy.array(['a'*20]))

array(['aaaaaaaa'],
      dtype='|S8')

I apologize if there is an edit function in Trac. I'm new to this bug tracker so I'm still getting up to speed.

[numpy-gitbot]

trac user jordigh wrote on 2011-08-10

Attempting to diagnose this problem, I can reproduce it with the following lines:

## Truncates to type '|S8'
array(array(['looooooool'],dtype=object),dtype='S')

## These three don't truncate
array(array(['looooooool'],dtype=object),dtype='|S10')
array(array(['looooooool']),dtype='S')
array(array(['looooooool']))

so I'm not sure if the bug is in vectorize itself or if in array. Is the truncation above desirable?

[numpy-gitbot]

@charris wrote on 2011-08-13

Looks like numpy needs to know how long the string for the object arrays is going to be before creation and falls back to a default size instead of trying to determine it.

The default size is a bit larger on my machine with current master:

In [2]: array(array(['0123456789'*10],dtype=object),dtype='S')
Out[2]: 
array(['0123456789012345678901234567890123456789012345678901234567890123'], 
      dtype='|S64')

[numpy-gitbot]

Title changed from numpy.vectorize truncates strings to object arrays converted to string arrays of 'S' dtype have default length. by @charris on 2011-08-13

[numpy-gitbot]

trac user ehiggs wrote on 2011-08-15

Replying to [comment:4 charris]:

so I'm not sure if the bug is in vectorize itself or if in array. Is the truncation above desirable?

There appears to be at least one bug with vectorize. If I've given an input dtype and an otypes argument to vectorize, I think any sort of array manipulation should be using one of these dtypes to know the length of the strings. However, this information is still lost. For example:

In [7]: numpy.vectorize(lambda x: x[:10], otypes=[numpy.dtype('|S10')])(numpy.array(['a'*20], dtype=numpy.dtype('|S20')))
Out[7]: 
array(['aaaaaaaa'], 
      dtype='|S8')

Looks like numpy needs to know how long the string for the object arrays is going to be before creation and falls back to a default size instead of trying to determine it.

I'm not sure how the interface to array should handle the default case. However, if I provide an otypes argument to vectorize, I think that this should help the array interface understand what I believe the length of the string to be. This didn't appear to be the case in the following example:

>>> numpy.vectorize(lambda x: x[:10], otypes=[numpy.dtype('|S10')])(numpy.array(['a'*20]))

array(['aaaaaaaa'],
      dtype='|S8')

I'm not familiar with the underlying code, but perhaps vectorize could pick up the otypes argument and offer it as the dtype to any object->string conversions in the function. As jordigh pointed out, array creation works fine when the string length is specified, so this would at least protect vectorize from ignoring the intended output type.

Thanks for taking a look.

[numpy-gitbot]

trac user jordigh wrote on 2011-09-16

I'd like to get this fixed. It's easiest for me to patch vectorize to use more specific string types, which would avoid the truncated strings. Would such a patch be accepted?

[numpy-gitbot]

trac user jordigh wrote on 2011-12-13

Ping. I would still like to get this fixed. Would a workaround for np.vectorize that uses more specific types be a good patch?

[kolesman]

Ping. It looks like a serious bug. Can somebody fix it?

[seberg]

This has been fixed in the gh-3270 by javius.