unicodedata: unicodedata.name() undocumented TypeError on empty string

Bug report

Bug description:

Consider the following:

import unicodedata
unicodedata.name('')

unicodedata.name('a')

unicodedata.name('\x00')

Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    unicodedata.name('')
TypeError: name() argument 1 must be a unicode character, not str

'LATIN SMALL LETTER A'

Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    unicodedata.name('\x00')
ValueError: no such name

It seems there are two issues here:

unicodedata.name() and possibly other unicodedata methods can have an undocumented TypeError exception. https://docs.python.org/3/library/unicodedata.html
The TypeError on an empty string doesn't really seem accurate, since the function can still take other string values of the same str type.

Either TypeError should be mentioned in documentation, or the handling of empty strings should be fixed in the library to return a ValueError instead.

CPython versions tested on:

3.12

Operating systems tested on:

Windows, Other

Linked PRs

We do not try to document every possible exception, especially the common ones when they are completely clear as to the cause. There are other cases where one could argue that TypeError should be ValueError, or vice versa, but changing the status quo will break code that depends on the status quo. We could leave this alone.

The message for the empty string looks like it might possibly be a leftover from when str was bytes.

This appears to be an argument clinic issue, in particular the handling of int(accept={str}), which ends up in this part of argument clinic:

cpython/Tools/clinic/libclinic/converters.py

Lines 268 to 282 in 0edde64

    
                   elif self.format_unit == 'C': 
        
                       return self.format_code(""" 
        
                           if (!PyUnicode_Check({argname})) {{{{ 
        
                               {bad_argument} 
        
                               goto exit; 
        
                           }}}} 
        
                           if (PyUnicode_GET_LENGTH({argname}) != 1) {{{{ 
        
                               {bad_argument} 
        
                               goto exit; 
        
                           }}}} 
        
                           {paramname} = PyUnicode_READ_CHAR({argname}, 0); 
        
                           """, 
        
                           argname=argname, 
        
                           bad_argument=self.bad_argument(displayname, 'a unicode character', limited_capi=limited_capi), 
        
                       )

This ends up with the same general error format for invalid types (unicodedata.name(42.5)) and for strings with an unexpected length like here (both empty strings and strings longer than one code point).

pythongh-117557: Raise ValueError iso. TypeError if str length is wro…

I proposed a fix in #117558. Exceptions raised and their error messages are a part of the Python API. We should take this in consideration before eventually settling on the proposed solution.

Moreover, I agree with what Terry said: we do not need to¹ document every kind of exception that can be raised.

and in some cases we should not ↩

@serhiy-storchaka

cc. @serhiy-storchaka

Historically, it happened that in some cases character and string are treated as different types, even if technically characters are implemented as strings with length 1 in Python. It is not an Argument Clinic issue, PyArg_Parse* functions do the same, and maybe some handwritten parsers too.

It is a very old behavior, and I do not think that changing TypeError to ValueError is well justified here. It will break user code for sure. From semantic point of view, passing a non-1-character string is usually a programming error instead of a data driven error, and TypeError is more suitable for this.

Example of such behavior besides Argument Clinic and the PyArg_Parse* C API:

>>> 'abc'.center(10, '')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    'abc'.center(10, '')
    ~~~~~~~~~~~~^^^^^^^^
TypeError: The fill character must be exactly one character long

We could align Argument Clinic to match the error message of PyArg_Parse.

Other examples:

>>> ord('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    ord('')
    ~~~^^^^
TypeError: ord() expected a character, but string of length 0 found
>>> '%c' % ''
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    '%c' % ''
    ~~~~~^~~~
TypeError: %c requires int or char
>>> b'%c' % b''
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    b'%c' % b''
    ~~~~~~^~~~~
TypeError: %c requires an integer in range(256) or a single byte

In any case, the error message can be improved, so I think it is still an Argument Clinic issue.

If the code of the Argument Clinic converter looks weird to you, it is because it was aligned to match PyArg_Parse.

Example of such behavior besides Argument Clinic and the PyArg_Parse* C API:

>>> 'abc'.center(10, '')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    'abc'.center(10, '')
    ~~~~~~~~~~~~^^^^^^^^
TypeError: The fill character must be exactly one character long

Actually, this error message does not come from PyArg_Parse*, it comes from a custom Argument Clinic converter in Objects/unicodeobject.c. Perhaps we should consider embedding that converter in Argument Clinic for better reuse.

Yes, it is what I said. These examples do not use PyArg_Parse* or standard Argument Clinic converter.

As for the error message, I think the "a unicode character" part is fine, but "not str" may be confusing, because "a unicode character" is a string. If it was "not a string of length 2" or "not an empty string", it would be much more clear.

Yes, this is the issue ;)

If you agree, are you interesting to implement such change (adding the length instead of the type in the error message)? It requires slightly more complex changes than what you did in your PR. I could easily do it myself, but you've already gotten into this and I don't want to waste your efforts.

If you agree, are you interesting to implement such change (adding the length instead of the type in the error message)? It requires slightly more complex changes than what you did in your PR. I could easily do it myself, but you've already gotten into this and I don't want to waste your efforts.

Yep, I agree we should do that. I don't have time for it the following days, but perhaps within the week, or the following week.

Well, then I have created it myself. It is much larger if include all cases.

gh-117557: Improve error messages when a string, bytes or bytearray o…

pythongh-117557: Improve error messages when a string, bytes or bytea…

Navigation Menu

Repository navigation

unicodedata: unicodedata.name() undocumented TypeError on empty string #117557

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	elif self.format_unit == 'C':
	return self.format_code("""
	if (!PyUnicode_Check({argname})) {{{{
	{bad_argument}
	goto exit;
	}}}}
	if (PyUnicode_GET_LENGTH({argname}) != 1) {{{{
	{bad_argument}
	goto exit;
	}}}}
	{paramname} = PyUnicode_READ_CHAR({argname}, 0);
	""",
	argname=argname,
	bad_argument=self.bad_argument(displayname, 'a unicode character', limited_capi=limited_capi),
	)

unicodedata: unicodedata.name() undocumented TypeError on empty string #117557

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

terryjreedy commented on Apr 5, 2024

ronaldoussoren commented on Apr 5, 2024

erlend-aasland commented on Apr 5, 2024

Footnotes

erlend-aasland commented on Apr 5, 2024

serhiy-storchaka commented on Apr 5, 2024

serhiy-storchaka commented on Apr 5, 2024

erlend-aasland commented on Apr 5, 2024

serhiy-storchaka commented on Apr 5, 2024

erlend-aasland commented on Apr 5, 2024

serhiy-storchaka commented on Apr 5, 2024

erlend-aasland commented on Apr 5, 2024

serhiy-storchaka commented on Apr 5, 2024

serhiy-storchaka commented on Apr 6, 2024

erlend-aasland commented on Apr 6, 2024

serhiy-storchaka commented on Apr 7, 2024

erlend-aasland commented on Apr 8, 2024

serhiy-storchaka commented on Apr 8, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions