format(value) and value.__format__() behave differently with unicode format #60156

Activity

1. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   MemberAuthor

   More actions

   format(value) and value.__format__() behave differently even though the documentation says otherwise:

   "Note: format(value, format_spec) merely calls value.__format__(format_spec)."

   (from http://docs.python.org/library/functions.html?#format )

   The difference happens when the format string is unicode. For example:

   >>> format(10, u'n')
   u'10'
   >>> (10).__format__(u'n')  # parentheses needed to prevent SyntaxError
   '10'

   So either the documentation should be changed, or the behavior should be changed to match.

   Related to this: neither the "Format Specification Mini-Language" documentation nor the string.Formatter docs seem to say anything about the effect that a unicode format string should have on the return value (in particular, should it cause the return value to be unicode or not):

   http://docs.python.org/library/string.html#formatspec
   http://docs.python.org/library/string.html#string-formatting

   See also bpo-15276 (int formatting), bpo-15951 (empty format string), and bpo-7300 (unicode arguments).
2. cjerdonek

   assigned

   docspython

   on Sep 16, 2012
3. cjerdonek

   added

   docsDocumentation in the Doc dirDocumentation in the Doc dir

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 16, 2012
4. cjerdonek commented on Sep 17, 2012

   

   cjerdonek

   on Sep 17, 2012

   MemberAuthor

   More actions

   See this code comment:

   /* don't define FORMAT_LONG, FORMAT_FLOAT, and FORMAT_COMPLEX, since
   we can live with only the string versions of those. The builtin
   format() will convert them to unicode. */

   from http://hg.python.org/cpython/file/19601d451d4c/Python/formatter_unicode.c

   In other words, it was deliberate not to make value.__format__(format_spec) return unicode when format_spec is unicode. So the docs should be adjusted to say that they are not always the same.
5. ericvsmith commented on Sep 17, 2012

   

   ericvsmith

   on Sep 17, 2012

   Member

   More actions

   I believe the conversion is happening in Objects/abstract.c in PyObject_Format, around line 864, near this comment:

   /* Convert to unicode, if needed.  Required if spec is unicode
      and result is str */
   

   I think changing the docs will result in more confusion than clarity, but if you can come up with some good wording, I'd be okay with it. I think changing the code will likely break things with little or no benefit.
6. cjerdonek commented on Sep 18, 2012

   

   cjerdonek

   on Sep 18, 2012

   MemberAuthor

   More actions

   Here is a proposed patch.

   One note on the patch. I feel the second sentence of the note is worth adding because value.__format__() departs from what PEP-3101 says:

   "Note for Python 2.x: The 'format_spec' argument will be either
   a string object or a unicode object, depending on the type of the
   original format string. The __format__ method should test the type
   of the specifiers parameter to determine whether to return a string or
   unicode object. It is the responsibility of the __format__ method
   to return an object of the proper type."

   The extra sentence will help in heading off and when responding to issues about value.__format__() that are similar to bpo-15951.
7. cjerdonek commented on Sep 18, 2012

   

   cjerdonek

   on Sep 18, 2012

   MemberAuthor

   More actions

   To clarify, one of the sentences above should have read, "I feel the second sentence of the note *in the patch* was worth adding..." (not the second sentence of the PEP note I quoted).
8. ezio-melotti commented on Sep 23, 2012

   

   ezio-melotti

   on Sep 23, 2012

   Member

   More actions

   ``format(value, format_spec)`` merely calls
   -      ``value.__format__(format_spec)``.
   +      ``value.__format__(format_spec)`` and, if *format_spec* is Unicode,
   +      converts the value to Unicode if it is not already Unicode.

   This is correct, but should be rephrased (and "value" should be "return value").

   •  The method ``value.__format__(format_spec)`` may return 8-bit strings
     

   •  for some built-in types when *format_spec* is Unicode.
     

   This is not limited to built-in types. __format__() might return either str or unicode, and format() returns the same -- except for the aforementioned case.

   This is a summary of the possible cases.

   __format__ can return unicode or str:

     >>> class Uni(object):
     ...   def __format__(*args): return u'uni'
     ... 
     >>> class Str(object):
     ...   def __format__(*args): return 'str'
     ... 

   format() and __format__ return the same value, except when the format_spec is unicode and __format__ returns str:

     >>> format(Uni(),  'd'),  Uni().__format__( 'd')  # same
     (u'uni', u'uni')
     >>> format(Uni(), u'd'),  Uni().__format__(u'd')  # same
     (u'uni', u'uni')
     >>> format(Str(),  'd'),  Str().__format__( 'd')  # same
     ('str', 'str')
     >>> format(Str(), u'd'),  Str().__format__(u'd')  # different
     (u'str', 'str')

   It is also not true that the type of return value is the same of the format_spec, because in the first case the returned type is unicode even if the format_spec is str. Therefore this part of the patch should be changed:

   + Per :pep:`3101`, the function returns a Unicode object if *format_spec* is
   + Unicode. Otherwise, it returns an 8-bit string.

   The behavior might be against PEP-3101 (see quotation in msg170669), even thought the wording of the PEP is somewhat lenient IMHO ("proper type" doesn't necessary mean "same type").
9. serhiy-storchaka

   closed this as completedon May 31, 2020
10. ezio-melotti

    transferred this issue fromon Apr 10, 2022