string.Formatter returns str for empty unicode template #60155

Activity

1. AlekseySivokon commented on Sep 16, 2012

   

   AlekseySivokonmannequin

   on Sep 16, 2012

   MannequinAuthor

   More actions

   Expected behavior of string.Formatter() is to return unicode strings for unicode templates, and "byte" strings for str templates. Which is exactly what it does, with one frustrating exception: for empty unicode string it returns byte str. Test follows:

   import string
   template = u""
   result = string.Formatter().format(template)
   assert isinstance(result, unicode)
   # AssertionError
2. AlekseySivokon

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 16, 2012
3. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   Member

   More actions

   Adding failing test. Patch coming next.
4. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   Member

   More actions

   Here are some related failing cases that I found:

   >>> f = string.Formatter()
   >>> f.format(u"{0}", "")
   ''
   >>> f.format(u"{0}", 1)
   '1'
   >>> f.format(u"{0}", "a")
   'a'
   >>> f.format(u"{0}{1}", "a", "b")
   'ab'
   >>> f.format("{0}", u"a") 
   u'a'

   Note that PEP-3101 says the following:

   "In all cases, the type of the format string dominates - that
   is, the result of the conversion will always result in an object
   that contains the same representation of characters as the
   input format string."
5. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   Member

   More actions

   Actually, I'm going to defer on creating a patch because this covers more scenarios than I originally thought and so may require more time.
6. bitdancer commented on Sep 16, 2012

   

   bitdancer

   on Sep 16, 2012

   Member

   More actions

   Format with unicode is a bit of a mess in 2.7. It would be consistent with the rest of python2 for

     >>> f.format("{0}", u"a")
     u'a'

   to be correct.

   See also bpo-7300 and bpo-15276.
7. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   Member

   More actions

   What about cases like this?

   >>> f.format(u'{0}', '\xe9')
   '\xe9'

   It seems fixing this issue for non-empty strings would cause formerly running cases like this to raise UnicodeDecodeError.

   >>> unicode('\xe9')
    ...
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128)

   Would that be acceptable?
8. bitdancer commented on Sep 16, 2012

   

   bitdancer

   on Sep 16, 2012

   Member

   More actions

   Note that I didn't say it was correct, I just said it was consistent :)

   And no, breaking stuff that current works is a non-starter for 2.7.
9. cjerdonek commented on Sep 16, 2012

   

   cjerdonek

   on Sep 16, 2012

   Member

   More actions

   I filed bpo-15952 for the behavior difference between format(value) and value.__format__() and the related lack of documentation re: unicode format strings.

   Given that the expected behavior for the current issue doesn't seem to be documented (aside from PEP-3101, which is probably too late to follow), we should probably agree on what the behavior should be (as well as documenting it) before or while addressing this issue.
10. cjerdonek commented on Sep 19, 2012

    

    cjerdonek

    on Sep 19, 2012

    Member

    More actions

    Attached is a proposed patch.

    Some explanation behind the patch that stems from the above comments:

    The following is an example of Formatter.format() returning str in the current implementation that would break if we made Formatter.format() return unicode whenever format_string is unicode:

    >>> f.format(u"{0}", "\xc3\xa9")  # UTF-8 encoded "e-acute".
    '\xc3\xa9'

    (It would break with a UnicodeDecodeError because 'ascii' is the default encoding.)

    Since we can't change Formatter.format(format_string) to return unicode whenever format_string is unicode without breaking existing code, I believe the best we can do is to document the departure from PEP-3101. Since the caller has to handle return values of type str anyways, I don't think it helps to ensure that more return values are unicode.
11. serhiy-storchaka

    closed this as completedon May 31, 2020
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022