Unicode arguments in str.format() #51549

Activity

1. doerwalter commented on Nov 10, 2009

   

   doerwalter

   on Nov 10, 2009

   ContributorAuthor

   More actions

   str.format() doesn't handle unicode arguments:

   Python 2.6.4 (r264:75706, Oct 27 2009, 15:18:04) 
   [GCC 4.2.1 (Apple Inc. build 5646)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> '{0}'.format(u'\u3042')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   UnicodeEncodeError: 'ascii' codec can't encode character u'\u3042' in
   position 0: ordinal not in range(128)

   Unicode arguments should be treated in the same way as the % operator
   does it: by promoting the format string to unicode:

   >>> '%s' % u'\u3042'
   u'\u3042'
2. doerwalter

   assigned

   ericvsmith

   on Nov 10, 2009
3. doerwalter

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Nov 10, 2009
4. vstinner commented on Mar 9, 2010

   

   vstinner

   on Mar 9, 2010

   Member

   More actions

   PyString_Format() uses a "goto unicode;" if a '%c' or '%s' argument is unicode. The unicode label converts the partial formatted result (byte string) to unicode, and use PyUnicode_Format() to finish to formatting.

   I don't think that you can apply the same algorithm here (converts the partial result to unicode) because it requires to rewrite the format string: arguments can be used twice or more, and used in any order.

   Example: "{0} {1}".format("bytes", u"unicode") => switch to unicode occurs at result="bytes ", format=" {1}", arguments=(u"unicode"). Converts "bytes " to unicode is easy, but the format have to be rewritten in " {0}" or something else.

   Call trace of str.format(): do_string_format() -> build_string() -> output_markup() -> render_field(). The argument type is proceed in render_field().
5. vstinner commented on Mar 9, 2010

   

   vstinner

   on Mar 9, 2010

   Member

   More actions

   *Draft* patch fixing the issue: render_field() raises an error if the argument is an unicode argument, string_format() catchs this error and converts self to unicode and call unicode.format(*args, **kw).

   Pseudo-code:

   try:
   # self.format() raises an error if any argument is
   # an unicode string)
   return self.format(*args,**kw)
   except UnicodeError:
   unicode = self.decode(default_encoding)
   return unicode.format(*args, **kw)

   The patch changes the result type of '{}'.format(u'ascii'): it was str and it becomes unicode. The new behaviour is consistent with "%s" % u"ascii" => u"ascii" (unicode).

   I'm not sure that catching *any* unicode error is a good idea. I think that it would be better to use a new exception type dedicated to this issue, but it looks complex to define a new exception. I will may do it for the next patch version ;-)
6. vstinner commented on Mar 9, 2010

   

   vstinner

   on Mar 9, 2010

   Member

   More actions

   My patch converts the format string to unicode using the default encoding. It's inconsistent with str%args: str%args converts str to unicode using the ASCII charset (if a least one argument is an unicode string), not the default encoding.

   >>> "\xff%s" % u'\xe9'
   ...
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
7. ericvsmith commented on Mar 11, 2010

   

   ericvsmith

   on Mar 11, 2010

   Member

   More actions

   I'm not sure I'm wild about doing the work twice, once as string and once as unicode if need be. But I'll consider it, especially since this is only a 2.7 issue.

   There could be side effects of evaluating the replacement strings, but I'm not sure it's worth worrying about. Attribute (or index) access having side effects isn't something I think we need to cater to.
8. s0undt3ch commented on Dec 30, 2012

   

   s0undt3chmannequin

   on Dec 30, 2012

   Mannequin

   More actions

   This is not a 2.7 issue only:

   >>> import sys
   >>> sys.version_info
   (2, 6, 5, 'final', 0
   >>> 'Foo {0}'.format(u'bár')
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
   UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 1: ordinal not in range(128)
   >>>
9. ezio-melotti commented on Dec 30, 2012

   

   ezio-melotti

   on Dec 30, 2012

   Member

   More actions

   2.6 only gets security fixes.

   My patch converts the format string to unicode using the default
   encoding. It's inconsistent with str%args: str%args converts str to
   unicode using the ASCII charset (if a least one argument is an unicode
   string), not the default encoding.

   I think it's better to be consistent and use ASCII.
10. vstinner commented on Dec 30, 2012

    

    vstinner

    on Dec 30, 2012

    Member

    More actions

    Another option is to decide that this issue will *not* be fixed in Python 2, and Python 3 *is* the good solution if you have this issue.

    Doing the work twice can cause new problems, formatting an argument twice may return two different values :-( It may have an impact on performances and may introduce regressions.

    Oh by the way, it's trivial to workaround this issue in Python 2: just use a Unicode format string. For example, replace '{0}'.format(u'\u3042') with u'{0}'.format(u'\u3042').

    I hate implicit conversion from bytes to Unicode in Python 2, it's maybe better to not add a new special case?
11. ericvsmith commented on Dec 30, 2012

    

    ericvsmith

    on Dec 30, 2012

    Member

    More actions

    I agree that we should close this as "won't fix" in 2.7.
12. birkenfeld commented on Mar 28, 2013

    

    birkenfeld

    on Mar 28, 2013

    Member

    More actions

    Agreed with Eric.
13. birkenfeld

    closed this as completedon Mar 28, 2013
14. ezio-melotti

    transferred this issue fromon Apr 10, 2022