Skip to content

PyUnicode_FromFormatV() doesn't support utf-8 text #60547

Closed

Description

@reingart
mannequin
BPO 16343
Nosy @vstinner, @ezio-melotti, @cjerdonek
Superseder
  • bpo-9769: PyUnicode_FromFormatV() doesn't handle non-ascii text correctly
  • Files
  • pyunicode_fromformat_utf8.patch: PyUnicode_FromFormatV patch to use UTF-8
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-10-28.20:14:48.045>
    created_at = <Date 2012-10-28.04:06:11.172>
    labels = ['interpreter-core', 'type-feature', 'expert-unicode']
    title = "PyUnicode_FromFormatV() doesn't support utf-8 text"
    updated_at = <Date 2012-10-28.20:14:48.044>
    user = 'https://bugs.python.org/reingart'

    bugs.python.org fields:

    activity = <Date 2012-10-28.20:14:48.044>
    actor = 'chris.jerdonek'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-10-28.20:14:48.045>
    closer = 'chris.jerdonek'
    components = ['Interpreter Core', 'Unicode']
    creation = <Date 2012-10-28.04:06:11.172>
    creator = 'reingart'
    dependencies = []
    files = ['27755']
    hgrepos = []
    issue_num = 16343
    keywords = ['patch']
    message_count = 4.0
    messages = ['173996', '174022', '174072', '174073']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'ezio.melotti', 'chris.jerdonek', 'reingart']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '9769'
    type = 'enhancement'
    url = 'https://bugs.python.org/issue16343'
    versions = ['Python 3.4']

    Activity

    1. reingart commented on Oct 28, 2012

      reingartmannequin
      MannequinAuthor

      Working in an internationalization proposal <http://python.org.ar/pyar/TracebackInternationalizationProposal\>
      I've stopped at bpo-9769 where multi byte encodings (like utf-8) is not supported by PyUnicode_FromFormatV()

      Beside my proposal, I think utf-8 should be supported for consistency with the other unicode functions, like PyUnicode_FromString() or even unicode_fromformat_arg()

      Attached is a patch that:

      • enhanced the iterator to detect multibyte sequences, with sanity checks about start & continuation bytes
      • replaced unicode_write_cstr with PyUnicode_DecodeUTF8Stateful
      • tests

      Hope it helps, this is my first patch for cpython and my C skills are a bit rusty, so excuse me if there is any newbie glitch

    2. cjerdonek commented on Oct 28, 2012

      @cjerdonek
      Member

      Shouldn't this patch be attached to the referenced bpo-9769 instead of creating a new issue? Even the issue title is nearly the same:

      9769: PyUnicode_FromFormatV() doesn't handle non-ascii text correctly
      16343: PyUnicode_FromFormatV() doesn't support utf-8 text

    3. reingart commented on Oct 28, 2012

      reingartmannequin
      MannequinAuthor

      I thought bpo-9769 was closed (in fact, that patch was already applied).
      Now, PyUnicode_FromFormatV() doesn't handle non-ascii text at all.
      Maybe I misread the part telling to open a new issue in the comments, sorry for that.

    4. cjerdonek commented on Oct 28, 2012

      @cjerdonek
      Member

      bpo-9769 is still open. It looks like there was some disagreement in the comments between Alexander and Victor as to whether a new issue should be created (since Victor had a different idea when first opening the issue), but it looks like Victor deferred to Alexander in his final comment.

      I will close this issue as a duplicate, and if you could repost your patch on the other issue, that would be great. The discussion there is relevant to your patch.

    5. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions