Skip to content

Update html.entities.html5 dictionary and parseentities.py #60449

Closed
Closed
Update html.entities.html5 dictionary and parseentities.py#60449
Assignees
ezio-melotti
Labels
deferred-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@ezio-melotti
ezio-melotti
opened on Oct 16, 2012
BPO 16245
Nosy @birkenfeld, @terryjreedy, @larryhastings, @ezio-melotti, @merwok, @kushaldas
Files
  • issue16245.diff: New Tools/scripts/parse_html5_entities.py
  • issue16245-2.diff
  • issue16245-3.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ezio-melotti'
closed_at = <Date 2014-03-16.22:01:33.346>
created_at = <Date 2012-10-16.09:41:51.441>
labels = ['deferred-blocker', 'type-bug', 'library']
title = 'Update html.entities.html5 dictionary and parseentities.py'
updated_at = <Date 2019-11-09.02:00:34.386>
user = 'https://github.com/ezio-melotti'

    bugs.python.org fields:

    activity = <Date 2019-11-09.02:00:34.386>
actor = 'terry.reedy'
assignee = 'ezio.melotti'
closed = True
closed_date = <Date 2014-03-16.22:01:33.346>
closer = 'eric.araujo'
components = ['Library (Lib)']
creation = <Date 2012-10-16.09:41:51.441>
creator = 'ezio.melotti'
dependencies = []
files = ['27593', '27670', '27676']
hgrepos = []
issue_num = 16245
keywords = ['patch']
message_count = 16.0
messages = ['173021', '173345', '173589', '173601', '173618', '173619', '173629', '173631', '182506', '193983', '194033', '194527', '213714', '213762', '213763', '356281']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'terry.reedy', 'larry', 'ezio.melotti', 'eric.araujo', 'python-dev', 'Ramchandra Apte', 'kushal.das', 'iuliia.proskurnia']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16245'
versions = ['Python 3.4']

    Activity

    1. ezio-melotti commented on Oct 16, 2012

      @ezio-melotti
      ezio-melotti
      on Oct 16, 2012
      MemberAuthor

      A JSON file containing all the HTML5 entities is now available at http://dev.w3.org/html5/spec/entities.json.
      I tested from the interpreter to see if it matches the values in html.entities.html5 and there are a dozen of entities that need to be updated:

      >>> s = json.load(open('entities.json'))
>>> from html.entities import html5
>>> for (k1,i1),(k2,i2) in zip(sorted(s.items()), sorted(html5.items())):
...   if i1['characters'] != i2: (k1, k2, i1['characters'], i2, i1['codepoints'], list(map(ord, i2)))
... 
('&DotDot;', 'DotDot;', '⃜', '◌⃜', [8412], [9676, 8412])
('&DownBreve;', 'DownBreve;', '̑', '◌̑', [785], [9676, 785])
('&LeftAngleBracket;', 'LeftAngleBracket;', '⟨', '〈', [10216], [9001])
('&NewLine;', 'NewLine;', '\n', '␊', [10], [9226])
('&RightAngleBracket;', 'RightAngleBracket;', '⟩', '〉', [10217], [9002])
('&Tab;', 'Tab;', '\t', '␉', [9], [9225])
('&TripleDot;', 'TripleDot;', '⃛', '◌⃛', [8411], [9676, 8411])
('&lang;', 'lang;', '⟨', '〈', [10216], [9001])
('&langle;', 'langle;', '⟨', '〈', [10216], [9001])
('&rang;', 'rang;', '⟩', '〉', [10217], [9002])
('&rangle;', 'rangle;', '⟩', '〉', [10217], [9002])
('&tdot;', 'tdot;', '⃛', '◌⃛', [8411], [9676, 8411])

      The Tools/scripts/parseentities.py script should also be updated (or possibly a new, separate script should be added), so it can be used to generate the html5 dict. I'm setting this as release blocker so that the update gets done before the release (other values might change in the meanwhile).

    2. ezio-melotti
      added
      release-blocker
      on Oct 16, 2012
    3. ezio-melotti
      self-assigned this
      on Oct 16, 2012
    4. ezio-melotti
      added
      type-bugAn unexpected behavior, bug, or error
      on Oct 16, 2012

    5. merwok commented on Oct 19, 2012

      @merwok
      merwok
      on Oct 19, 2012
      Member

      I say replace the code. HTML 4.01 won’t be updated.

    6. ezio-melotti commented on Oct 23, 2012

      @ezio-melotti
      ezio-melotti
      on Oct 23, 2012
      MemberAuthor

      I think it's ok to have a separate file rather than patching the existing one (see attached patch). If the old script is not used anymore it could be removed, otherwise we could just leave it there.

    7. iuliiaproskurnia commented on Oct 23, 2012

      @iuliiaproskurnia
      iuliiaproskurniamannequin
      on Oct 23, 2012
      Mannequin

      Version with --patch to modify Lib/html/entities.py automatically

    8. python-dev commented on Oct 23, 2012

      @python-dev
      python-devmannequin
      on Oct 23, 2012
      Mannequin

      New changeset dd8b969d7459 by Ezio Melotti in branch 'default':
      bpo-16245: add a script to generate the html.entities.html5 dict.
      http://hg.python.org/cpython/rev/dd8b969d7459

    9. python-dev commented on Oct 23, 2012

      @python-dev
      python-devmannequin
      on Oct 23, 2012
      Mannequin

      New changeset 1eb1c6942ac8 by Ezio Melotti in branch '3.3':
      bpo-16245: Fix the value of a few entities in html.entities.html5.
      http://hg.python.org/cpython/rev/1eb1c6942ac8

      New changeset 70fab10cd542 by Ezio Melotti in branch 'default':
      bpo-16245: merge with 3.3.
      http://hg.python.org/cpython/rev/70fab10cd542

    10. python-dev commented on Oct 23, 2012

      @python-dev
      python-devmannequin
      on Oct 23, 2012
      Mannequin

      New changeset fb80df16c4ff by Ezio Melotti in branch 'default':
      Add Misc/NEWS entry for dd8b969d7459/bpo-16245.
      http://hg.python.org/cpython/rev/fb80df16c4ff

    11. ezio-melotti commented on Oct 23, 2012

      @ezio-melotti
      ezio-melotti
      on Oct 23, 2012
      MemberAuthor

      I now committed an improved version of the script (thanks Iuliia!) and updated the html.entities.html5 dictionary accordingly.

      I'm leaving this open because we will have to check if the dictionary is still updated before the release of Python 3.4.

    12. ezio-melotti
      added
      stdlibStandard Library Python modules in the Lib/ directory
      on Oct 23, 2012

    13. RamchandraApte commented on Feb 20, 2013

      @RamchandraApte
      RamchandraAptemannequin
      on Feb 20, 2013
      Mannequin

      Shouldn't this be deferred blocker?

    14. larryhastings commented on Jul 31, 2013

      @larryhastings
      larryhastings
      on Jul 31, 2013
      Contributor

      This is still marked as a release blocker. I guess this is a "tickler" for Ezio to go check and see if there's a new entities file.

      Ezio: can you get this issue closed or downgraded in the next two days?

    15. larryhastings commented on Aug 1, 2013

      @larryhastings
      larryhastings
      on Aug 1, 2013
      Contributor

      I'm downgrading this to "deferred blocker". We'll make sure it happens before Python 3.4.0, but there's no need to hold up Python 3.4a1 for this.

    16. larryhastings
      added
      deferred-blocker
      and removed
      release-blocker
      on Aug 1, 2013

    17. ezio-melotti commented on Aug 6, 2013

      @ezio-melotti
      ezio-melotti
      on Aug 6, 2013
      MemberAuthor

      I run Tools/scripts/parse_html5_entities.py and it says that "The current dictionary is updated.". We should check this again, and eventually close the issue, when 3.4 is released.

    18. larryhastings commented on Mar 16, 2014

      @larryhastings
      larryhastings
      on Mar 16, 2014
      Contributor

      Was this done? I'm tagging 3.4.0 final soon.

    19. merwok commented on Mar 16, 2014

      @merwok
      merwok
      on Mar 16, 2014
      Member

      I just ran the script:

      $ Tools/scripts/parse_html5_entities.py 
The current dictionary is updated.

      This is done :‑)

    20. merwok
      closed this as completedon Mar 16, 2014

    21. merwok commented on Mar 16, 2014

      @merwok
      merwok
      on Mar 16, 2014
      Member

      BTW this message does not mean that the dictionary was just updated, but that is was already up to date.

    22. terryjreedy commented on Nov 9, 2019

      @terryjreedy
      terryjreedy
      on Nov 9, 2019
      Member

      According to git blame, the html5 dict in https://github.com/python/cpython/blob/master/Lib/html/entities.py has changed in 7 years. On the other hand, the standard on which it is based, https://html.spec.whatwg.org/multipage/named-characters.html, was last revised yesterday, and I presume several other times since. On the third hand, I just ran the update script and there was no change to entities.py, so maybe is has been run with every release.

      Should a comment be added to the file listing the unicode source and the update script?

    23. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    deferred-blockerstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions