Skip to content

plistlib rejects strings containing control characters #54942

Closed as not planned
Closed as not planned
plistlib rejects strings containing control characters#54942
Assignees
ronaldoussoren
Labels
OS-macstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@MLModel
MLModel
mannequin
opened on Dec 18, 2010
BPO 10733
Nosy @loewis, @ronaldoussoren, @MLModel
Files
  • com.apple.Terminal.plist: output of plutil -convert xml1 ~/Library/Preferences/com.apple.Terminal.plist -o ~/tmp/com.apple.Terminal.plist
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ronaldoussoren'
closed_at = None
created_at = <Date 2010-12-18.19:39:58.777>
labels = ['OS-mac', 'type-bug', 'library']
title = 'plistlib rejects strings containing control characters'
updated_at = <Date 2015-04-23.18:33:14.694>
user = 'https://github.com/MLModel'

    bugs.python.org fields:

    activity = <Date 2015-04-23.18:33:14.694>
actor = 'Behdad.Esfahbod'
assignee = 'ronaldoussoren'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'macOS']
creation = <Date 2010-12-18.19:39:58.777>
creator = 'MLModel'
dependencies = []
files = ['20104']
hgrepos = []
issue_num = 10733
keywords = []
message_count = 7.0
messages = ['124311', '124313', '124325', '124437', '124608', '124625', '241874']
nosy_count = 4.0
nosy_names = ['loewis', 'ronaldoussoren', 'MLModel', 'Behdad.Esfahbod']
pr_nums = []
priority = 'low'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10733'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

    Activity

    1. MLModel commented on Dec 18, 2010

      @MLModel
      MLModelmannequin
      on Dec 18, 2010
      MannequinAuthor

      plistlib rejects control characters found in XML plists that Apple's 'plutil lint' accepts. I have attached my Terminal preferences as an example. (plistlib accepts the contents of the default Terminal preferences file)

    2. MLModel
      assigned
      ronaldoussoren
      on Dec 18, 2010
    3. MLModel
      added
      stdlibStandard Library Python modules in the Lib/ directory
      OS-mac
      type-bugAn unexpected behavior, bug, or error
      on Dec 18, 2010

    4. loewis commented on Dec 18, 2010

      @loewis
      loewismannequin
      on Dec 18, 2010
      Mannequin

      This is tricky. It's clearly ill-formed XML, so I'm not sure that this needs to be bug-compatible with Apple's implementation.

    5. MLModel commented on Dec 18, 2010

      @MLModel
      MLModelmannequin
      on Dec 18, 2010
      MannequinAuthor

      I can see where that does make it tricky. (I also tried reading the plist after opening the file as binary, but no luck.) The problem here, of course, is that the only reason for the existence of this library is to read Apple's plist files, however XML-invalid some may be. (It is only a small number of my very many .plist files that have invalid characters -- I just happened to pick one of them to try to access in order to print a simple summary of its contents.) I guess since the plist is read using xml.parsers.expat, there's not much that can be done, and it wouldn't be worth anyone's time to hack around this for plistlib, especially since nearly all .plist files appear to be conforming. Thanks for the clarification.

    6. bitdancer
      closed this as completedon Dec 20, 2010

    7. ronaldoussoren commented on Dec 21, 2010

      @ronaldoussoren
      ronaldoussoren
      on Dec 21, 2010
      Contributor

      I agree with Martin that this is a tricky one.

      The file is problematic because it is invalid XML[1], however Apple's tools are perfectly happy to proces the file and as Mitchell notes plistlib exists to interoperate with Apple's plist files.

      I'm therefore reopening the issue, but with a low priority. It is unlikely that I'll work on this in the near future though.

      Replacing all control characters by entities before trying to parse the Plist XML would likely be the best way forward. A patch (including testcases) would definitely be appreciated.

      BTW. I've checked that Apple's Cocoa libraries will read the file, this is not just a bug in the xml1 output formatter of plutil.

      Using PyObjC:
>>> from Foundation import NSDictionary
>>> d = NSDictionary.dictionaryWithContentsOfFile_('com.apple.Terminal.plist')

      [1] It is invalid XML because it contains control characters which are invalid according to the XML specification (<http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char\>).

    8. ronaldoussoren
      reopened this on Dec 21, 2010

    9. terryjreedy commented on Dec 24, 2010

      @terryjreedy
      terryjreedy
      on Dec 24, 2010
      Member

      Mitchell: 2.6 is closed to revision except for security issues

    10. MLModel commented on Dec 25, 2010

      @MLModel
      MLModelmannequin
      on Dec 25, 2010
      MannequinAuthor

      Thanks for letting me know (and with a personalized message, yet!). I wasn't paying attention -- i verified that the problem exists in 2.7 and 3.1 and I just dragged 3.1 down to 2.6. Although I've been working furiously in Python for the past six months, I haven't been writing or teaching so I haven't been combing the documentation or testing examples or using obscure or forgotten features, which together are the source of nearly all my bug reports. So I just automatically did what I used to do. I understand the issue and difference; I just didn't realize 2.6 was closed -- I didn't really mean that this should be fixed in anything other than the current or even next release of anything.
      On Dec 24, 2010, at 3:36 PM, Terry J. Reedy wrote:

      Terry J. Reedy <tjreedy@udel.edu> added the comment:

      Mitchell: 2.6 is closed to revision except for security issues

      ----------
      nosy: +terry.reedy
      resolution: wont fix ->
      versions: +Python 3.2 -Python 2.6

      Python tracker <report@bugs.python.org>
      <http://bugs.python.org/issue10733\>

    11. BehdadEsfahbod commented on Apr 23, 2015

      @BehdadEsfahbod
      BehdadEsfahbodmannequin
      on Apr 23, 2015
      Mannequin

      Replacing all control characters by entities before trying to parse the Plist XML would likely be the best way forward.

      That wouldn't work. Control characters are disallowed in XML's character set, so they are invalid even if input as entities.

      Unfortunately this causes a lot of trouble for clients [0], because it means that XML cannot represent the full Unicode repertoire. I'm curious about alternates. Perhaps the expat module can be extended to allow recovering from this if the client chooses to...

      [0] eg. fonttools/fonttools#249

    12. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    13. ronaldoussoren commented on Nov 16, 2022

      @ronaldoussoren
      ronaldoussoren
      on Nov 16, 2022
      Contributor

      I'm closing this issue:

      1. The file that triggered this issue is invalid XML and trying to coax a parser into parsing it is hard for a very limited use-case
      2. On recent(ish) versions of macOS the Terminal plist file is a binary plist, taking away the need for a workaround on our end.
    14. ronaldoussoren
      closed this as not plannedon Nov 16, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    OS-macstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions