Create PyUnicode_FSDecoder() function #53751

Activity

1. vstinner commented on Aug 8, 2010

   

   vstinner

   on Aug 8, 2010

   MemberAuthor

   More actions

   For my work on bpo-9425 (Rewrite import machinery to work with unicode paths), I need a PyArg_Parse converter converting bytes and str to str. PyUnicode_FSConverter() is the opposite because it encodes str to bytes.

   To handle (input) filenames in a function, we have 3 choices:

   1/ use bytes: that's the current choice for most Python functions. It gives full unicode support for POSIX OSes (FS using a bytes API), but it is not enough for Windows (Windows uses mbcs encoding which is a very small subset of Unicode)
   2/ use str with the PEP-383 (surrogateescape): it begins to be used in Python 3.1, and more seriously in Python 3.2. It offers full unicode support on all OSes (POSIX and Windows)
   3/ use the native type for each OS (bytes on POSIX, str on Windows): I dislike this solution because it implies code duplication

   PyUnicode_FSConverter() is the converter for solution (1). PyUnicode_FSDecoder() will be the converter for the solution (2).
2. vstinner

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-unicode

   on Aug 8, 2010
3. vstinner commented on Aug 13, 2010

   

   vstinner

   on Aug 13, 2010

   MemberAuthor

   More actions

   Lib/os.py may also be patched to add a Python implementation. Eg.

   def fsdecode(value):
       if isinstance(value, str):
           return value
       elif isinstance(value, bytes):
           encoding = sys.getfilesystemencoding()
           if encoding == 'mbcs':
               return value.decode(encoding)
           else:
               return value.decode(encoding, 'surrogateescape')
       else:
           raise TypeError("expect bytes or str, not %s" % type(value).__name__)

   --

   Note: Solution (1) (use bytes API) is not deprecated by this issue. PyUnicode_FSConverter is still useful if the underlying library has a bytes API (eg. OpenSSL only supports char*).

   Solution (2) is preferred if we have access to a character API, eg. Windows wide character API.
4. vstinner commented on Aug 14, 2010

   

   vstinner

   on Aug 14, 2010

   MemberAuthor

   More actions

   Commited to 3.2 as r83990.
5. vstinner

   closed this as completedon Aug 14, 2010
6. ezio-melotti

   transferred this issue fromon Apr 10, 2022