Skip to content

Change sys.getfilesystemencoding() on Windows to UTF-8 #71968

Closed
Closed
Change sys.getfilesystemencoding() on Windows to UTF-8#71968
Assignees
zooba
Labels
3.7 (EOL)end of lifeOS-windowsrelease-blockertype-bugAn unexpected behavior, bug, or error

Description

@zooba
zooba
opened on Aug 17, 2016
BPO 27781
Nosy @brettcannon, @pfmoore, @mdickinson, @ncoghlan, @vstinner, @tjguk, @jkloth, @ned-deily, @zware, @zooba, @yan12125, @AraHaan
Files
  • fsencoding.diff
  • fsencoding.diff
  • test_cmd_line_unicode.py: Possible new test case for command line Unicode handling
  • 27781_1.patch
  • osx_failed_compile.txt
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
closed_at = <Date 2016-11-07.03:36:05.023>
created_at = <Date 2016-08-17.03:49:40.936>
labels = ['type-bug', '3.7', 'OS-windows', 'release-blocker']
title = 'Change sys.getfilesystemencoding() on Windows to UTF-8'
updated_at = <Date 2016-11-07.03:36:05.021>
user = 'https://github.com/zooba'

    bugs.python.org fields:

    activity = <Date 2016-11-07.03:36:05.021>
actor = 'python-dev'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-11-07.03:36:05.023>
closer = 'python-dev'
components = ['Windows']
creation = <Date 2016-08-17.03:49:40.936>
creator = 'steve.dower'
dependencies = []
files = ['44130', '44131', '44369', '44414', '44491']
hgrepos = []
issue_num = 27781
keywords = ['patch']
message_count = 24.0
messages = ['272899', '272900', '272916', '272917', '272935', '272949', '272950', '272961', '272962', '272963', '274392', '274691', '274887', '274910', '275063', '275075', '275078', '275095', '275289', '275290', '275326', '275331', '279847', '280186']
nosy_count = 13.0
nosy_names = ['brett.cannon', 'paul.moore', 'mark.dickinson', 'ncoghlan', 'vstinner', 'tim.golden', 'jkloth', 'ned.deily', 'python-dev', 'zach.ware', 'steve.dower', 'yan12125', 'Decorater']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27781'
versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. zooba commented on Sep 7, 2016

      @zooba
      zooba
      on Sep 7, 2016
      MemberAuthor

      PEP-529 has been accepted, so this really needs a review now. But since it's experimental and all the tests pass, I'll be committing it shortly anyway and will be tidying up issues during beta.

    2. zooba commented on Sep 7, 2016

      @zooba
      zooba
      on Sep 7, 2016
      MemberAuthor

      One minor change - I removed the unused definition of Py_FileSystemDefaultDecodeErrors.

    3. zooba commented on Sep 8, 2016

      @zooba
      zooba
      on Sep 8, 2016
      MemberAuthor

      Thanks for that review, Eryk, but I'm going to defer those to other issues (specifically bpo-27998 for scandir and we should file a new issue for the symlink concerns).

      I've got some more doc updates to do though, and then I'll check in if there are no other concerns.

    4. python-dev commented on Sep 8, 2016

      @python-dev
      python-devmannequin
      on Sep 8, 2016
      Mannequin

      New changeset e20c7d8a8187 by Steve Dower in branch 'default':
      Issue bpo-27781: Change file system encoding on Windows to UTF-8 (PEP-529)
      https://hg.python.org/cpython/rev/e20c7d8a8187

    5. zooba commented on Sep 8, 2016

      @zooba
      zooba
      on Sep 8, 2016
      MemberAuthor

      This is pushed now - let the bug fixing begin :)

    6. python-dev commented on Sep 8, 2016

      @python-dev
      python-devmannequin
      on Sep 8, 2016
      Mannequin

      New changeset faca0730270b by Steve Dower in branch 'default':
      Fixes tests broken by issue bpo-27781.
      https://hg.python.org/cpython/rev/faca0730270b

    7. mdickinson commented on Sep 9, 2016

      @mdickinson
      mdickinson
      on Sep 9, 2016
      Member

      It looks as though this change might have broken the compile on OS X. On my OS X 10.9 machine, building from a clean Git checkout of the master branch fails; the tail of the failed build looks like this:

      ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
Fatal Python error: Py_Initialize: unable to load the file system codec
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 962, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 668, in exec_module
  File "<frozen importlib._bootstrap_external>", line 782, in get_code
  File "<frozen importlib._bootstrap_external>", line 842, in _cache_bytecode
  File "<frozen importlib._bootstrap_external>", line 867, in set_data
  File "<frozen importlib._bootstrap_external>", line 117, in _write_atomic
ValueError: negative file descriptor
/bin/sh: line 1: 35829 Abort trap: 6           ./python.exe -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1

      Full build output attached.

    8. mdickinson commented on Sep 9, 2016

      @mdickinson
      mdickinson
      on Sep 9, 2016
      Member

      It looks as though this change in posixmodule.c is the cause:

       #ifdef MS_WINDOWS
-        if (path->wide)
-            fd = _wopen(path->wide, flags, mode);
-        else
+        fd = _wopen(path->wide, flags, mode);
 #endif
 #ifdef HAVE_OPENAT
         if (dir_fd != DEFAULT_DIR_FD)
             fd = openat(dir_fd, path->narrow, flags, mode);
         else
-#endif
             fd = open(path->narrow, flags, mode);
+#endif

      The move of the final #endif means that fd is not defined on OS X. If I move the #endif back again, the compile succeeds.

    9. python-dev commented on Sep 9, 2016

      @python-dev
      python-devmannequin
      on Sep 9, 2016
      Mannequin

      New changeset 801634d3c105 by Steve Dower in branch 'default':
      Issue bpo-27781: Fixes uninitialized fd when !MS_WINDOWS and !HAVE_OPENAT
      https://hg.python.org/cpython/rev/801634d3c105

    10. mdickinson commented on Sep 9, 2016

      @mdickinson
      mdickinson
      on Sep 9, 2016
      Member

      That seems to have done the trick. Thanks!

    11. zooba commented on Nov 1, 2016

      @zooba
      zooba
      on Nov 1, 2016
      MemberAuthor

      Before 3.6.0 beta 4 I need to make this change permanent. From memory, it's just an exception message that needs changing (and PEP-529 becomes final), but I'll review the changeset to be sure.

    12. zooba
      added
      3.7 (EOL)end of life
      release-blocker
      on Nov 1, 2016

    13. python-dev commented on Nov 7, 2016

      @python-dev
      python-devmannequin
      on Nov 7, 2016
      Mannequin

      New changeset b26c8104e54f by Steve Dower in branch '3.6':
      Closes bpo-27781: Removes special cases for the experimental aspect of PEP-529
      https://hg.python.org/cpython/rev/b26c8104e54f

      New changeset b8233c779ff7 by Steve Dower in branch 'default':
      Closes bpo-27781: Removes special cases for the experimental aspect of PEP-529
      https://hg.python.org/cpython/rev/b8233c779ff7

    14. python-dev
      closed this as completedon Nov 7, 2016
    15. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.7 (EOL)end of lifeOS-windowsrelease-blockertype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions