Skip to content

asyncio subprocess accepts string as parameter which lead to UnicodeEncodeError #79195

Closed
Closed
asyncio subprocess accepts string as parameter which lead to UnicodeEncodeError#79195
Labels
3.7 (EOL)end of life3.8 (EOL)end of lifetopic-asyncio

Description

@Natim
Natim
mannequin
opened on Oct 18, 2018
BPO 35014
Nosy @vstinner, @asvetlov, @1st1, @Natim
Files
  • demo.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-10-18.12:43:29.663>
created_at = <Date 2018-10-18.08:50:17.771>
labels = ['invalid', '3.7', '3.8', 'expert-asyncio']
title = 'asyncio subprocess accepts string as parameter which lead to UnicodeEncodeError'
updated_at = <Date 2018-10-18.12:44:24.184>
user = 'https://github.com/Natim'

    bugs.python.org fields:

    activity = <Date 2018-10-18.12:44:24.184>
actor = 'natim'
assignee = 'none'
closed = True
closed_date = <Date 2018-10-18.12:43:29.663>
closer = 'natim'
components = ['asyncio']
creation = <Date 2018-10-18.08:50:17.771>
creator = 'natim'
dependencies = []
files = ['47879']
hgrepos = []
issue_num = 35014
keywords = []
message_count = 15.0
messages = ['327945', '327953', '327954', '327955', '327962', '327964', '327965', '327966', '327967', '327970', '327974', '327975', '327976', '327977', '327978']
nosy_count = 4.0
nosy_names = ['vstinner', 'asvetlov', 'yselivanov', 'natim']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35014'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

    Activity

    1. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      Asyncio.create_subprocess_exec accepts a list of str as parameter which lead to UnicodeEncodeError I think it should accept only bytes shouldn't it?

    2. Natim
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      topic-asyncio
      on Oct 18, 2018

    3. asvetlov commented on Oct 18, 2018

      @asvetlov
      asvetlov
      on Oct 18, 2018
      Contributor

      List of strings works on both my local Linux box and CPython test suite.

      Please provide more info about the error. Stacktrace can help

    4. vstinner commented on Oct 18, 2018

      @vstinner
      vstinner
      on Oct 18, 2018
      Member

      Hi Remy,

      Asyncio.create_subprocess_exec accepts a list of str as parameter which lead to UnicodeEncodeError I think it should accept only bytes shouldn't it?

      Can you elaborate? On which OS? What is your error message? Can you paste a traceback?

    5. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      List of strings works on both my local Linux box and CPython test suite.

      Indeed that's why I posted this bug report, in my opinion it should work only with bytes string.

      Can you elaborate? On which OS? What is your error message? Can you paste a traceback?

      If you try to send a UTF-8 string on a linux box for instance, you might get a UnicodeEncodeError.

      Let me try to provide you with a script to reproduce this error.

    6. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      I though this would be sufficient to actually reproduce the issue.
      However it seems that if the system encoding is UTF-8 it does work properly.

      Here is the traceback I had:

      UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 69: ordinal not in range(128)
  File "worker.py", line 393, in <module>
    return_code = loop.run_until_complete(main(loop))
  File "asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "worker.py", line 346, in main
    '-f mp4', '-o', '{}/{}.mp4'.format(download_tempdir, video_id))
  File "worker.py", line 268, in run_command
    proc = await create
  File "asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "asyncio/base_events.py", line 1191, in subprocess_exec
    bufsize, **kwargs)
  File "asyncio/unix_events.py", line 191, in _make_subprocess_transport
    **kwargs)
  File "asyncio/base_subprocess.py", line 39, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "asyncio/unix_events.py", line 697, in _start
    universal_newlines=False, bufsize=bufsize, **kwargs)
  File "python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "python3.6/subprocess.py", line 1267, in _execute_child
    restore_signals, start_new_session, preexec_fn)

    7. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      I am adding the following info:

      If I run the following on the Docker image where I got the error I get:

      import sys
import locale

print(sys.getdefaultencoding())
print(locale.getpreferredencoding())

      utf-8
      ANSI_X3.4-1968

      While if I run it on my machine I get:

      utf-8
      UTF-8

      I don't know how to force the usage of the later locally to reproduce.

      Settings LC_ALL=C and LANG=C didn't do the trick

    8. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      Here we go:

      $ python3.7 demo.py 
utf-8
UTF-8
Traceback (most recent call last):
  File "demo.py", line 21, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "demo.py", line 14, in main
    sys.stdout.write(out.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)

    9. asvetlov commented on Oct 18, 2018

      @asvetlov
      asvetlov
      on Oct 18, 2018
      Contributor

      I think you'll get the same error on subprocess.run() call if your current locale is not UTF-8.

      I don't recall the details but the Intenet has a lot info about setting locale per user and system-wide.

    10. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      I believe Python 3.7 brings explicit unicode encoding/decoding.

      If depending on the environment the create_subprocess_exec method can fail, I believe we should not try to encode the command lines attribute but rather enforce it to be bytes.

    11. vstinner commented on Oct 18, 2018

      @vstinner
      vstinner
      on Oct 18, 2018
      Member

      I added the UTF-8 Mode for you, for the Docker use case: python3.7 -X utf8. Using that, Python ignores your locale and speaks UTF-8.

      What is your locale? Try the "locale" command.

    12. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      Here are the locale set:

      LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

    13. vstinner commented on Oct 18, 2018

      @vstinner
      vstinner
      on Oct 18, 2018
      Member

      LC_CTYPE="POSIX"

      I modified Python 3.7.1 to enable the UTF-8 Mode when the LC_CTYPE is "POSIX". In Python 3.7.0, the UTF-8 Mode is only enabled if the LC_CTYPE is "C".

    14. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      Unicode is complicated, the answer is somewhere here: https://unicodebook.readthedocs.io/

      Sorry for the bothering, I thought it was a bug but apparently it's a feature. Thank you for your help, thank you for making Python better.

    15. Natim
      closed this as completedon Oct 18, 2018
    16. Natim
      added
      invalid
      on Oct 18, 2018

    17. vstinner commented on Oct 18, 2018

      @vstinner
      vstinner
      on Oct 18, 2018
      Member

      This issue is not an asyncio bug: the bug occurs in subprocess.

      The bug is not a subprocess bug: subprocess works as expected, it encodes Unicode with sys.getfilesystemencoding() (see os.fsencode()).

      The bug is that you use non-ASCII strings whereas your filesystem encoding is ASCII.

      You have a different options to fix *your* issue:

      • Use a different locale which uses a UTF-8 locale
      • Enable the Python 3.7 UTF-8 mode
      • Wait for Python 3.7.1 (which enables automatically the UTF-8 Mode for LC_CTYPE="POSIX")

      Note: You might want to read my ebook http://unicodebook.readthedocs.io/ which explains how to deal with Unicode.

    18. Natim commented on Oct 18, 2018

      @Natim
      Natimmannequin
      on Oct 18, 2018
      MannequinAuthor

      I modified Python 3.7.1 to enable the UTF-8 Mode when the LC_CTYPE is "POSIX". In Python 3.7.0, the UTF-8 Mode is only enabled if the LC_CTYPE is "C"

      Ok works for me thanks :)

    19. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifetopic-asyncio

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions