Skip to content

pty.spawn hangs on FreeBSD 9.3, 10.x, 12.1 #70416

Closed
Closed
pty.spawn hangs on FreeBSD 9.3, 10.x, 12.1#70416
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@christorek
christorek
mannequin
opened on Jan 28, 2016
BPO 26228
Nosy @ambv, @vadmium, @RadicalZephyr, @diekmann, @miss-islington, @JarryShaw, @8vasu
PRs
  • bpo-29070: Integration tests for pty module with patch from bpo-26228 #2932
  • bpo-26228: pty.spawn hangs on FreeBSD, OS X, and Solaris #4167
  • bpo-26228: Fix pty EOF handling #12049
  • bpo-41818: Updated tests for the standard pty library #22962
  • [3.10] bpo-26228: Fix pty EOF handling (GH-12049) #27732
  • bpo-26228: [doc] Adapt PTY documentation updates from GH-4167 #27754
  • [3.10] bpo-26228: [doc] Adapt PTY documentation updates from GH-4167 (GH-27754) #27758
    		•
    Files
  • pty.patch: minor restructuring of pty.py, plus a change to its test modules
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-08-12.12:37:09.625>
created_at = <Date 2016-01-28.04:04:21.544>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'pty.spawn hangs on FreeBSD 9.3, 10.x, 12.1'
updated_at = <Date 2021-08-13.11:21:14.284>
user = 'https://bugs.python.org/christorek'

    bugs.python.org fields:

    activity = <Date 2021-08-13.11:21:14.284>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-12.12:37:09.625>
closer = 'lukasz.langa'
components = ['Library (Lib)']
creation = <Date 2016-01-28.04:04:21.544>
creator = 'chris.torek'
dependencies = []
files = ['41738']
hgrepos = []
issue_num = 26228
keywords = ['patch']
message_count = 17.0
messages = ['259088', '259093', '284493', '284552', '284590', '292382', '306934', '336601', '336637', '336689', '336708', '375084', '399420', '399445', '399446', '399521', '399526']
nosy_count = 9.0
nosy_names = ['lukasz.langa', 'martin.panter', 'jbeck', 'RadicalZephyr', 'chris.torek', 'Cornelius Diekmann', 'miss-islington', 'jarryshaw', 'soumendra']
pr_nums = ['2932', '4167', '12049', '22962', '27732', '27754', '27758']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26228'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. christorek commented on Jan 28, 2016

      @christorek
      christorekmannequin
      on Jan 28, 2016
      MannequinAuthor

      The pty.spawn() code assumes that when the process on the slave side of the pty quits, the master side starts raising OSError when read-from or written-to.

      That used to be true in FBSD, but then someone fixed (?) it, and now the master side simply returns EOF when read-from. Furthermore, writes to the master simply disappear into the aether (this may be an OS bug, but even if the writes raised OSError, you would still have to type something in on stdin to get it copied over to get the error raised to get out of the loop).

      The fix here makes an assumption that is true when using the built-in read calls: EOF on the master indicates that the slave is no longer reachable in any way and the whole thing should finish up immediately. It might perhaps need a bit of documentation should someone want to substitute in their own read function (see enhancement request in bpo-22865).

      I also fixed (sort of) bpo-17824, but only barely minimally, and put in a comment that it should really use the same mechanism as subprocess (but I think that should be a separate patch).

    2. vadmium commented on Jan 28, 2016

      @vadmium
      vadmium
      on Jan 28, 2016
      Member

      I agree with all the changes you made. I made one review comment.

      It would be nice to add a test case to expose the problem. Correct me if I am wrong, but it doesn’t look like pty.spawn() is tested at all.

      FWIW on Linux, reading from the master end seems to raise EIO if the slave has been closed. And writing to the master when the slave is closed seems to fill up a buffer and eventually blocks.

      Ideally I think the best solution for handing exec() failure (bpo-17824) would be to eliminate fork-exec with posix_spawn(); see bpo-20104. But as you say, that’s a separate problem.

    3. vadmium
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Jan 28, 2016

    4. diekmann commented on Jan 2, 2017

      @diekmann
      diekmannmannequin
      on Jan 2, 2017
      Mannequin

      I just tested pty.spawn() on OS X 10.6.8 and FreeBSD 11 and it also hangs. It works on Linux.

      Your patch solves the problem. I proposed a test suite for pty.spawn() in bpo-29070. The test suite currently only exposes the problem, as suggested by Martin.

    5. vstinner commented on Jan 3, 2017

      @vstinner
      vstinner
      on Jan 3, 2017
      Member

      That used to be true in FBSD, but then someone fixed (?) it, and now the master side simply returns EOF when read-from.

      Is it a behaviour change in a new version of Python? Or a change in FreeBSD? I don't understand.

    6. vadmium commented on Jan 3, 2017

      @vadmium
      vadmium
      on Jan 3, 2017
      Member

      Behaviour change in Free BSD as I understand. Nothing changed in Python, but perhaps older versions of Free BSD behaved like Linux and raised EIO (or another errno; it is not clear).

    7. jbeck commented on Apr 26, 2017

      @jbeck
      jbeckmannequin
      on Apr 26, 2017
      Mannequin

      Solaris has this problem also, and Chris' patch fixes it for us.

    8. vadmium commented on Nov 25, 2017

      @vadmium
      vadmium
      on Nov 25, 2017
      Member

      If it helps, here is a basic test case I wrote for “pty.spawn”. I hope that it exposes the problem on Free BSD, but I have only tested it on Linux.

      parent = r'''\
import pty, sys
pty.spawn((sys.executable, "-c", sys.argv[1]))
'''
child = r'''\
import sys
# Read input first of all to minimize output buffering
input = sys.stdin.readline()
print("isatty: {}, {}, {}".format(
    sys.stdin.isatty(), sys.stdout.isatty(), sys.stderr.isatty()))
print("input: " + repr(input))
sys.stdout.write("stdout data\n")
sys.stderr.write("stderr data\n")
'''
args = (sys.executable, "-c", parent, child)
parent = subprocess.Popen(args,
    stdin=subprocess.PIPE, stdout=subprocess.PIPE)
try:
    parent.stdin.write(b"stdin data\n")
    parent.stdin.flush()
    # Leave input open. When the child closes the slave terminal on
    # Free BSD, “spawn” used to keep waiting for input (BPO 26228).
    output = parent.stdout.read()
finally:
    parent.stdout.close()
    parent.stdin.close()
    parent.wait()
self.assertEqual(0, parent.returncode, repr(output))
self.assertIn(b"isatty: True, True, True", output)
self.assertIn(br"input: 'stdin data\n'", output)
self.assertIn(b"stdout data", output)
self.assertIn(b"stderr data", output)

    9. JarryShaw commented on Feb 26, 2019

      @JarryShaw
      JarryShawmannequin
      on Feb 26, 2019
      Mannequin

      Chris' patch works on macOS 10.14 (Mojave); but after it terminates, tty attributes are not restored (new-lines are missing). btw, I've implemented a workaround library with solution through either ps command or psutil package, see ptyng on PyPI.

    10. RadicalZephyr commented on Feb 26, 2019

      @RadicalZephyr
      RadicalZephyrmannequin
      on Feb 26, 2019
      Mannequin

      I took a shot at fixing this in a smaller more targeted patch. I think this should still solve the major issue of pty.spawn hanging on platforms that don't raise an error. In addition, pty.spawn should now _ALWAYS_ return the terminal to the previous settings.

    11. vstinner commented on Feb 26, 2019

      @vstinner
      vstinner
      on Feb 26, 2019
      Member

      Please have a look at this pty.spawn() documentation change:
      #11980

    12. RadicalZephyr commented on Feb 26, 2019

      @RadicalZephyr
      RadicalZephyrmannequin
      on Feb 26, 2019
      Mannequin

      I'm aware of it. I actually wrote that patch as well. :D

    13. 8vasu commented on Aug 9, 2020

      @8vasu
      8vasumannequin
      on Aug 9, 2020
      Mannequin

      Hi! Can anyone please take a look at

      https://bugs.python.org/issue41494 [ https://github.com//pull/21752 ]?

      I think these are related. I wrote a new function called wspawn, which is like spawn+the following differences.

      1. It sets window size at the beginning.
      2. It registers a SIGWINCH handler.
      3. It does not depend on OSError to return. It does a clean return instead.
    14. 8vasu
      changed the title [-]pty.spawn hangs on FreeBSD 9.3, 10.x[/-] [+]pty.spawn hangs on FreeBSD 9.3, 10.x, 12.1[/+] on Aug 21, 2020

    15. ambv commented on Aug 11, 2021

      @ambv
      ambv
      on Aug 11, 2021
      Contributor

      New changeset 81ab8db by Zephyr Shannon in branch 'main':
      bpo-26228: Fix pty EOF handling (GH-12049)
      81ab8db

    16. ambv
      added
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      3.11only security fixes
      on Aug 11, 2021

    17. ambv commented on Aug 12, 2021

      @ambv
      ambv
      on Aug 12, 2021
      Contributor

      New changeset 5d44443 by Miss Islington (bot) in branch '3.10':
      bpo-26228: Fix pty EOF handling (GH-12049) (GH-27732)
      5d44443

    18. ambv commented on Aug 12, 2021

      @ambv
      ambv
      on Aug 12, 2021
      Contributor

      This is now merged. Thanks for all input, everyone! ✨ 🍰 ✨

    19. ambv
      closed this as completedon Aug 12, 2021

    20. ambv commented on Aug 13, 2021

      @ambv
      ambv
      on Aug 13, 2021
      Contributor

      New changeset dd8eb30 by Łukasz Langa in branch 'main':
      bpo-26228: [doc] Adapt PTY documentation updates from #48417 (GH-27754)
      dd8eb30

    21. miss-islington commented on Aug 13, 2021

      @miss-islington
      miss-islington
      on Aug 13, 2021
      Contributor

      New changeset d412848 by Miss Islington (bot) in branch '3.10':
      bpo-26228: [doc] Adapt PTY documentation updates from #48417 (GH-27754)
      d412848

    22. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions