Skip to content

2to3 crashes when converting doctest using reduce() #56820

Closed
Closed
2to3 crashes when converting doctest using reduce()#56820
Labels
3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifetopic-2to3type-bugAn unexpected behavior, bug, or error

Description

@VPeric
VPeric
mannequin
opened on Jul 22, 2011
BPO 12611
Nosy @benjaminp, @merwok, @asmeurer, @akheron, @aldwinaldwin, @iritkatriel
PRs
  • bpo-12611: 2to3 crashes when converting doctest using reduce() #15249
    		•
    Superseder
  • bpo-45544: Close 2to3 issues and list them here
    		•
    Files
  • issue12611_test.patch: Test case
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-10-20.22:53:31.045>
created_at = <Date 2011-07-22.15:53:01.748>
labels = ['3.8', 'type-bug', 'expert-2to3', '3.9', '3.10']
title = '2to3 crashes when converting doctest using reduce()'
updated_at = <Date 2021-10-20.22:53:31.044>
user = 'https://bugs.python.org/VPeric'

    bugs.python.org fields:

    activity = <Date 2021-10-20.22:53:31.044>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.22:53:31.045>
closer = 'iritkatriel'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2011-07-22.15:53:01.748>
creator = 'VPeric'
dependencies = []
files = ['22851']
hgrepos = []
issue_num = 12611
keywords = ['patch']
message_count = 8.0
messages = ['140892', '141310', '141320', '141421', '141582', '141727', '142072', '380469']
nosy_count = 8.0
nosy_names = ['benjamin.peterson', 'eric.araujo', 'VPeric', 'Aaron.Meurer', 'catalin.iacob', 'petri.lehtinen', 'aldwinaldwin', 'iritkatriel']
pr_nums = ['15249']
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45544'
type = 'behavior'
url = 'https://bugs.python.org/issue12611'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    Activity

    1. VPeric commented on Jul 22, 2011

      @VPeric
      VPericmannequin
      on Jul 22, 2011
      MannequinAuthor

      2to3 crashes when run on a doctest which uses reduce(). This happens with both 2.7 and 3.2's 2to3. I have the following code in a compatibility file, but even using that it fails with the same error:

      try:
      from functools import reduce
      except ImportError:
      reduce = reduce

      This is the traceback produced:

      Traceback (most recent call last):
  File "/usr/local/bin/2to3", line 6, in <module>
    sys.exit(main("lib2to3.fixes"))
  File "/usr/local/lib/python2.6/lib2to3/main.py", line 173, in main
    options.processes)
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 620, in refactor
    items, write, doctests_only)
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 275, in refactor
    self.refactor_file(dir_or_file, write, doctests_only)
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 661, in refactor_file
    *args, **kwargs)
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 321, in refactor_file
    output = self.refactor_docstring(input, filename)
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 500, in refactor_docstring
    indent, filename))
  File "/usr/local/lib/python2.6/lib2to3/refactor.py", line 530, in refactor_doctest
    assert clipped == [u"\n"] * (lineno-1), clipped
AssertionError: [u'from functools import reduce\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n', u'\n']
    2. VPeric
      added
      topic-2to3
      on Jul 22, 2011

    3. akheron commented on Jul 28, 2011

      @akheron
      akheron
      on Jul 28, 2011
      Member

      I cannot reproduce the crash on neither 2.7 nor 3.2. Can you provide more details; attach the exact python source file that crashes 2to3 and give the complete crashing 2to3 command that you're running.

    4. asmeurer commented on Jul 28, 2011

      @asmeurer
      asmeurermannequin
      on Jul 28, 2011
      Mannequin

      Vladimir will need to confirm how to reproduce this exactly, but here is corresponding SymPy issue: http://code.google.com/p/sympy/issues/detail?id=2605.

      The problem is with the sympy/ntheory/factor_.py file at https://github.com/sympy/sympy/blob/sympy-0.7.1.rc1/sympy/ntheory/factor_.py#L453 (linking to the file from our release candidate, as a workaround is likely to be pushed to master soon).

      Vladimir, can you confirm that this particular version of the file reproduces the problem?

    5. akheron commented on Jul 30, 2011

      @akheron
      akheron
      on Jul 30, 2011
      Member

      Still cannot reproduce with the linked file. Setting as pending until we hear the details from Vladimir.

    6. VPeric commented on Aug 2, 2011

      @VPeric
      VPericmannequin
      on Aug 2, 2011
      MannequinAuthor

      Confirmed with the file Aaron linked to. I'm using "2to3-3.2 -w -n -d sympy/ntheory/factor_.py". This is what Python says about itself:

      Python 3.2 (r32:88445, Jun 8 2011, 16:34:06)
      [GCC 4.5.1 20101208 [gcc-4_5-branch revision 167585]] on linux2

    7. akheron commented on Aug 6, 2011

      @akheron
      akheron
      on Aug 6, 2011
      Member

      Attached a patch that adds a test case reproducing the issue. Hopely someone with more experience with lib2to3 can do the actual fixing...

    8. akheron
      added
      type-bugAn unexpected behavior, bug, or error
      on Aug 6, 2011

    9. cataliniacob commented on Aug 14, 2011

      @cataliniacob
      cataliniacobmannequin
      on Aug 14, 2011
      Mannequin

      I looked at this and understood why it's happening. I don't know exactly how to fix it though, so here's what I found out.

      When a doctest appears in a docstring at line n in a file, RefactorTool.parse_block will return a tree corresponding to n - 1 newline characters followed by the code in the doctest. That tree is refactored by RefactoringTool.refactor_tree which usually returns n - 1 newline characters followed by the refactored doctest. However, for the reduce fixer, the tree returned by refactor_tree starts with from functools import reduce followed by n - 1 newline characters and then the doctest reduce line. The failing assert happens when stripping those newlines because they are expected to be at the beginning of the output while in reality they're after the import line.

      So the problem is a mismatch between the expectations of the doctest machinery (refactoring code that starts with some newlines results in code that starts with the same number of newlines) and the reduce fixer which adds an import, imports are added at the beginning of the file, therefore something appears before the newlines. Other fixers could exhibit the same problem.

    10. iritkatriel commented on Nov 6, 2020

      @iritkatriel
      iritkatriel
      on Nov 6, 2020
      Member

      The test in issue12611_test.patch is still failing.

    11. iritkatriel
      added
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      on Nov 6, 2020
    12. iritkatriel
      closed this as completedon Oct 20, 2021
    13. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifetopic-2to3type-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions