Skip to content

Refactor the dis module to provide better building blocks for bytecode analysis #56025

Closed
Closed
Refactor the dis module to provide better building blocks for bytecode analysis#56025
Assignees
ncoghlan
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@eltoder
eltoder
mannequin
opened on Apr 10, 2011
BPO 11816
Nosy @loewis, @rhettinger, @jcea, @ncoghlan, @abalkin, @pitrou, @vstinner, @alex, @meadori, @ericsnowcurrently, @takluyver, @1st1
Files
  • issue11816_get_opinfo_branch_20111204.diff: Remote diff against python.org after merging from default
  • 5ce60675e572.diff
  • dis_api.diff
  • dis_api2.diff: Updated patch after review
  • dis_api3.diff: + Bytecode docs and tests
  • test_peepholer.diff: use bytecode_helper in test_peepholer
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2013-05-07.09:55:08.281>
created_at = <Date 2011-04-10.00:20:08.483>
labels = ['type-feature', 'library']
title = 'Refactor the dis module to provide better building blocks for bytecode analysis'
updated_at = <Date 2016-01-19.07:51:22.304>
user = 'https://github.com/eltoder'

    bugs.python.org fields:

    activity = <Date 2016-01-19.07:51:22.304>
actor = 'python-dev'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2013-05-07.09:55:08.281>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2011-04-10.00:20:08.483>
creator = 'eltoder'
dependencies = []
files = ['23849', '23853', '29051', '29114', '29694', '29695']
hgrepos = ['94']
issue_num = 11816
keywords = ['patch']
message_count = 48.0
messages = ['133437', '133438', '133439', '133441', '133464', '133465', '133466', '133467', '133471', '133479', '133521', '142825', '142831', '143491', '144322', '144400', '144401', '144409', '146694', '148227', '148382', '148383', '148833', '148839', '148840', '148841', '148842', '148859', '163018', '179260', '181970', '181998', '182321', '186148', '188502', '188529', '188536', '188537', '188538', '188539', '188572', '188573', '188574', '188601', '188602', '188604', '188637', '258574']
nosy_count = 16.0
nosy_names = ['loewis', 'rhettinger', 'jcea', 'ncoghlan', 'belopolsky', 'pitrou', 'vstinner', 'ron_adam', 'alex', 'rfk', 'meador.inge', 'neologix', 'python-dev', 'eric.snow', 'takluyver', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11816'
versions = ['Python 3.4']

    Activity

    1. eltoder commented on Apr 10, 2011

      @eltoder
      eltodermannequin
      on Apr 10, 2011
      MannequinAuthor

      As discussed in bpo-11549 a couple of tests need to inspect disassembly of some code. Currently they have to override sys.stdout, run dis and restore stdout back. It would be much nicer if dis module provided functions that return disassembly as a string.

      Provided is a patch that adds file argument to most dis functions, defaulting to sys.stdout. On top of that there are 2 new functions: dis_to_str and disassembly_to_str that return disassembly as a string instead of writing it to a file.

    2. eltoder
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Apr 10, 2011

    3. rhettinger commented on Apr 10, 2011

      @rhettinger
      rhettinger
      on Apr 10, 2011
      Contributor

      Inspecting the text disassembly is a bit fragile for testing. It would be better to scan a list of (opcode, oparg) pairs for given pattern (i.e. (LOAD_CONST, 3) where consts[3] --> some target value).

    4. eltoder commented on Apr 10, 2011

      @eltoder
      eltodermannequin
      on Apr 10, 2011
      MannequinAuthor

      Agreed, but that would require rewriting of all tests in test_peepholer.

    5. rhettinger commented on Apr 10, 2011

      @rhettinger
      rhettinger
      on Apr 10, 2011
      Contributor

      Yep!

    6. ncoghlan commented on Apr 10, 2011

      @ncoghlan
      ncoghlan
      on Apr 10, 2011
      Contributor

      I really like the idea of adding some lower level infrastructure to dis to make it generator based, making the disassembly more amenable to programmatic manipulation.

      Consider if, for each line disassemble() currently prints, we had an underlying iterator that yielded a named tuple consisting of (index, opcode, oparg, linestart, details). I've created a proof-of-concept for that in my sandbox (http://hg.python.org/sandbox/ncoghlan/file/get_opinfo/Lib/dis.py) which adds a get_opinfo() function that does exactly. With disassemble() rewritten to use that, test_dis and test_peepholer still pass as currently written.

      Near-term, test_peepholer could easily continue to do what it does now (i.e. use the higher level dis() function and redirect sys.stdout). Longer term, it could be written to analyse the opcode stream instead of doing string comparisons.

    7. ncoghlan commented on Apr 10, 2011

      @ncoghlan
      ncoghlan
      on Apr 10, 2011
      Contributor

      Changed issue title to cover ideas like get_opinfo().

    8. ncoghlan
      changed the title [-]Add functions to return disassembly as string[/-] [+]Refactor the dis module to provide better building blocks for bytecode analysis[/+] on Apr 10, 2011

    9. ncoghlan commented on Apr 10, 2011

      @ncoghlan
      ncoghlan
      on Apr 10, 2011
      Contributor

      Oops, I forgot to edit my comment to match the OpInfo definition I used in the proof-of-concept:

      OpInfo = collections.namedtuple("OpInfo",
            "opindex opcode opname oparg details starts_line is_jump_target")

    10. eltoder commented on Apr 10, 2011

      @eltoder
      eltodermannequin
      on Apr 10, 2011
      MannequinAuthor

      So in the near term, dis-based tests should continue to copy/paste sys.stdout redirection code?

    11. ncoghlan commented on Apr 10, 2011

      @ncoghlan
      ncoghlan
      on Apr 10, 2011
      Contributor

      If we decide our long term goal is the use of the opcode stream for programmatic access, then yes.

    12. alex commented on Apr 10, 2011

      @alex
      alex
      on Apr 10, 2011
      Member

      FWIW in PyPy we have https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py which we use for some of our tools.

    13. rhettinger
      self-assigned this
      on Apr 10, 2011

    14. jcea commented on Apr 11, 2011

      @jcea
      jcea
      on Apr 11, 2011
      Member

      Do not forget to update docs too.

    15. 34 remaining items

    16. python-dev commented on May 6, 2013

      @python-dev
      python-devmannequin
      on May 6, 2013
      Mannequin

      New changeset f65b867ce817 by Nick Coghlan in branch 'default':
      Issue bpo-11816: multiple improvements to the dis module
      http://hg.python.org/cpython/rev/f65b867ce817

    17. python-dev commented on May 6, 2013

      @python-dev
      python-devmannequin
      on May 6, 2013
      Mannequin

      New changeset d3fee4c64654 by Nick Coghlan in branch 'default':
      Issue bpo-11816: switch test_peepholer to bytecode_helper
      http://hg.python.org/cpython/rev/d3fee4c64654

    18. ncoghlan commented on May 6, 2013

      @ncoghlan
      ncoghlan
      on May 6, 2013
      Contributor

      And two-and-a-bit years later, we're done - thanks all, any further feedback or problems can be filed as a new issue :)

    19. ncoghlan
      closed this as completedon May 6, 2013

    20. neologix commented on May 6, 2013

      @neologix
      neologixmannequin
      on May 6, 2013
      Mannequin

      test_dis is failing on some buildbots:

      http://buildbot.python.org/all/builders/AMD64 Ubuntu LTS 3.x/builds/1674/steps/test/logs/stdio

      Re-running test 'test_dis' in verbose mode
test test_dis crashed -- Traceback (most recent call last):
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/regrtest.py", line 1294, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/opt/python/3.x.langa-ubuntu/build/Lib/importlib/__init__.py", line 92, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1603, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1584, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1551, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 591, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1053, in load_module
  File "<frozen importlib._bootstrap>", line 1034, in load_module
  File "<frozen importlib._bootstrap>", line 567, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 901, in _load_module
  File "<frozen importlib._bootstrap>", line 297, in _call_with_frames_removed
  File "/opt/python/3.x.langa-ubuntu/build/Lib/test/test_dis.py", line 4, in <module>
    from test.bytecode_helper import BytecodeTestCase
ImportError: No module named 'test.bytecode_helper'
    21. neologix
      reopened this on May 6, 2013

    22. pitrou commented on May 6, 2013

      @pitrou
      pitrou
      on May 6, 2013
      Member

      Yes, this is bytecode_helper hasn't been added to the repository.

    23. pitrou commented on May 6, 2013

      @pitrou
      pitrou
      on May 6, 2013
      Member

      (this is *because*, sorry)

    24. vstinner commented on May 6, 2013

      @vstinner
      vstinner
      on May 6, 2013
      Member

      Ping! The test is still failing.

    25. takluyver commented on May 6, 2013

      @takluyver
      takluyvermannequin
      on May 6, 2013
      Mannequin

      bytecode_helper is there in dis_api3.diff - anyone with commit rights should be able to add it to the repository.

    26. python-dev commented on May 6, 2013

      @python-dev
      python-devmannequin
      on May 6, 2013
      Mannequin

      New changeset 84d1a0e32d3b by Nick Coghlan in branch 'default':
      Issue bpo-11816: Add missing test helper
      http://hg.python.org/cpython/rev/84d1a0e32d3b

    27. ncoghlan commented on May 7, 2013

      @ncoghlan
      ncoghlan
      on May 7, 2013
      Contributor

      I checked in the missing file after I woke up this morning. Maybe I'll learn to use hg import instead of patch some day...

      Sorry for the noise.

    28. ncoghlan
      closed this as completedon May 7, 2013

    29. python-dev commented on Jan 19, 2016

      @python-dev
      python-devmannequin
      on Jan 19, 2016
      Mannequin

      New changeset bf997b22df06 by Victor Stinner in branch '3.5':
      Fix BytecodeTestCase.assertNotInBytecode()
      https://hg.python.org/cpython/rev/bf997b22df06

    30. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions