Skip to content

trace.py bug with the main file being traced #53569

Closed
Closed
trace.py bug with the main file being traced#53569
Assignees
abalkin
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@elibendersky
elibendersky
mannequin
opened on Jul 21, 2010
BPO 9323
Nosy @abalkin, @ezio-melotti, @florentx
Files
  • traceme.py
  • issue9323.1.patch
  • issue9323_regrtest.diff: Patch, apply to 3.x
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/abalkin'
closed_at = <Date 2010-09-13.17:41:50.771>
created_at = <Date 2010-07-21.14:30:31.553>
labels = ['type-bug', 'library']
title = 'trace.py bug with the main file being traced'
updated_at = <Date 2010-09-13.17:41:50.770>
user = 'https://bugs.python.org/elibendersky'

    bugs.python.org fields:

    activity = <Date 2010-09-13.17:41:50.770>
actor = 'belopolsky'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2010-09-13.17:41:50.771>
closer = 'belopolsky'
components = ['Library (Lib)']
creation = <Date 2010-07-21.14:30:31.553>
creator = 'eli.bendersky'
dependencies = []
files = ['18105', '18106', '18870']
hgrepos = []
issue_num = 9323
keywords = ['patch']
message_count = 10.0
messages = ['111066', '111074', '111094', '113144', '113152', '113153', '116307', '116321', '116333', '116334']
nosy_count = 4.0
nosy_names = ['belopolsky', 'ezio.melotti', 'eli.bendersky', 'flox']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9323'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

    Activity

    1. elibendersky commented on Jul 21, 2010

      @elibendersky
      elibenderskymannequin
      on Jul 21, 2010
      MannequinAuthor

      [This bug was discovered by Alexander Belopolsky, during the work on bpo-9317]

      Bug report

      The attached traceme.py file demonstrates the following problem:

      With python 2.7:
      $ python2 -m trace -c -s traceme.py
      lines cov% module (path)
      1 100% threading (Lib/threading.py)
      6 100% traceme (traceme.py)

      The first entry is clearly spurious, but traceme looks right. With py3k, however, I get

      $ python3 -m trace  -c -s traceme.py
lines   cov%   module   (path)
    1   100%   threading   (Lib/threading.py)

      No traceme line at all.

      Analysis

      When trace.py actually runs the script, it uses this code in 3.x:

             t.run('exec(%r)' % (script,))

      instead of this code in 2.x:

              t.run('execfile(%r)' % (progname,))

      exec doesn't have the program name to attach to the code object created, and calls it '' by default. However, this file name is ignored by the trace module.

      execfile doesn't exist in 3.x, so an alternative approach is needed.

      Proposed solution

      Instead of t.run as displayed above, use this code:

                  with open(progname) as fp:
                code = compile(fp.read(), progname, 'exec')
                t.run(code)

      The code object created by compile attaches the program name and is executable by t.run (because t.run actually calls exec which can run the result of compile).

      This solution fixes the problem:

      $ py3d -m trace -c -s traceme.py 
lines   cov%   module   (path)
    1   100%   threading   (/home/eliben/python_src/eliben-py3k/Lib/threading.py)
    6   100%   traceme   (traceme.py)

      I'll attach a patch file that was creates vs. the latest trunk, with this fix.

    2. elibendersky
      added
      type-bugAn unexpected behavior, bug, or error
      stdlibStandard Library Python modules in the Lib/ directory
      on Jul 21, 2010

    3. abalkin commented on Jul 21, 2010

      @abalkin
      abalkin
      on Jul 21, 2010
      Member

      [Copying msg111059 from bpo-9317]

      On Wed, Jul 21, 2010 at 9:45 AM, Eli Bendersky <report@bugs.python.org> wrote:
      ..

      As far as I understand, when you run:

      py3d -m trace -C pickle-trace.d -c -m test_pickle.py

      The first -m flag applies to the trace module. Python uses
      runpy.run_module to load it, and then passes it its arguments (-C and
      onwards).

      When I said "run from command line", I meant something like

      $ python Lib/test/test_pickle.py

      I am not sure if this uses runpy machinery, but I suspect it does.

      The trace output should not include runpy calls that load the trace
      module itself, but

      $ python -m trace <flags> Lib/test/test_pickle.py

      might show the runpy calls that occur when you run

      $ python Lib/test/test_pickle.py

      without tracing. BTW, too bad that '-m' is taken, but I think
      trace.py should grow a --run-module option so that you can trace
      equivalents of python -m executions:

      $ python -m trace <flags> --run-module test.test_pickle

      Moreover, since runpy is only the machinery to execute trace.py, does
      it even make sense to include it in coverage?

      What do you think?

      I think you should ask python-dev. :-) Note that most C/C++ coverage
      tools do include pre-main calls in coverage. In case of C++, a lot
      of things may happen before main().

      Whatever the solution is, it should be unified between trace, profile,
      and pdb modules.

    4. abalkin
      self-assigned this
      on Jul 21, 2010

    5. abalkin commented on Jul 21, 2010

      @abalkin
      abalkin
      on Jul 21, 2010
      Member

      Committed with a minor change. (There is no need to keep the source file open during the trace run.) See r83035 (r83037 in release31-maint.)

      I will open a separate issue to discuss a possible RFE related to msg111074.

    6. abalkin
      closed this as completedon Jul 21, 2010

    7. abalkin commented on Aug 6, 2010

      @abalkin
      abalkin
      on Aug 6, 2010
      Member

      Reopening because I think this fix is responsible for a new problem:

      $ ./python.exe -m trace -C. -s -c Lib/test/regrtest.py 
Traceback (most recent call last):
  ...

      File "Lib/test/regrtest.py", line 1458, in <module>
      assert __file__ == os.path.abspath(sys.argv[0])
      AssertionError
      [98796 refs]

    8. abalkin
      reopened this on Aug 6, 2010

    9. elibendersky commented on Aug 7, 2010

      @elibendersky
      elibenderskymannequin
      on Aug 7, 2010
      MannequinAuthor

      It appears to affect 2.7 as well (probably because of a back-port from py3k to 27-maint in revision 83421).

    10. elibendersky commented on Aug 7, 2010

      @elibendersky
      elibenderskymannequin
      on Aug 7, 2010
      MannequinAuthor

      Looking at Lib/test/regrtest.py:

        if __name__ == '__main__':
      # Simplification for findtestdir().
      assert __file__ == os.path.abspath(sys.argv[0])

      I'm not sure regrtest.py is prepared to be run from trace at all. Note that this error also happens when running 'python -m profile regrtest.py', so it's rather regrtest.py specific than trace.py specific.

      Anyway, trace.py itself currently passes the cmd-line argument as __file__ and not its full path, which causes the breakage in regrtest.py. The full path *could* be easily passed, but I think we should address this problem from regrtest.py's side.

    11. abalkin commented on Sep 13, 2010

      @abalkin
      abalkin
      on Sep 13, 2010
      Member

      The full path *could* be easily passed, but I think we should
      address this problem from regrtest.py's side.

      I disagree. There may be other scripts that rely on __main__.__file__ being an absolute path and we cannot require everyone to change their assumptions to make their scripts traceable.

      It looks like absolute path assumption was added by Florent in r78723 (r78719 in trunk). Florent, I would like to hear from you before deciding how to proceed. See also issue bpo-7712.

    12. florentx commented on Sep 13, 2010

      @florentx
      florentxmannequin
      on Sep 13, 2010
      Mannequin

      I've made the wrong assumption that test.regrtest.__file__ is always absolute.

      Here is a patch which should fix it.

    13. florentx commented on Sep 13, 2010

      @florentx
      florentxmannequin
      on Sep 13, 2010
      Mannequin

      regrtest fixed with r84776 (3.x) and r84779 (2.7)

    14. abalkin commented on Sep 13, 2010

      @abalkin
      abalkin
      on Sep 13, 2010
      Member

      Thanks, Florent.

    15. abalkin
      closed this as completedon Sep 13, 2010
    16. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions