Skip to content

gh-119786: move interpreter doc from devguide to InternalDocs #125715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iritkatriel merged 6 commits into from
Oct 21, 2024
Merged

gh-119786: move interpreter doc from devguide to InternalDocs #125715

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d65ee0f
gh-119786: move interpreter doc from devguide to InternalDocs
iritkatriel Oct 18, 2024
7cb1441
remove title
iritkatriel Oct 18, 2024
bf33ab6
Apply suggestions from code review
iritkatriel Oct 21, 2024
2916a14
relative paths to other InternalDocs
iritkatriel Oct 21, 2024
4ed6f8b
relative paths to code
iritkatriel Oct 21, 2024
11b9047
fix link
iritkatriel Oct 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
relative paths to code
  • Loading branch information
@iritkatriel
iritkatriel committed Oct 21, 2024
commit 4ed6f8b1e749586fc853d24313e8e9d95f8f239a
57 changes: 24 additions & 33 deletions InternalDocs/interpreter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,13 @@ Overview

This document describes the workings and implementation of the bytecode
interpreter, the part of python that executes compiled Python code. Its
entry point is in
[Python/ceval.c](https://github.com/python/cpython/blob/main/Python/ceval.c).
entry point is in [Python/ceval.c](../Python/ceval.c).

At a high level, the interpreter consists of a loop that iterates over the
bytecode instructions, executing each of them via a switch statement that
has a case implementing each opcode. This switch statement is generated
from the instruction definitions in
[Python/bytecodes.c](https://github.com/python/cpython/blob/main/Python/bytecodes.c)
which are written in
[a DSL](https://github.com/python/cpython/blob/main/Tools/cases_generator/interpreter_definition.md)
from the instruction definitions in [Python/bytecodes.c](../Python/bytecodes.c)
which are written in [a DSL](../Tools/cases_generator/interpreter_definition.md)
developed for this purpose.

Recall that the [Python Compiler](compiler.md) produces a [`CodeObject`](code_object.md),
Expand Down Expand Up @@ -139,8 +136,7 @@ Although they are represented by code units, cache entries do not conform to the
`opcode` / `oparg` format.

If an instruction has an inline cache, the layout of its cache is described by
a `struct` definition in
(`pycore_code.h`)[https://github.com/python/cpython/blob/main/Include/internal/pycore_code.h].
a `struct` definition in (`pycore_code.h`)[../Include/internal/pycore_code.h].
This allows us to access the cache by casting `next_instr` to a pointer to this `struct`.
The size of such a `struct` must be independent of the machine architecture, word size
and alignment requirements. For a 32-bit field, the `struct` should use `_Py_CODEUNIT field[2]`.
Expand Down Expand Up @@ -170,8 +166,8 @@ stack can be pre-allocated is a contiguous array of `PyObject*` pointers, when t
is created.

The stack effects of each instruction are also exposed through the
[opcode metadata](https://github.com/python/cpython/blob/main/Include/internal/pycore_opcode_metadata.h)
through two functions that report how many stack elements the instructions consumes,
[opcode metadata](../Include/internal/pycore_opcode_metadata.h) through two
functions that report how many stack elements the instructions consumes,
and how many it produces (`_PyOpcode_num_popped` and `_PyOpcode_num_pushed`).
For example, the `BINARY_OP` instruction pops two objects from the stack and pushes the
result back onto the stack.
Expand Down Expand Up @@ -202,8 +198,7 @@ Error handling
--------------

When the implementation of an opcode raises an exception, it jumps to the
`exception_unwind` label in
[Python/ceval.c](https://github.com/python/cpython/blob/main/Python/ceval.c).
`exception_unwind` label in [Python/ceval.c](../Python/ceval.c).
The exception is then handled as described in the
[`exception handling documentation`](exception_handling.md#handling-exceptions).

Expand Down Expand Up @@ -311,29 +306,27 @@ a new feature or change the way that existing features are compiled.
This section describes the changes required to do this.

First, you must choose a name for the bytecode, implement it in
[`Python/bytecodes.c`](https://github.com/python/cpython/blob/main/Python/bytecodes.c)
and add a documentation entry in
[`Doc/library/dis.rst`](https://github.com/python/cpython/blob/main/Doc/library/dis.rst).
[`Python/bytecodes.c`](../Python/bytecodes.c) and add a documentation
entry in [`Doc/library/dis.rst`](../Doc/library/dis.rst).
Then run `make regen-cases` to assign a number for it (see
[`Include/opcode_ids.h`](https://github.com/python/cpython/blob/main/Include/opcode_ids.h))
and regenerate a number of files with the actual implementation of the bytecode in
[`Python/generated_cases.c.h`](https://github.com/python/cpython/blob/main/Python/generated_cases.c.h)
and metadata about it in additional files.
[`Include/opcode_ids.h`](../Include/opcode_ids.h)) and regenerate a
number of files with the actual implementation of the bytecode in
[`Python/generated_cases.c.h`](../Python/generated_cases.c.h) and
metadata about it in additional files.

With a new bytecode you must also change what is called the "magic number" for
.pyc files: bump the value of the variable `MAGIC_NUMBER` in
[`Lib/importlib/_bootstrap_external.py`](https://github.com/python/cpython/blob/main/Lib/importlib/_bootstrap_external.py).
[`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py).
Changing this number will lead to all .pyc files with the old `MAGIC_NUMBER`
to be recompiled by the interpreter on import. Whenever `MAGIC_NUMBER` is
changed, the ranges in the `magic_values` array in
[`PC/launcher.c`](https://github.com/python/cpython/blob/main/PC/launcher.c)
may also need to be updated. Changes to
[`Lib/importlib/_bootstrap_external.py`](https://github.com/python/cpython/blob/main/Lib/importlib/_bootstrap_external.py)
[`PC/launcher.c`](../PC/launcher.c) may also need to be updated. Changes to
[`Lib/importlib/_bootstrap_external.py`](../Lib/importlib/_bootstrap_external.py)
will take effect only after running `make regen-importlib`.

> [!NOTE]
> Running `make regen-importlib` before adding the new bytecode target to
> [`Python/bytecodes.c`](https://github.com/python/cpython/blob/main/Python/bytecodes.c)
> [`Python/bytecodes.c`](../Python/bytecodes.c)
> (followed by `make regen-cases`) will result in an error. You should only run
> `make regen-importlib` after the new bytecode target has been added.

Expand All @@ -342,15 +335,13 @@ will take effect only after running `make regen-importlib`.
> regenerate the required files without requiring additional arguments.

Finally, you need to introduce the use of the new bytecode. Update
[`Python/codegen.c`](https://github.com/python/cpython/blob/main/Python/codegen.c)
to emit code with this bytecode. Optimizations in :cpy-file:`Python/flowgraph.c`
may also need to be updated. If the new opcode affects a control flow or the
block stack, you may have to update the `frame_setlineno()` function in
[`Objects/frameobject.c`](https://github.com/python/cpython/blob/main/Objects/frameobject.c).
It may also be necessary to update
[`Lib/dis.py`](https://github.com/python/cpython/blob/main/Lib/dis.py)
if the new opcode interprets its argument in a special way (like `FORMAT_VALUE`
or `MAKE_FUNCTION`).
[`Python/codegen.c`](../Python/codegen.c) to emit code with this bytecode.
Optimizations in :cpy-file:`Python/flowgraph.c` may also need to be updated.
If the new opcode affects a control flow or the block stack, you may have to
update the `frame_setlineno()` function in
[`Objects/frameobject.c`](../Objects/frameobject.c). It may also be necessary
to update [`Lib/dis.py`](../Lib/dis.py) if the new opcode interprets its
argument in a special way (like `FORMAT_VALUE` or `MAKE_FUNCTION`).

If you make a change here that can affect the output of bytecode that
is already in existence and you do not change the magic number, make
Expand Down