gh-154194: Degrade frames in Tachyon instead of failing the sample by maurycy · Pull Request #154195 · python/cpython

maurycy · 2026-07-19T20:01:33Z

As always, see #154194 for the context. Shortly: right now remote inspection fails the whole sample when it hits the limits (and we're really not sure what are the best limits) OR always when reading a part fails. This is too strict, wasteful and introduces scary bias, as observed in https://discuss.python.org/t/tachyon-97-error-rate/107619.

The PR introduces graceful degradation with the sentinels like <unreadable frame> (funcname), <unknown function> (funcname), <unknown file> (filename). Also, attemps to better differentiate between transient failures in _Py_RemoteDebug_IsFatalReadError().

Importantly: degraded frames are never cached, while the truncated strings are cached.

Issue: _remote_debugging: Degrade gracefully reads instead of failing the whole read completely #154194

maurycy · 2026-07-19T21:26:53Z

I believe that the CI failures are caused by the fact that test_async_aware_running_sees_only_cpu_task does not terminate. It worked only because a torn read caused an exception. That's what inspired #153365 in the first place.

pablogsal

I still need more time but some nits

maurycy · 2026-07-24T22:37:41Z

            # Build coroutine stack string
            frames = [frame for coro in task_info.coroutine_stack
                     for frame in coro.call_stack]
-            coro_stack = " -> ".join(_format_stack_entry(x).split(" ")[0]


This does not work for the <unknown function> sentinel.

maurycy · 2026-07-24T22:38:27Z

 def _format_stack_entry(elem: str|FrameInfo) -> str:
    if not isinstance(elem, str):


I believe that isinstance(elem, str) is dead since #135436. Perhaps we should clean this defensive code everywhere.

maurycy · 2026-07-24T22:43:45Z

 # ─── indexing helpers ───────────────────────────────────────────
 def _format_stack_entry(elem: str|FrameInfo) -> str:
    if not isinstance(elem, str):
+        if elem.location is None:


Regular collectors rely on these helpers:

cpython/Lib/profiling/sampling/collector.py

Lines 46 to 47 in 40f7fbf

if location is None:

return 0

cpython/Lib/profiling/sampling/collector.py

Lines 29 to 30 in 40f7fbf

if location is None:

return DEFAULT_LOCATION

maurycy · 2026-07-24T23:03:59Z

    PyObject *func = NULL;
    PyObject *file = NULL;
    PyObject *linetable = NULL;
+    int code_metadata_incomplete = 0;


Perhaps parse_code_object() should return this information, and we should not store incomplete objects in the frame_cache?

Perhaps we should collect degradation stats? Important caveat is that we will likely have more types of degradation.

maurycy · 2026-07-24T23:13:56Z

+                "<GC>",
+                "<native>",
+                "<unknown function>",
+                "<unknown file>",
+                "<unreadable frame>",


Perhaps we could add something like is_synthetic to FrameInfo?

…cefully

maurycy · 2026-10-04T19:53:41Z

@pablogsal I've synced with main, and removed the (len=) suffix. The #157790 did not include it. #153365 still needs to go first, because of #154195 (comment)

pablogsal

LGTM!

miss-islington-app · 2026-10-05T09:19:59Z

Thanks @maurycy for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

miss-islington-app · 2026-10-05T09:20:07Z

Sorry, @maurycy and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 7d25916b41de4fc28e04b4944864ad1094a9e8e0 3.15

bedevere-app · 2026-10-05T09:24:16Z

GH-158831 is a backport of this pull request to the 3.15 branch.

bedevere-app · 2026-10-05T09:26:40Z

GH-158833 is a backport of this pull request to the 3.14 branch.

…mple (GH-154195) (#158831) * gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195) * degrade gracefully * news * better NEWS wording * do not raise on MAX_REMOTE_STR_READ * bye MAX_REMOTE_STR_READ * fix -m asyncio ps|pstree * test truncation and linetable sentinel * simpler * simpler * redundant now * respect #157790 in the news --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 7d25916) * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>

…4195) (#158833) * gh-154194: Degrade unreadable frames on 3.14 (cherry picked from commit 7d25916) Co-authored-by: Pablo Galindo Salgado <pablogsal@gmail.com> * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>

…failing the sample (pythonGH-154195) (python#158831)" This reverts commit c27f494.

…58882) * [3.15] gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) (#158813) gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded (GH-153365) * let me declare single limit * use our new limit in process_frame_chain() * add it in parse_async_frame_chain() * parse_coro_chain() * NEWS * async in the message? * test * no race * process_task_awaited_by * process_task_awaited_by limit test * NEWS * MAX_TASK_WAITER_CHAIN_DEPTH * TASK_WAITER_CHAIN_DEPTH in test * TASK_WAITER_CHAIN_DEPTH 256 * prevent the drift with the comment * better naming, better style * MAX_TASK_WAITER_CHAIN_DEPTH comment * task-waiter iterative bfs walk * iterative coro-walk * nicer news * 1 << 14 * comment * unused read_Py_ssize_t * fix tombstones * simplify * correct msg * better test * news for tombstones * left-over from when testing buggy version * redundant new line (cherry picked from commit e0861c6) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commit 9e401cf) * [3.15] gh-155811: Add a seqcount to `gc_stats` to prevent torn reads (GH-155828) (#158829) * update_seq * no need for XCHGL, MOVL is enough? * gh-155811: Retry an inconsistent GC snapshot once --------- (cherry picked from commit 5fecd44) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 0281240) * [3.15] gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) (#158830) gh-151292: `_remote_debugging`: Do not corrupt the binary file when hitting `OverflowError` (GH-152892) * the kolektor * test * better test * news * =Q, move const to the base, not self.running * gh-151292: Track binary writer finalization state --------- (cherry picked from commit f839c06) Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 8a7c23f) * [3.15] gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (#158691) gh-158583: Fix uninitialized memory read in bytes.fromhex() (GH-158584) (cherry picked from commit 9d22a53) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit 4f7af46) * [3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) (#158831) * gh-154194: Degrade frames in Tachyon instead of failing the sample (#154195) * degrade gracefully * news * better NEWS wording * do not raise on MAX_REMOTE_STR_READ * bye MAX_REMOTE_STR_READ * fix -m asyncio ps|pstree * test truncation and linetable sentinel * simpler * simpler * redundant now * respect #157790 in the news --------- Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 7d25916) * Preserve the stable ABI when creating fallback frame names --------- Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> (cherry picked from commit c27f494) * [3.15] Add MSan to CI (GH-158625) (#158832) Run a job that the test suite with MSan to the CI (#158625) * Run the test suite with MSan in CI * Additional fixes * Add `_Py_MSAN_UNPOISON_STRING` * Apply Victor's suggestions * Apply Victor's suggestions --------- (cherry picked from commit b93fb19) Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> (cherry picked from commit 1bc78de) * [3.15] gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (#156814) gh-156810: Write the profiler's collapsed-stack export as UTF-8 (GH-156811) (cherry picked from commit c3706f4) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commit fce28da) * [3.15] gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) (#158845) gh-158552: Wait for Windows threads to suspend before blocking sampling (GH-158802) * gh-158552: Wait for Windows threads to suspend before blocking sampling * Use a named Windows thread enumeration status constant (cherry picked from commit 1643525) Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> (cherry picked from commit 31288da) * [3.15] gh-152721: Fix quadratic RLE replay time in the profiling binary reader (GH-152722) (#158850) Backport of GH-152722. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commit ebaca2a) * [3.15] gh-156545: Fix flamegraph export RecursionError on deeply recursive programs (GH-156546) (#158851) Backport of GH-156546. Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> (cherry picked from commit 0ac7217) * [3.15] gh-158540: Add the profiled script's directory to sys.path (GH-158548) (#158844) gh-158540: Add the profiled script's directory to sys.path (GH-158548) * gh-158540: Add the profiled script's directory to sys.path When a script is profiled with ``python -m profiling.sampling run script.py`` from another directory, the script cannot import modules placed next to it, because ``_sync_coordinator._execute_script()`` executes it with the working directory (added by ``_setup_environment()`` for the module case) as ``sys.path[0]`` instead of the script's own directory. Make the script's directory importable in ``_execute_script()``, matching the behavior of ``python script.py``. Add a regression test that runs the coordinator on a script importing a sibling module. * Update _sync_coordinator.py Comment simplified * gh-158540: Resolve symlinks when adding the script directory to sys.path ``python script.py`` resolves symlinks when computing ``sys.path[0]``, so a script reached through a symlink (``link.py -> sub/where.py``) imports modules from the real script's directory. Apply ``os.path.realpath()`` before taking the directory name, and make sure the result is placed at the front of ``sys.path`` even if it was already listed. Add a regression test for a symlinked script. --------- (cherry picked from commit 3f02aab) Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> (cherry picked from commit f52d831) * [3.15] gh-153838: Skip non-regular source files in the heatmap exporter (GH-153839) (#158853) Backport of GH-153839. Co-authored-by: tonghuaroot <tonghuaroot@gmail.com> (cherry picked from commit 48998df) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) (#158852) * [3.15] gh-158539: Fix exception mode missing handlers in generators/coroutines (GH-158581) Backport of GH-158581. Co-authored-by: LucasZhou <donghao.zhou@outlook.com> * [3.15] gh-158539: Use portable static assertion messages * [3.15] gh-158539: Keep layout assertions with debug-offset validation * [3.15] gh-158539: Use the platform guard for in-process inspection tests --------- Co-authored-by: LucasZhou <donghao.zhou@outlook.com> (cherry picked from commit d625ecb) * [3.15] gh-158522: Fix truncated stack for a task whose coroutine recurses (GH-158526) (#158870) Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com> (cherry picked from commit a904b39) * gh-156545, gh-158539: Fix deep flamegraph export on small C stacks and macOS runtime lookup (#158874) (cherry picked from commit 114de19) Include the C-stack test helper from main, introduced by ce5ae29, which the regression test requires but 3.15 does not yet provide. --------- Co-authored-by: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Stan Ulbrych <stan@python.org> Co-authored-by: Victor Stinner <victor.stinner@gmail.com> Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: he_tao <53343436+hetaozdh@users.noreply.github.com> Co-authored-by: Eduardo Villalpando Mello <eduardo.villalpando.mello@gmail.com> Co-authored-by: LucasZhou <donghao.zhou@outlook.com> Co-authored-by: Timofei Ivankov <128279579+deadlovelll@users.noreply.github.com>

degrade gracefully

8aaf6f0

bedevere-app Bot mentioned this pull request Jul 19, 2026

_remote_debugging: Degrade gracefully reads instead of failing the whole read completely #154194

Closed

maurycy mentioned this pull request Jul 19, 2026

tachyon: degrade reads instead of failing the whole read completely maurycy/cpython#7

Closed

news

f45ef66

maurycy marked this pull request as ready for review July 19, 2026 20:17

maurycy requested a review from pablogsal as a code owner July 19, 2026 20:17

bedevere-app Bot added the awaiting review label Jul 19, 2026

pablogsal reviewed Jul 20, 2026

View reviewed changes

Comment thread Modules/_remote_debugging/object_reading.c Outdated

Comment thread Misc/NEWS.d/next/Library/2026-07-19-22-08-05.gh-issue-154194.1bqRdx.rst Outdated

better NEWS wording

ad797d5

maurycy mentioned this pull request Jul 21, 2026

gh-153364: Make frame, coroutine, and task-waiter chain walks iterative and bounded #153365

Merged

maurycy changed the title ~~gh-154194: Do not fail the whole sample in Tachyon if we can degrade~~ gh-154194: Degrade frames in Tachyon instead of failing the sample Jul 21, 2026

maurycy added 4 commits July 24, 2026 23:06

do not raise on MAX_REMOTE_STR_READ

3e58d13

bye MAX_REMOTE_STR_READ

55632e9

fix -m asyncio ps|pstree

6736133

test truncation and linetable sentinel

17b4197

maurycy requested review from 1st1, asvetlov, kumaraditya303 and willingc as code owners July 24, 2026 22:18

maurycy commented Jul 24, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into tachyon-degrade-gra…

8213bd3

…cefully

maurycy mentioned this pull request Oct 4, 2026

gh-157227: _remote_debugging: Validate lists and tuples in write_sample #157228

Open

maurycy and others added 5 commits October 4, 2026 22:21

simpler

443f5c1

simpler

bb28257

redundant now

be34b1f

respect python#157790 in the news

2a88472

pythongh-154194: Integrate bounded walks and simplify degradation tests

c6a44a8

pablogsal approved these changes Oct 5, 2026

View reviewed changes

pablogsal enabled auto-merge (squash) October 5, 2026 00:09

bedevere-app Bot added awaiting merge and removed awaiting review labels Oct 5, 2026

pablogsal merged commit 7d25916 into python:main Oct 5, 2026
57 of 58 checks passed

bedevere-app Bot removed the awaiting merge label Oct 5, 2026

maurycy deleted the tachyon-degrade-gracefully branch October 5, 2026 04:28

pablogsal added the needs backport to 3.15 bugs and security fixes label Oct 5, 2026

miss-islington-app Bot assigned pablogsal Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.15] gh-154194: Degrade frames in Tachyon instead of failing the sample (GH-154195) #158831

Merged

bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Oct 5, 2026

pablogsal mentioned this pull request Oct 5, 2026

[3.14] gh-154194: Degrade frames instead of failing the sample (GH-154195) #158833

Merged

pablogsal added a commit to pablogsal/cpython that referenced this pull request Oct 5, 2026

Revert "[3.15] pythongh-154194: Degrade frames in Tachyon instead of …

28d4699

…failing the sample (pythonGH-154195) (python#158831)" This reverts commit c27f494.

		def _format_stack_entry(elem: str\|FrameInfo) -> str:
		if not isinstance(elem, str):

Conversation

maurycy commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maurycy commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maurycy Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

maurycy commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

miss-islington-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

bedevere-app Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

maurycy commented Jul 19, 2026 •

edited

Loading

maurycy commented Jul 19, 2026 •

edited

Loading

maurycy Jul 24, 2026 •

edited

Loading

maurycy Jul 24, 2026 •

edited

Loading

maurycy commented Oct 4, 2026 •

edited

Loading