Annotate test cases which are timing sensitive

Feature or enhancement

Proposal:

Many tests in the cpython test suite are sensitive to timing and assume an entirely unloaded system. For example, the TimerfdTests check for 1ms accuracy in expected durations. Some people (eg people building and testing python) may be running the test suite on build machines which are potentially heavily loaded so this sort of expectation isn't feasible.

It seems like a good solution to this would be to annotate the tests which are timing sensitive in some way, so they can be skipped easily.

For reference, we're manually patching python to skip tests which are failing under load but we're now having to rebase these patches on upgrades:

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

gh-130363: add a test resource to mark tests that need an idle system #130508

Sounds like it could be a new resource, see python -m test --help and test.support.requires.

I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource. Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable?

Agreed - we've a short list of known flaky tests, and need to triage it to identify the ones which are genuinely sensitive vs those with subtle race conditions which should be fixed.

I will prototype a new resource and see how it works.

If it helps, to add a new resource, you just need to add it here:

cpython/Lib/test/libregrtest/utils.py

Lines 35 to 36 in d8ce092

    
           ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network', 
        
                            'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 'walltime')

And then you can annotate the tests with @support.requires_resource('your_new_resource')

Though for some of the other tests (e.g. test_input_no_stdout_fileno) it's not immediately obvious to me why they would fail, so perhaps instead of skipping them we could look into making them more reliable?

How about a flaky resources. At least, we know which tests are meant to be flaky and which are not. And for the CI, we could ask the CI to rerun flaky tests on failures so that contributors don't need to wait for a triager/core dev to rerun a failed workflow when needed.

I think it makes sense to hide the tests that clearly depend on timings (e.g. test_process_time) behind a test resource

I suggest adding stats resources which could cover all tests that are meant to gather timing and check that some condition is statistically verified. Other possible names that are more or less generic: timeprec, measures, experiments. I don't really have better names =/

I like the flaky test idea! For timing-related tests, I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings?

I like the flaky test idea!

Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.

I don't have a good suggestion either, though stats seems a bit too generic to me (but I guess how generic we want this resource to be). timeprec is more self-evident, or maybe timing or timings?

I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this. Memory cannot really be flaky unless the kernel is doing some weird stuff and network can fail but generally it's a timeout issue, not a data issue itself. So timings looks quite promising.

Let's open a separate issue for the flaky tests as it could be orthogonal to tests that require an idle system due to stats gathering.

Issue's here: #130474

I suggested "stats" because I don't think we should have a test that doesn't check against a population that some hypothesis is verified. However, I'm not sure we have any test that would test something else than timings like this.

Yeah, I'm not sure if we assert anything else besides timings that is similarly stochastic.. maybe we can stick with some time-related name for now and adjust it if something else comes up?

pythongh-130363: add a test resource to mark tests that need an idle …

Posted a minimal PR: #130508. This just adds an "idle" resource.

I'm not comfortable with this issue. "Skip tests because they fail on our CI" sounds too specific to me. IMO a "skip list" for a specific CI is perfectly fine.

Your patches use the description:

This test hangs frequently when run on the Autobuilder. Disable it in testing until the cause can be determined.

Well, yes, the cause should be determined.

0001-test_storlines-skip-due-to-load-variability.patch: it would be interesting to investigate why this specific test "intermittently fails on the Yocto AB when a worker is under heavy load".

We already have the "walltime" resources for "slow tests". You might start by skipping this resource?

Slow tests isn't a problem, it's tests that take 1s but have precise timing restrictions that are.

Another test case that is load-sensitive: #129266

We've just disabled test_only_active_thread because it was randomly failing:

FAIL: test_only_active_thread (test.test_external_inspection.TestGetStackTrace.test_only_active_thread)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.14/test/test_external_inspection.py", line 1246, in test_only_active_thread
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        len(gil_traces), 1, "Should have exactly one GIL holder"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
AssertionError: 0 != 1 : Should have exactly one GIL holder

So that's another candidate for "sensitive to load".

More load-sensitive tests for reference:

FAIL: test_repr_rlock (test.test_multiprocessing_forkserver.test_processes.WithProcessesTestLock.test_repr_rlock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.13/test/_test_multiprocessing.py", line 1494, in test_repr_rlock
    self.assertEqual('<RLock(SomeOtherThread, nonzero)>', repr(lock))
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '<RLock(SomeOtherThread, nonzero)>' != '<RLock(None, 0)>'
- <RLock(SomeOtherThread, nonzero)>
+ <RLock(None, 0)>

ERROR: test_taskgroup_23 (test.test_asyncio.test_taskgroups.TestTaskGroup.test_taskgroup_23)
----------------------------------------------------------------------
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.12/unittest/async_case.py", line 90, in _callTestMethod
  |     if self._callMaybeAsync(method) is not None:
  |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.12/unittest/async_case.py", line 112, in _callMaybeAsync
  |     return self._asyncioRunner.run(
  |            ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.12/test/test_asyncio/test_taskgroups.py", line 657, in test_taskgroup_23
  |     async with taskgroups.TaskGroup() as g:
  |   File "/usr/lib/python3.12/asyncio/taskgroups.py", line 145, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.12/test/test_asyncio/test_taskgroups.py", line 664, in test_taskgroup_23
    |     self.assertEqual(len(g._tasks), 0)
    | AssertionError: 2 != 0
    +------------------------------------

FAIL: test_attach_to_process_with_colors (test.test_remote_pdb.PdbAttachTestCase.test_attach_to_process_with_colors)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.14/test/test_remote_pdb.py", line 1616, in test_attach_to_process_with_colors
    self.assertEqual(output["server"]["stdout"], "Function returned: 42\n")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Function returned: 1\n' != 'Function returned: 42\n'
- Function returned: 1
?                    ^
+ Function returned: 42
?                    ^^

python3: ptest: skip flaky test_attach_to_process_with_colors

python3: ptest: skip flaky test_interrupt on musl

python3: ptest: skip flaky test_attach_to_process_without_colors

python3: ptest: skip flaky test_taskgroup_23

Another potentially timing sensitive test case in test_asyncio, although it might just be buggy: #158097.

Navigation Menu

Repository navigation

Annotate test cases which are timing sensitive #130363

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network',
	'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 'walltime')

Annotate test cases which are timing sensitive #130363

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

encukou commented on Feb 21, 2025

tomasr8 commented on Feb 22, 2025

rossburton commented on Feb 22, 2025

tomasr8 commented on Feb 22, 2025

picnixz commented on Feb 22, 2025

tomasr8 commented on Feb 22, 2025

picnixz commented on Feb 22, 2025

tomasr8 commented on Feb 22, 2025

rossburton commented on Feb 24, 2025

vstinner commented on Feb 24, 2025

rossburton commented on Feb 24, 2025

8 remaining items

rossburton commented on Jan 22, 2026

rossburton commented on Mar 10, 2026

rossburton commented on Jun 11, 2026

rossburton commented on Sep 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions