[ci][docker] build py3.14 ray image by elliot-barn · Pull Request #63237 · ray-project/ray

elliot-barn · 2026-05-08T21:44:48Z

Adds CPython 3.14 as a supported target in Ray's CI/release infrastructure: wheel builder, image build pipeline, dependency locks, release-test config schema, and a hello_world_py314 smoke release test.

Follow-up PR carries the _raylet.so runtime fix that makes async actors actually work on cp3.14. This PR enables the build/test plumbing only; without the runtime PR landing first, any release test that exercises async actors will crash on JobSupervisor.

Buildkite image build pipeline

.buildkite/_images.rayci.yml — adds py314 entries to the base/cuda/slim image matrices so the image-matrix steps emit cp3.14 variants.
.buildkite/build.rayci.yml — image build + publish steps for ray-py3.14 and ray-extra-py3.14.
.buildkite/linux_aarch64.rayci.yml — aarch64 base + image build + publish for cp3.14.
.buildkite/release/build.rayci.yml — anyscale release-test images (raycpubaseextra-testdeps, raycudabaseextra-testdeps, ray-anyscale-cpu-build, ray-anyscale-cuda-build, anyscalecpubuild, anyscalecudabuild) extended to cp3.14. This is what unblocks the prior CI error Step dependencies not found: anyscalecpubuild--python314.
ci/raydepsets/configs/rayimg.depsets.yaml — declares the cp3.14 depset for the ray image.

Dependency locks (raydepsets)

New cp3.14 lock files generated via raydepsets:

Wheel builder + release-test config

ci/build/build_wheel.py — SUPPORTED_PYTHON_VERSIONS includes "3.14", so python ci/build/build_wheel.py produces cp314 wheels.
release/ray_release/schema.json — "3.14" added to the python enum (without this, release-test config validation rejects python: "3.14").
release/ray_release/config.py — "3.14" added to the BYOD-type allowlist (parallel change to schema.json — both gates the release runner walks before accepting a test).

Release smoke test

release/release_tests.yaml — adds hello_world_py314 (aws variation only, nightly cadence) that submits a trivial ray.get(hello_world.remote()) job. Uses the same hello_world_compute_config.yaml as the existing hello_world test, just pinned to python: "3.14" and anyscale_sdk_2026: true.

Misc

docker/base-deps/Dockerfile — bumps libffi (cp3.14's _ctypes build needs the newer ABI).

Verification

hello_world_py314 release test passes against an image built from this branch

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

gemini-code-assist

Code Review

This pull request adds support for Python 3.14 across various CI and dependency configuration files. However, the review highlights that this addition is premature due to a lack of upstream support for Python 3.14 in critical libraries like PyTorch, TensorFlow, and JAX. Additionally, the newly added python/requirements_compiled_py3.14.txt contains invalid package versions and incorrect environment markers that will lead to immediate build failures and runtime errors.

gemini-code-assist · 2026-05-08T21:47:32Z

+    #   dask
+    #   distributed
+    #   partd
+torch==2.7.0


This requirements file contains numerous non-existent or "future" package versions that are not available on PyPI or the specified indices. Examples include torch==2.7.0 (line 2373), numpy==2.3.2 (line 1267), and several packages with futuristic 2026.x versions (e.g., certifi==2026.4.22 on line 284). Additionally, the --find-links URL on line 2 points to a non-existent version. These errors will cause the Docker image build to fail immediately.

gemini-code-assist · 2026-05-08T21:47:32Z

        - "3.11"
        - "3.12"
        - "3.13"
+        - "3.14"


Adding support for Python 3.14 is premature. Python 3.14 is currently in early alpha development and is not yet supported by critical upstream dependencies such as PyTorch, TensorFlow, and JAX. Furthermore, the Python ABI is not yet stable for this version. Adding it to the build matrix at this stage will result in broken CI builds and non-functional Docker images.

gemini-code-assist · 2026-05-08T21:47:32Z

+    # via msrestazure
+adlfs==2026.4.0
+    # via -r python/requirements/cloud-requirements.txt
+aim==3.23.0 ; python_version < "3.12"


Many dependencies in this file (e.g., aim on line 33, array-record on line 120, configspace on line 387) include environment markers such as ; python_version < "3.12". Since this file is intended for Python 3.14, these dependencies will be skipped during installation, which will lead to missing packages and runtime failures in the Ray environment.

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…oject/ray into elliot-barn-314-ray-image

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…ot-barn-314-ray-image

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

github-actions · 2026-05-29T13:28:09Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

m-k-l-s · 2026-06-12T14:55:22Z

Hi, is there anything blocking this, please?

elliot-barn · 2026-06-18T22:27:58Z

Hi, is there anything blocking this, please?

Was busy with other tasks but back to testing this before merging

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d7b2d3f. Configure here.}

cursor · 2026-06-18T22:38:26Z

        - "3.11"
        - "3.12"
        - "3.13"
+        - "3.14"


CUDA 13 py3.14 CI mismatch

Medium Severity

Python 3.14 is added to CUDA release and image-build matrices (including 13.0.0-cudnn), but dependency tooling explicitly excludes cp3.14 from the cu130 chain because cupy-cuda13x==13.6.0 has no cp314 wheel. Final CUDA images still run ray-image.Dockerfile, which installs that pin on cu13* platforms, so those new pipeline jobs are expected to fail at image build time.

Additional Locations (2)

.buildkite/build.rayci.yml#L149-L166

.buildkite/_images.rayci.yml#L59-L77

^{Reviewed by Cursor Bugbot for commit d7b2d3f. Configure here.}

…mage resolving CPython 3.14.0 (#64857) ## Description ### Release smoke test Adds a `hello_world_py314` nightly smoke release test (aws variation only), mirroring the existing `hello_world_py313` entry: - `release/ray_release/schema.json` — add `"3.14"` to the `python` enum (release-test config validation rejects `python: "3.14"` without this). - `release/ray_release/config.py` — add `"3.14"` to the cpu/cu123 BYOD python allowlist (the parallel gate the release runner walks). - `release/release_tests.yaml` — new `hello_world_py314` test, nightly, `byod: {}`, same `hello_world_compute_config.yaml` as the other hello_world tests. The py3.14 `ray-anyscale` release-test images (cpu + cuda) are already built and published on master via `.buildkite/release/build.rayci.yml`, so no image plumbing is needed here. ### Base-image fix (what made the smoke test fail) The first run of this test failed with the JobSupervisor dying at job startup: `Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow`. Root cause: the py3.14 images ship **CPython 3.14.0**, which fatally crashes any Ray async actor running on a boost fiber stack (python/cpython#141944, fixed upstream in **3.14.2**). Why the images resolve 3.14.0: `docker/base-deps/Dockerfile` installs an exact `libffi=3.4.6` pin as a *separate* conda step after installing python. That second solve downgrades python to the only 3.14 build compatible with `libffi<3.5` — which is 3.14.0. (The stale wanda layer cache compounds this, but even a fresh rebuild today re-resolves 3.14.0 because of the pin.) Fix: replace the two-step install with a **single solve using a libffi floor** (`libffi>=3.4.6`, preserving the intent of the original pin — the 3.4.2/defaults-channel libffi was buggy). No per-version special casing; every python version resolves its newest patch release with a compatible libffi. Resolved versions today: py3.10→3.10.20 (libffi 3.7.0), py3.11→3.11.15, py3.12→3.12.13, py3.13→3.13.14 (libffi 3.5.2), py3.14→**3.14.6** (libffi 3.5.2). The Dockerfile change also busts the stale wanda cache. Note: #64772 (fiber stack-protection re-anchoring) is complementary, not a fix for this crash — its `PyUnstable_ThreadState_SetStackProtection` call only exists on 3.14.2+, so it no-ops on the 3.14.0 currently in the images. Once this lands, #64772 fixes the remaining per-task async-actor memory leak. ## Verification - Reproduced the crash: async-actor repro (`ray.get(A.remote().hi.remote())` with an `async def` method) dies in `rayproject/ray:nightly-py314-cpu` (CPython 3.14.0) with the exact failure signature from release-test job `prodjob_d4dctduzm3h6eu812vrrehiuzl`. - Verified the fix: the same unpatched nightly cp314 wheel on CPython 3.14.6 (`python:3.14-slim`) runs the repro successfully. - Verified the combined solve under miniforge 24.11.3-0 (same as the Dockerfile): dry-runs for python 3.10–3.14 all resolve (versions above), plus real installs with `ctypes` smoke tests on 3.10 (libffi 3.7.0) and 3.14.6 (libffi 3.5.2). - `python -m pytest -q release/ray_release/tests/test_config.py` — 23 passed; full collection validates (319 tests) including `hello_world_py314.aws`. ## Duplicate-work note #63237 contains an earlier version of the release-test config bundled with image-build plumbing that has since landed on master through other PRs. This PR carves out the remaining release-test config plus the base-image fix; #63237 can be closed or rebased down to the raylet fix. AI assistance (Claude Code) was used for this PR; all changes reviewed by the submitter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…mage resolving CPython 3.14.0 (ray-project#64857) ## Description ### Release smoke test Adds a `hello_world_py314` nightly smoke release test (aws variation only), mirroring the existing `hello_world_py313` entry: - `release/ray_release/schema.json` — add `"3.14"` to the `python` enum (release-test config validation rejects `python: "3.14"` without this). - `release/ray_release/config.py` — add `"3.14"` to the cpu/cu123 BYOD python allowlist (the parallel gate the release runner walks). - `release/release_tests.yaml` — new `hello_world_py314` test, nightly, `byod: {}`, same `hello_world_compute_config.yaml` as the other hello_world tests. The py3.14 `ray-anyscale` release-test images (cpu + cuda) are already built and published on master via `.buildkite/release/build.rayci.yml`, so no image plumbing is needed here. ### Base-image fix (what made the smoke test fail) The first run of this test failed with the JobSupervisor dying at job startup: `Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow`. Root cause: the py3.14 images ship **CPython 3.14.0**, which fatally crashes any Ray async actor running on a boost fiber stack (python/cpython#141944, fixed upstream in **3.14.2**). Why the images resolve 3.14.0: `docker/base-deps/Dockerfile` installs an exact `libffi=3.4.6` pin as a *separate* conda step after installing python. That second solve downgrades python to the only 3.14 build compatible with `libffi<3.5` — which is 3.14.0. (The stale wanda layer cache compounds this, but even a fresh rebuild today re-resolves 3.14.0 because of the pin.) Fix: replace the two-step install with a **single solve using a libffi floor** (`libffi>=3.4.6`, preserving the intent of the original pin — the 3.4.2/defaults-channel libffi was buggy). No per-version special casing; every python version resolves its newest patch release with a compatible libffi. Resolved versions today: py3.10→3.10.20 (libffi 3.7.0), py3.11→3.11.15, py3.12→3.12.13, py3.13→3.13.14 (libffi 3.5.2), py3.14→**3.14.6** (libffi 3.5.2). The Dockerfile change also busts the stale wanda cache. Note: ray-project#64772 (fiber stack-protection re-anchoring) is complementary, not a fix for this crash — its `PyUnstable_ThreadState_SetStackProtection` call only exists on 3.14.2+, so it no-ops on the 3.14.0 currently in the images. Once this lands, ray-project#64772 fixes the remaining per-task async-actor memory leak. ## Verification - Reproduced the crash: async-actor repro (`ray.get(A.remote().hi.remote())` with an `async def` method) dies in `rayproject/ray:nightly-py314-cpu` (CPython 3.14.0) with the exact failure signature from release-test job `prodjob_d4dctduzm3h6eu812vrrehiuzl`. - Verified the fix: the same unpatched nightly cp314 wheel on CPython 3.14.6 (`python:3.14-slim`) runs the repro successfully. - Verified the combined solve under miniforge 24.11.3-0 (same as the Dockerfile): dry-runs for python 3.10–3.14 all resolve (versions above), plus real installs with `ctypes` smoke tests on 3.10 (libffi 3.7.0) and 3.14.6 (libffi 3.5.2). - `python -m pytest -q release/ray_release/tests/test_config.py` — 23 passed; full collection validates (319 tests) including `hello_world_py314.aws`. ## Duplicate-work note ray-project#63237 contains an earlier version of the release-test config bundled with image-build plumbing that has since landed on master through other PRs. This PR carves out the remaining release-test config plus the base-image fix; ray-project#63237 can be closed or rebased down to the raylet fix. AI assistance (Claude Code) was used for this PR; all changes reviewed by the submitter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: 400Ping <jiekaichang@apache.org>

m-k-l-s · 2026-09-11T06:38:07Z

Hi, what's the status on this, please? Do I understand it correctly that ray package already supports Python 3.14, but we're just missing the docker image with py3.14?

elliot-barn added 5 commits May 1, 2026 23:26

generating py3.14 lock file

29e2035

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

allowing for 3.14 image building

279464b

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

upgrade libffi

37082a9

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

merge master

2d6c7f0

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

build py3.14 image

32c4918

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

gemini-code-assist Bot reviewed May 8, 2026

View reviewed changes

elliot-barn added 10 commits May 8, 2026 22:03

adding 3.14 release test

424d7bf

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating release schema

f0fd2f2

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

adding 3.14 builds

facbafa

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating raylet

6a4ed08

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

updating raylet

8138e8c

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

fix recursion error

4fb3449

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

adding logging

9f7587c

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

last fix

762d981

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

formatting

611c9e8

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

removing logs

5fa8eb0

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

elliot-barn marked this pull request as ready for review May 11, 2026 22:47

elliot-barn requested review from a team as code owners May 11, 2026 22:47

cursor Bot reviewed May 11, 2026

View reviewed changes

Comment thread .buildkite/build.rayci.yml

elliot-barn mentioned this pull request May 11, 2026

[Core] Fix async actor crash on CPython 3.14 fiber-stack recursion guard #63284

Closed

Merge branch 'master' into elliot-barn-314-ray-image

b7e1bed

ray-gardener Bot added core Issues that should be addressed in Ray Core devprod release-test release test labels May 12, 2026

elliot-barn added 5 commits May 12, 2026 22:23

updating raylet

ed18655

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

Merge branch 'elliot-barn-314-ray-image' of https://github.com/ray-pr…

7ecd0e2

…oject/ray into elliot-barn-314-ray-image

bumping long running actor release tests to 3.14

17a6d82

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

upgrading core tests

b4c4695

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

bumping py ver on serve tests

09be94f

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

elliot-barn added 2 commits May 14, 2026 18:27

Merge branch 'master' of https://github.com/ray-project/ray into elli…

d340c48

…ot-barn-314-ray-image

core-nightly-test python bump

bc4dc64

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label May 29, 2026

github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Jun 13, 2026

merging master

d7b2d3f

Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

cursor Bot reviewed Jun 18, 2026

View reviewed changes

elliot-barn added the go add ONLY when ready to merge, run all tests label Jun 18, 2026

elliot-barn mentioned this pull request Jul 17, 2026

[release] Add hello_world_py314 smoke release test; fix py3.14 base image resolving CPython 3.14.0 #64857

Merged

elliot-barn closed this Oct 20, 2026

Navigation Menu

Repository navigation

[ci][docker] build py3.14 ray image - #63237

[ci][docker] build py3.14 ray image#63237
elliot-barn wants to merge 24 commits into
masterfrom
elliot-barn-314-ray-image

elliot-barn commented May 8, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 8, 2026

Uh oh!

gemini-code-assist Bot May 8, 2026

Uh oh!

gemini-code-assist Bot May 8, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

m-k-l-s commented Jun 12, 2026

Uh oh!

elliot-barn commented Jun 18, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Jun 18, 2026

Uh oh!

m-k-l-s commented Sep 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

elliot-barn commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Buildkite image build pipeline

Dependency locks (raydepsets)

Wheel builder + release-test config

Release smoke test

Misc

Verification

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

m-k-l-s commented Jun 12, 2026

Uh oh!

elliot-barn commented Jun 18, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 18, 2026

Choose a reason for hiding this comment

CUDA 13 py3.14 CI mismatch

Uh oh!

m-k-l-s commented Sep 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elliot-barn commented May 8, 2026 •

edited

Loading