Skip to content

CI: add cp314/cp314t nighly wheel builds - #23187

Merged
lucascolley merged 4 commits into
scipy:mainfrom
rgommers:py314
Jun 20, 2025
Merged

CI: add cp314/cp314t nighly wheel builds#23187
lucascolley merged 4 commits into
scipy:mainfrom
rgommers:py314

Conversation

@rgommers

@rgommers rgommers commented Jun 19, 2025

Copy link
Copy Markdown
Member

We're about 5-6 weeks away from needing to do a release with Python 3.14 support (see the release schedule in PEP 745), so it'll be good to enable nightly wheels now and fix/report issues.

There are a couple of tests that needed skipping - issues filed for those: gh-23172, gh-23185. Also, there is one wheel (cp314t-musllinux_1_2_aarch64) for which the test suite is hanging. The hang is the most serious problem and needs investigating next. That wheel is probably the least critical of all wheels though, so it's best to get the other ones deployed now.

We shouldn't backport this PR just yet; once 3.14.0rc1 is out and numpy has done a release that supports it, this will be significantly simpler to support on the maintenance/1.16.x branch.

@rgommers rgommers added this to the 1.17.0 milestone Jun 19, 2025
@rgommers rgommers added Official binaries Items related to the official SciPy binaries, including wheels and vendored libs CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure labels Jun 19, 2025
@rgommers rgommers added the free-threading Items related to supporting free-threaded (a.k.a. "no-GIL") builds of CPython label Jun 19, 2025
@rgommers

rgommers commented Jun 19, 2025

Copy link
Copy Markdown
Member Author

Hmm, the one wheel build failure here, for cp314t-musllinux x86_64, looks like:

   [591/1474] Generating 'scipy/stats/_qmvnt_cy.cpython-314t-x86_64-linux-musl.so.p/_qmvnt_cy.cpp'
  [592/1474] Generating 'scipy/stats/_stats_pythran.cpython-314t-x86_64-linux-musl.so.p/_stats_pythran.cpp'
  FAILED: scipy/stats/_stats_pythran.cpython-314t-x86_64-linux-musl.so.p/_stats_pythran.cpp 
  /opt/python/cp314-cp314t/bin/pythran -E ../scipy/stats/_stats_pythran.py -o scipy/stats/_stats_pythran.cpython-314t-x86_64-linux-musl.so.p/_stats_pythran.cpp
  [593/1474] Compiling C object scipy/stats/_levy_stable/levyst.cpython-314t-x86_64-linux-musl.so.p/meson-generated_levyst.c.o
  [594/1474] Compiling C++ object scipy/stats/_qmvnt_cy.cpython-314t-x86_64-linux-musl.so.p/meson-generated__qmvnt_cy.cpp.o
  [595/1474] Compiling C object scipy/stats/_stats.cpython-314t-x86_64-linux-musl.so.p/meson-generated__stats.c.o
  [596/1474] Compiling C++ object scipy/sparse/sparsetools/_sparsetools.cpython-314t-x86_64-linux-musl.so.p/csr.cxx.o
  [597/1474] Compiling C++ object scipy/sparse/sparsetools/_sparsetools.cpython-314t-x86_64-linux-musl.so.p/bsr.cxx.o
  ninja: build stopped: subcommand failed.
  
  ERROR Backend subprocess exited when trying to invoke build_wheel

The exact same job passed on my fork an hour earlier. All the relevant packages seem to be identical:

  Successfully installed beniget-0.4.2.post1 cython-3.1.2 gast-0.6.0 meson-1.8.2 meson-python-0.18.0 ninja-1.11.1.4 ply-3.11 pybind11-2.13.6 pyproject-metadata-0.9.1 pythran-0.18.0 setuptools-80.9.0

Normally pythran gives a sensible error message. Perhaps it ran out of memory, given that there's multiple heavy C++ compilations right before and after it? I'll rerun to see if it fails again.

@rgommers

rgommers commented Jun 19, 2025

Copy link
Copy Markdown
Member Author

Failed again at the same point - back to debugging on fork.

@rgommers

rgommers commented Jun 19, 2025

Copy link
Copy Markdown
Member Author

Re-ran it for a third time with debug logging enabled. It failed even earlier, on a different pythran invocation:

  [156/1474] Generating 'scipy/linalg/_linalg_pythran.cpython-314t-x86_64-linux-musl.so.p/_linalg_pythran.cpp'
  FAILED: scipy/linalg/_linalg_pythran.cpython-314t-x86_64-linux-musl.so.p/_linalg_pythran.cpp 
  /opt/python/cp314-cp314t/bin/pythran -E ../scipy/linalg/_linalg_pythran.py -o scipy/linalg/_linalg_pythran.cpython-314t-x86_64-linux-musl.so.p/_linalg_pythran.cpp

Re-ran again as well on my own fork, and the build passed: CI log. This is very mysterious.

@rgommers
CI: exclude musllinux + cp314t wheel builds [wheel build]
6088b6d 
Follow up on this later, likely one or more issues in CPython, NumPy,
and/or Pythran.

The aarch64 job was hanging at various places in the test suite.
The x86-64 job was failing on `pythran` generation .cpp code at build
time, repeatedly and on multiple source files.
@rgommers

rgommers commented Jun 19, 2025

Copy link
Copy Markdown
Member Author

Skipping the second musllinux + cp314t job as well for now.

@rgommers

rgommers commented Jun 20, 2025

Copy link
Copy Markdown
Member Author

This is good to merge now. The one failure in a non-wheel job is unrelated.

lucascolley
lucascolley approved these changes Jun 20, 2025
View reviewed changes

@lucascolley lucascolley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks Ralf!

@lucascolley
lucascolley merged commit be834e8 into scipy:main Jun 20, 2025
@rgommers
rgommers deleted the py314 branch June 20, 2025 16:56
@rgommers rgommers mentioned this pull request Jun 20, 2025
Merged
This was referenced Jun 20, 2025
Open
Closed
@rgommers rgommers mentioned this pull request Jul 24, 2025
Merged
rgommers added a commit to rgommers/scipy that referenced this pull request Jul 25, 2025
@rgommers
CI: disable special-casing of 3.14-dev now that Python 3.14rc1 is out
91cf58e 
This reverts changes made in scipy#23187, keeping the commented out code
for once 3.15-dev arrives.
rgommers added a commit to rgommers/scipy that referenced this pull request Jul 25, 2025
@rgommers
CI: disable special-casing of 3.14-dev now that Python 3.14rc1 is out
6dab6a9 
This reverts changes made in scipy#23187, keeping the commented out code
for once 3.15-dev arrives.

[ci skip]
@rgommers rgommers mentioned this pull request Jul 25, 2025
Merged
@rgommers rgommers added the backport-candidate This fix should be ported by a maintainer to previous SciPy versions. label Jul 25, 2025
@tylerjereddy tylerjereddy removed this from the 1.17.0 milestone Jul 26, 2025
@tylerjereddy tylerjereddy added this to the 1.16.1 milestone Jul 26, 2025
tylerjereddy pushed a commit to tylerjereddy/scipy that referenced this pull request Jul 26, 2025
@rgommers @tylerjereddy
CI: disable special-casing of 3.14-dev now that Python 3.14rc1 is out
ea54bed 
This reverts changes made in scipy#23187, keeping the commented out code
for once 3.15-dev arrives.

[ci skip]
@tylerjereddy tylerjereddy mentioned this pull request Jul 26, 2025
Merged
5 tasks done
@tylerjereddy tylerjereddy removed the backport-candidate This fix should be ported by a maintainer to previous SciPy versions. label Jul 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucascolley lucascolley lucascolley approved these changes
@larsoner larsoner Awaiting requested review from larsoner larsoner is a code owner
@andyfaff andyfaff Awaiting requested review from andyfaff andyfaff is a code owner

Assignees

No one assigned

Labels

CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure free-threading Items related to supporting free-threaded (a.k.a. "no-GIL") builds of CPython Official binaries Items related to the official SciPy binaries, including wheels and vendored libs scipy.io scipy.stats

Projects

None yet

Milestone

1.16.1

Development

Successfully merging this pull request may close these issues.

3 participants

@rgommers @lucascolley @tylerjereddy