Repository navigation
Build wheels in CI and publish them #164
Description
Activity
Scikit-build recently has teamed up with cibuildwheel. Maybe this is a viable approach to get wheels which work in a standard pip install.
cibuildwheel still has no auditwheel equivalent for Windows.
I don't know if this is helpful info for you, but
cibuildwheelnow seems to recommend delvewheel as a Windows equivalent forauditwheel. They show an example of how to use it in their documentation here.Reacted by Ben GreinerFor an overview of the problems involved, there's this, published late 2022: https://pypackaging-native.github.io/ . See especially https://pypackaging-native.github.io/key-issues/native-dependencies/blas_openmp/
That references this https://web.archive.org/web/20210616203153/https://pav.iki.fi/blog/2017-10-08/pywingfortran.html (archive link, original down). It looks like this needs
numpy.distutils, but I don't think scikit-build uses that (maybe it does?). Maybescikit-buildalso has the statically-link-gfortran-runtime magicI had another go at this over the Christmas break and go nowhere, though my effort was admittedly half-hearted.
Reacted by Steven DahdahNice find!
numpy.distutilsis deprecated and will be removed in a future version.scikit-buildis in the middle of a major overhaul.SciPy moved to Meson recently. Maybe an option for us is to switch from
scikit-build/CMaketomeson-pythonas well and build wheels withcibuildwheelas SciPy does.Reacted by Steven DahdahI've had a another go at this, with a tiny model of Slycot: a Python package with Fortran 77 source calling a LAPACK routine. It uses scikit-build-core and scipy_openblas32, and so far I've managed to build a manylinux wheel with a vendored OpenBLAS library. Repo is https://github.com/roryyorke/f77ex, you can find the Github Action-generated wheel at https://github.com/roryyorke/f77ex/releases/tag/v0.0.1.
This uses auditwheel; there's a higher-level package repairwheel that invokes the right OS tool (Linux, Windows, macos).
As of now I see two known issues, namely cross-platform build, and connecting "normal" and scipy_openblas LAPACK/BLAS symbols.
Any comments, suggestions, warnings, or patches welcome.
Cross-platform
Maybe cibuildwheel and scikit-build-core take care of all this, though long experience suggests Fortran is a little-used corner-case.
scipy_openblas symbols
In the build I've added compile-time options
-cpp -DDGETRF=SCIPY_DGETRF(see CMakeLists.txt file), so that LAPACK functionDGETRFbecomesSCIPY_DGETRF. The other option is to useobjcopyto do the same thing to object files -- I don't know what the trade-offs are between these two options. Either way, for Slycot this would need a long list of such mappings. The mappings would need to be kept up-to-date, but SLICOT, LAPACK, and BLAS are hopefully slow-enough moving for this to not to be a major problem.Based on the pkg-config file distributed with scipy_openblas, I think for C code linking against this library there are other facilities to sort this out, but I don't think they apply for Fortran 77 (maybe Fortran in general?). I'm going to ask about this on the scipy_openblas repo.
Next steps
I'm going to look at cibuildwheel and Windows builds next; if that works, I'll make a list of LAPACK mappings for the preprocessor.
I've had some success with this; see MacPython/openblas-libs#254 (comment), which has links to two prototypes of manylinux and Windows Python 3.13 wheels that vendor the openblas libraries.
Three substantial tasks remain:
- getting the repairwheel process to work on macos; I'll see what numpy and scipy do here
- making this whole process configurable, as in, retain support for linking any (supported!) blas/lapack. We'd only bundle Openblas for binary wheels published to PyPI.
- getting the scikit-build-core approach, which I've built all this on, to work in conda-forge
I've produced wheels for Python 3.13 for macOS ARM, Windows x86_64 and Linux x864_64; see https://github.com/roryyorke/Slycot/actions/runs/23395663910 , wheels can be downloaded at the bottom of that page. I suspect GitHub only keeps those artefacts for 30 days or so.
I'm going to prepare a PR based on that branch.
Reacted by Juhan Oskar Hennoste, Steven Dahdah, Ben Greiner and Ilhan PolatWhat's blocking me on getting this done is Windows conda package build. This is not specific to the branch: master also doesn't build Windows conda packages anymore.
If I wholesale copy what conda-forge/slycot-feedstock does, the new (scikit-build-core) packages build OK on Windows. I've taken a look at the feedstock scripts: there's a lot of environment setup (e.g., removing entries from PATH) which I could try to copy into our CI. I couldn't get Linux and macOS conda packages to build with the feedstock recipes, but I didn't try too hard: I'm pretty sure they could be made to work, but they are different enough from the Slycot CI to be a pain.
@bnavigator what do you think about dropping the conda build step from this repo's CI? My take is this repo should focus on providing a package that can be built (1) with user-defined C and Fortran compilers, and user-defined BLAS/LAPACK; (2) for all supported Python versions, and supported platforms, and (3) as a self-contained wheel. Over at slycot-feedstock we can worry about conda-forge builds.
For python-control itself we can build Slycot from source (or build and install OpenBLAS-bundled wheels) instead of using conda.
We'd lose the matrix-of-LAPACK-provider checks. Is that a major problem?
I'll look at copying the Windows-special-code into our CI.
Reacted by Ben GreinerI fully support your proposal. Conda quirks should be dealt with on the conda feedstock.
The matrix of LAPACK-providers would still be available for python builds on unix-like systems: Your "(1)". Windows users should be more than happy to have wheels and conda-forge builds.
Replying to myself first:
I'll look at copying the Windows-special-code into our CI.
I didn't get this right; I could at least convince the build to use Ninja, but it was still in a "visual studio mode" of some sort that didn't support flang.
Replying to @bnavigator :
I fully support your proposal. Conda quirks should be dealt with on the conda feedstock.
OK, that's great. If a future Slycot release hampers the feedstock build, we can patch it in the feedstock, or fix and do a new release in Slycot.
The matrix of LAPACK-providers would still be available for python builds on unix-like systems
Should we test at least a few LAPACK-providers in the CI then? LAPACK-reference, OpenBLAS, MKL? (These seem to be available in Debian, I assume Ubuntu too).
I propose this for CI
- ruff, other lint
- sdist; use this sdist for all wheel builds (I hope this isn't easier said than done)
- check editable install works for 1 Python version and 1 LAPACK provider (LAPACK-reference?) on ubuntu-latest
- check "normal" (non-editable, non-wheel) install with 1 Python version and as many LAPACK providers as can easily be installed on Github ubuntu-latest
- build wheels for supported platforms and Python versions (this is a largish matrix). This uses cibuildwheel and various Github runner images.
For the '1 Python version' we choose the ... oldest supported? Newest supported? Doesn't matter?
We should certainly run Slycot's own tests for all of the builds; they barely register time-wise compared to the compile times. We could also run the python-control Slycot tests; there's a bit of overhead installing python-control and running the tests, but again probably not too much compared to the build time. This would be python-control from main branch, as we have it now.
Besides this we'd need to update python-control's CI to account for the change to Slycot build, though maybe only os-blas-test-matrix.yml , which was last run Jun 2025.
Besides this we'd need to update python-control's CI to account for the change to Slycot build, though maybe only os-blas-test-matrix.yml , which was last run Jun 2025.
For python-control, I think the CI runs just need to run on linux. If I remember correctly, we do one run there where we build slycot from scratch (using
pip install -v ., mainly to make sure that the main branch of slycot doesn't have any issues. Everything else uses whatever it finds via mamba.The os-blas-test-matrix job is run prior to release, to make sure that various combinations of different dependencies work. It would be good to preserve that functionality as much as possible, just because it lets us check for failing combinations ourselves, rather than having users report them.
pip install -v .
This will still work.
The os-blas-test-matrix job is run prior to release, to make sure that various combinations of different dependencies work. It would be good to preserve that functionality as much as possible, just because it lets us check for failing combinations ourselves, rather than having users report them.
This builds Slycot using conda-build, see os-blas-test-matrix.yml:101. I suspect this workflow will fail now: I've just triggered a CI run of Slycot master at https://github.com/roryyorke/Slycot/actions/runs/24635001396, and at https://github.com/roryyorke/Slycot/actions/runs/24635001396/job/72028828657 you can see the Windows build fails, with messages like
-- The Fortran compiler identification is Flang 99.99.1 -- Detecting Fortran compiler ABI info CMake Error in C:/Users/runneradmin/miniconda3/conda-bld/slycot_1776620232430/work/_cmake_test_compile/build/CMakeFiles/CMakeScratch/TryCompile-0r40wy/CMakeLists.txt: MSVC_DEBUG_INFORMATION_FORMAT value 'ProgramDatabase' not known for this Fortran compiler.We can ask on the conda-forge Zulip channel if there are any suggestions for this.
I figured out how to get our conda-recipe to work on the 3 main platoforms; PR is in (see above).
We don't publish wheels because we don't know how to build them.
I've tried and failed to build wheels for Windows before; if I recall correctly one basically has to bundle a BLAS&LAPACK library in the wheel, and I couldn't figure out how to do it.
Originally posted by @roryyorke in #155 (comment)