Skip to content

Consider moving CPython patches upstream #635

Closed
Closed
Consider moving CPython patches upstream#635

Description

@SimonBiggs
SimonBiggs
opened on Apr 14, 2020

Hi @mdboom,

Not sure what your thoughts are on this. I was attempting to bump some version numbers within the pyodide source with the hope that doing so may fix another issue (as described over at #441 (comment)).

Digging into things I realised just how difficult it is to update pyodide's dependencies given its strong reliance on patching very specific versions. My thought was, in the long term, don't we want Web Assembly + CPython to be a first party supported build target by the upstream libraries and tools? To get this rolling I created the following issue within the CPython bug tracker, not sure if that's the best way to go about things:

https://bugs.python.org/msg366369

I imagine it would look like having certain parts of Python switched off if the compilation target was Web Assembly with Emscripten (just as you have done with your patches). The patches applied upstream would just need to not interfere with normal building.

Keen for your thoughts Michael,
Cheers,
Simon

Activity

  1. mdboom commented on Apr 14, 2020

    @mdboom
    mdboom
    on Apr 14, 2020
    Collaborator

    Thanks for setting this up. Pyodide has largely stalled (for me, at least), as it's no longer a day job thing. But I wholeheartedly support this effort to upstream as much as we can.

    (The latest example of these problems is actually that a 3.7 point release updated a bunch of tests to use threading, which hadn't previously. Since pyodide doesn't support threading, that's a major lift that hasn't even been done...)

  2. pmp-p commented on Apr 14, 2020

    @pmp-p
    pmp-p
    on Apr 14, 2020
    Contributor

    Hi mdboom, SimonBiggs

    @SimonBiggs, sorry for cross posting but what i have to say is more for pyodide side of things.
    You said on BPO :

    With respect to blocking when running Python as WASM, I have found running the WebAssembly CPython within a webworker and signalling data back and
    forth causes there to be no UI issues.

    that sounds very promising since it would not need to change python code to run on wasm and worker loading is faster than on main thread.
    But i guess some RPC and fast serialization is needed there (see https://brionv.com/log/wp-content/uploads/2019/05/Wasm-RPC-thoughts.pdf and rust way) is there a consensus on that ?

    my thoughts :

    i guess a pip module settings things up and providing a standard communication channel for everyone to use could justify fixing the few build problems without "supporting a new platform".
    from my experience it would need :

    • a site.py patching the theading and some corner cases in stdlib
    • a tool to package source code ( or use some android tools )
    • a module for asynchronous RPC python side and javascript side.
    • access to native input, UI and 3D ( i think emscripten started to bridge that for workers but i don't know status yet).

    It would sound a lot like the beeware - briefcase pipeline work ( thinking of wasm/wasi as another iOS or android special case ) https://beeware.org/project/projects/tools/briefcase/

    My personnal view is there's no point in pushing patches if no one can discover/test/use them
    i mean by that : get Python sources, a compiler, read some docs (1 page maybe 2) and pip install some helper module to get something running. Nothing more.

    i like pyodide but it's too complicated to get on rails with it. Though i'm quite sure it can run Panda3D http://rdb.name/panda3d-webgl/editor.html or run some Brython samples https://brython.info/demo.html?lang=en

    Side question : on the long term are you sure you got the right Python for the job ?

  3. rth commented on Apr 14, 2020

    @rth
    rth
    on Apr 14, 2020
    Member

    Upstreaming patches in CPython would be nice, but there isn't that many of those and most are about skipping imports (e.g. threading for now). I imagine if we wanted to do it, we would need first to update to python 3.8 to be closer to upstream.

    The issue is that we are currently locked on Python 3.7 by a somewhat old version of scipy (that's working partially), possibly among other reasons. And updating scipy would likely be better with latest emscripten with upstream LLVM #476 . There were some attempts but they are time consuming, and someone would need to be quite motivated to do it.

    i like pyodide but it's too complicated to get on rails with it.

    Yes, how to simplify it is certainly a relevant question for long term maintainability of the project. Using latest emscripten and hopefully avoiding to build a patched LLVM would certainly be one step in the right direction.

  4. SimonBiggs commented on Apr 14, 2020

    @SimonBiggs
    SimonBiggs
    on Apr 14, 2020
    ContributorAuthor

    The issue is that we are currently locked on Python 3.7 by a somewhat old version of scipy (that's working partially)

    Due to the currently large file size of scipy, I personally drop it and all that depends on it from my pyodide installation anyway. If it was viable I would be in favour of temporarily dropping SciPy in favour of improving the maintainability of pyodide in the short term.

  5. SimonBiggs commented on Apr 14, 2020

    @SimonBiggs
    SimonBiggs
    on Apr 14, 2020
    ContributorAuthor

    But i guess some RPC and fast serialization is needed there (see brionv.com/log/wp-content/uploads/2019/05/Wasm-RPC-thoughts.pdf and rust way) is there a consensus on that ?

    I think Jupyter kernel messaging would be a brilliant standard to build on here. An ideal end point would be running JupyterLab with a Python WASM kernel running within a web worker. See https://github.com/deathbeds/jyve for a proof of concept.

  6. eulertour commented on Apr 15, 2020

    @eulertour
    eulertour
    on Apr 15, 2020
    Contributor

    As someone who would hate to see the work on this project go to waste, I'd suggest that it's much more important to have pyodide as a supported upstream build target than it is for pyodide to support scipy, especially since its load time makes for a lacking user experience anyway.

    And updating scipy would likely be better with latest emscripten with upstream LLVM #476 . There were some attempts but they are time consuming, and someone would need to be quite motivated to do it.

    If finding a way to use the latest emscripten and upstream LLVM would motivate the current maintainers to pursue this I'd be more than happy to dedicate my time to it. I'd also be willing to try to accommodate the list of requirements @pmp-p described if I can get some pointers with the parts I don't know as well, or just generally to simplify pyodide's build and installation processes.

    It'd be an enormous waste if this project simply fell too far behind upstream and the next team was left to start from scratch. Getting the major parts moved upstream would provide some satisfying closure.

  7. rth commented on Apr 15, 2020

    @rth
    rth
    on Apr 15, 2020
    Member

    I'd suggest that it's much more important to have pyodide as a supported upstream build target than it is for pyodide to support scipy, especially since its load time makes for a lacking user experience anyway.

    Yes, I agree that the currently scipy experience is not great (due LAPACK size #240) and because it's so large it also takes a while to build, making iteration speed slows as well. Currently scikit-learn, and statsmodels depend on them. I agree we should have an option to build without those. I was imprecise, officially scipy 0.17 only supports python 3.5 but since python is mostly backward compatible it works (maybe with a few deprecation warnings) for python 3.7. So the same should apply to 3.8. I'll try to update the scipy version.

    Anyway, even assuming we update to CPython master and get all cpython/patches/ upstreamed (again that are fairly small), that's not going to solve the long term maintenance issue IMO:

    • there is ~1k LoC Python in pyodide_build/ with all the package build mechanism. That's likely remain as a separate package, which is fine.
    • there is ~300 LoC of Makefiles to setup the environment
    • and then various patches for different projects (including emsdk at present).

    How to go from the current situation to one where WASM is indeed a CPython build target and once installed, one could do say python-wasm -m pip install numpy (as one can currently do for say for PyPy) to get the corresponding packages built is not entirely clear to me. For one thing because we cannot build C-extensions inside WASM VM, so one still has to have a plain python to build packages.

    So maybe python -m pyodide_build install numpy (without being called from inside of the pyodide Makefile) is more realistic. But then for this second use case, would wasm becoming an officially supported CPython immediately help? Yes, it would be possible to distribute CPython builds for WASM, but because one wouldn't be able to use it to install other packages with C extensions it would be of limited use in the context of pyodide.

    Maybe just CPython build with pyodide + micropip to pull packages with pure python wheels is a valid use case, in particular to allow iterating faster with the build.

    For the rest of the packages in pyodide we need to be able to use some package repository (e.g. PyPi or conda-forge) to avoid re-implementing our own dependency resolution etc. However, even assuming they support wasm target one day, they have no way re-build all the repository in one go which would be necessary with the quickly moving emscripten and other wasm related tools.

    So it's a challenge. I have not been following closely lately, maybe other projects have been working on some of it.

  8. eulertour commented on Apr 16, 2020

    @eulertour
    eulertour
    on Apr 16, 2020
    Contributor

    @rth If the end goal were to have wasm as an upstream build target for CPython together with the ability to install pure-Python packages via micropip, is the only problem to solve the fact that pyodide is currently built using old and patched versions of emsdk and LLVM?

    Not to say that it isn't unsatisfying for the packages with C extensions to be maintained separately, but at least in regard to the goal of bringing Python to the web I'd consider it to be an important milestone.

  9. pmp-p commented on Apr 19, 2020

    @pmp-p
    pmp-p
    on Apr 19, 2020
    Contributor

    at least in regard to the goal of bringing Python to the web I'd consider it to be an important milestone.

    yes and no, i feel more like "bringing CPython to pyodide", as my question suggested "are you sure you got the right Python for the job ?" : there are other Python flavour available on wasm and maybe more suited "for the web" (size/faster startup/vm preemption/easier codebase ...).

    if that can help anybody to bring pyodide to 3.8, i've setup a CI for both android and webassembly targeting CPython 3.8.2here https://github.com/pmp-p/pydk and relevant wasm patches are here https://github.com/pmp-p/pydk/tree/master/sources.wasm/Python-3.8.2

  10. rth commented on May 9, 2020

    @rth
    rth
    on May 9, 2020
    Member

    there are other Python flavour available on wasm and maybe more suited "for the web" (size/faster startup/vm preemption/easier codebase ...).

    I have opened #646 with a list of ideas we could implement to reduce the size and load time of the main pyodide package in a minimal configuration.

    Also #645 should allow optionally skipping the build of all larger scientific packages (and CLAPACK).

  11. westurner commented on Sep 26, 2020

    @westurner
    westurner
    on Sep 26, 2020

    re: conda-forge WASM build target, CPython WASM patches, pyodide conda package

    Here's the conda-forge python-feedstock meta.yml:

    Here's the patch that added support for [arm64] builds to conda-forge:

    Additional resources:

  12. rth commented on Feb 12, 2021

    @rth
    rth
    on Feb 12, 2021
    Member

    I think we can close this. We are trying to upstream all patches that we can, in particular @dalcde has been doing a lot it for emscripten. It's a continuous process, so there is probably no need to keep a separate issue about it.

    For the integration with conda, this should probably be discussed in a dedicated issue. There is #795

  13. rth
    closed this as completedon Feb 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions