Currently, Burn calls the Python interpreter to download datasets from HuggingFace, which requires users to have Python installed on their system. This introduces an external dependency that may not be desirable for all users and environments.
Proposal:
We could leverage RustPython, a Python interpreter implemented in Rust, to eliminate the need for a separate Python installation. This would allow us to run the necessary HuggingFace dataset scripts within our Rust environment.
Benefits:
- Removes the external Python dependency, making Burn more self-contained.
- Potentially simplifies the setup process for users.
- Could lead to better integration and control over the dataset downloading process.
Challenges:
- Need to ensure RustPython can handle all required HuggingFace dataset operations.
- May require adaptation of existing dataset loading code.
- Potential performance implications compared to CPython.
Next steps:
- Investigate RustPython's compatibility with HuggingFace datasets scripts.
- Create a proof-of-concept implementation using RustPython.
- Benchmark performance compared to the current approach.
- Discuss potential implications on build process and dependencies.
This change would enhance Burn's portability and reduce external dependencies, potentially making it easier for users to get started with the framework.
Posted a question on usage on RustPython: RustPython/RustPython#5379
Currently, Burn calls the Python interpreter to download datasets from HuggingFace, which requires users to have Python installed on their system. This introduces an external dependency that may not be desirable for all users and environments.
Proposal:
We could leverage RustPython, a Python interpreter implemented in Rust, to eliminate the need for a separate Python installation. This would allow us to run the necessary HuggingFace dataset scripts within our Rust environment.
Benefits:
Challenges:
Next steps:
This change would enhance Burn's portability and reduce external dependencies, potentially making it easier for users to get started with the framework.
Posted a question on usage on RustPython: RustPython/RustPython#5379