Skip to content

Use RustPython to eliminate Python dependency for downloading HuggingFace datasets #2093

Description

@antimora

Currently, Burn calls the Python interpreter to download datasets from HuggingFace, which requires users to have Python installed on their system. This introduces an external dependency that may not be desirable for all users and environments.

Proposal:
We could leverage RustPython, a Python interpreter implemented in Rust, to eliminate the need for a separate Python installation. This would allow us to run the necessary HuggingFace dataset scripts within our Rust environment.

Benefits:

  1. Removes the external Python dependency, making Burn more self-contained.
  2. Potentially simplifies the setup process for users.
  3. Could lead to better integration and control over the dataset downloading process.

Challenges:

  1. Need to ensure RustPython can handle all required HuggingFace dataset operations.
  2. May require adaptation of existing dataset loading code.
  3. Potential performance implications compared to CPython.

Next steps:

  1. Investigate RustPython's compatibility with HuggingFace datasets scripts.
  2. Create a proof-of-concept implementation using RustPython.
  3. Benchmark performance compared to the current approach.
  4. Discuss potential implications on build process and dependencies.

This change would enhance Burn's portability and reduce external dependencies, potentially making it easier for users to get started with the framework.

Posted a question on usage on RustPython: RustPython/RustPython#5379

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementEnhance existing features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions