Pythonx

Python interpreter embedded in Elixir.

Pythonx runs a Python interpreter in the same OS process as your Elixir application, allowing you to evaluate Python code and conveniently convert between Python and Elixir data structures.

The goal of this project is to better integrate Python workflows within Livebook and its usage in actual projects must be done with care due to Python's global interpreter lock (GIL), which prevents multiple threads from executing Python code at the same time. Consequently, calling Pythonx from multiple Elixir processes does not provide the concurrency you might expect and thus it can be a source of bottlenecks. However, this concerns regular Python code. Packages with CPU-intense functionality, such as numpy, have native implementation of many functions and invoking those releases the GIL. GIL is also released when waiting on I/O operations. In other words, if you are using this library to integrate with Python, make sure it happens in a single Elixir process or that its underlying libraries can deal with concurrent invocation. Otherwise, prefer to use Elixir's System.cmd/3 or Ports to manage multiple Python programs via I/O.

Add Pythonx to your dependencies:

Mix.install([
  {:pythonx, "~> 0.4.0"}
])

Initialize the interpreter, specifying the desired Python version and dependencies:

Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.2"
]
""")

Evaluate Python code:

{result, globals} =
  Pythonx.eval(
    """
    y = 10
    x + y
    """,
    %{"x" => 1}
  )

Pythonx.decode(result)
#=> 11

globals
#=> %{
#=>   "x" => #Pythonx.Object<
#=>     1
#=>   >,
#=>   "y" => #Pythonx.Object<
#=>     10
#=>   >
#=> }

In a dynamic evaluation environment, such as IEx and Livebook, you can also use the ~PY sigil:

import Pythonx

x = 1

~PY"""
y = 10
result = x + y
"""

result
#=> #Pythonx.Object<
#=>   11
#=> >

y
#=> #Pythonx.Object<
#=>   10
#=> >

Add Pythonx to your dependencies:

def deps do
  [
    {:pythonx, "~> 0.4.0"}
  ]
end

Configure the desired Python version and dependencies in your config/config.exs:

import Config

config :pythonx, :uv_init,
  pyproject_toml: """
  [project]
  name = "project"
  version = "0.0.0"
  requires-python = "==3.13.*"
  dependencies = [
    "numpy==2.2.2"
  ]
  """

Additionally, you can configure a specific version of the uv package manager for Pythonx to use. This can impact the available Python versions.

import Config

config :pythonx, :uv_init,
  ...,
  uv_version: "0.7.21"

With that, you can use Pythonx.eval/2 and other APIs in your application. The downloads will happen at compile time, and the interpreter will get initialized automatically on boot. All necessary files are placed in Pythonx priv directory, so it is compatible with Elixir releases.

Note that currently the ~PY sigil does not work as part of Mix project code. This limitation is intentional, since in actual applications it is preferable to manage the Python globals explicitly.

When a normal Python executable terminates, it runs a sophisticated shutdown sequence called "finalization". This is what triggers most of the cleanup behavior you might expect. This includes things like as object destructors, atexit handlers and module finalizers.

Without finalizing, many libraries can misbehave subtly--including the Python standard library!

Some real examples observed include:

• warnings about leaked multiprocessing semaphores
• log messages not actually emitted
• readline history not written
• files created with tempfile aren't deleted
• changes to a shelve database aren't persisted
• files and sockets aren't closed
• context managers in generators may not call their exit hooks

When running under a full OTP supervision tree (i.e. in most apps + IEx), Pythonx automatically finalizes the interpreter as part of a graceful shutdown.

Some environments don't run a full supervision tree. Typically that will be things like exscripts and Mix tasks. In these situations, you can also call Pythonx.finalize/0 manually.

After finalization, the interpreter can be re-initialized with Pythonx.uv_init/2 or Pythonx.init/4. Objects from the previous interpreter session are detected via an internal generation counter and rejected with a clear error message.

Re-initialization is reliable for pure-Python code, but C extensions with process-global state (numpy, torch) may fail or crash on their second import; see the "Re-initialization limits" section in the Pythonx.finalize/0 docs.

Pythonx provides a Python module named pythonx with extra interoperability features.

Sends a Python object to an Elixir process identified by pid.

The Elixir process receives the message as a {tag, object} tuple, where tag is an atom and object is a Pythonx.Object struct.

Long-running evaluation {: .warning}

If you are sending messages from Python to Elixir, it likely means you have a long-running Python evaluation. If the evaluation holds onto GIL for long, you should make sure to only do it from a single Elixir process to avoid bottlenecks. For more details see the "Concurrency" notes in Pythonx.eval/3.

Decoding {: .warning}

The Elixir process receives a Pythonx.Object, which you may want to decode right away. Keep in mind that Pythonx.decode/1 requires GIL, so if the ongoing evaluation holds onto GIL for long, decoding itself may be blocked.

Parameters:

• pid (pythonx.PID) – Opaque PID object, passed into the evaluation.
• tag (str) – A tag appearning as atom in the Elixir message.
• object (Any) – Any Python object to be sent as the message.

Opaque Python object that represents an Elixir PID.

This object cannot be created within Python, it needs to be passed into the evaluation as part of globals.

CPython (the reference implementation of the Python programming language) provides a python executable that runs Python code, and that is the usual interface that developers use to interact with the interpreter. However, most the CPython functionality is also available as a dynamically linked library (.so, .dylib or .dll, depending on the platform). The python executable can be thought of as a program build on top of that library.

With this design, any C/C++ application can link the Python library and use its API to execute Python code and interact with Python objects on a low level. Taking this a step further, any language with C/C++ interoperability can interact with Python in the same manner. This usage of CPython is referred to as embedding Python.

Elixir provides C/C++ interoperability via Erlang NIFs and that is exactly how Pythonx embeds Python. As a result, the Python interpreter operates in the same OS process as the BEAM.

For more details refer to the official documentation on embedding Python.

Thank you to Cocoa Xu (@cocoa-xu) for building the first prototype of embedded Python (source).

Copyright (c) 2025 Dashbit

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.