zarr-sqlite-python

Store Zarr datasets in a single SQLite database.

zarr-sqlite is a Python library which provides SQLiteStore, a single-file store implementation backed by SQLite for zarr-python. It combines Zarr's chunked, hierarchical data model with SQLite's single-file database format, mutability and ACID guarantees.

• Single file — complete Zarr hierarchy is stored in one .zarrdb file.
• ACID transactions — writes are backed by SQLite's transactional database engine. A crash or power loss during a write cannot corrupt the entire file, at most, a single chunk may be lost. Readers can safely access the store concurrently with writers and are guaranteed to see a consistent view of the data.
• Mutable storage — arrays can be modified, overwritten, resized, appended to and deleted.

SQLiteStore is primarily intended for relatively small Zarr datasets (typically up to a few gigabytes) where the convenience of a single file is useful for archival, distribution, or sharing. It is not intended as an alternative to cloud-based object storage designed for chunk-level access to multi-terabyte datasets.

Install from PyPI using your favorite package manager, for example:

pip install zarr-sqlite

uv add zarr-sqlite

import zarr

from zarr_sqlite import SQLiteStore

with SQLiteStore("my_zarr_file.zarrdb") as store:
    root = zarr.create_group(store=store)
    foo = root.create_group("foo")
    bar = foo.create_group("bar")
    z1 = bar.create_array(name="baz", shape=(10000, 10000), chunks=(1000, 1000), dtype="int32")
    z1[:] = 42

SQLiteStore otherwise behaves identically to other stores used with zarr. See the zarr-python user guide for more information.

There isn't much more to using SQLiteStore as it "just works" with zarr-python. Full API reference can be accessed by running uv run pdoc -d numpy (or just doc if you have just installed).

A copy of the API documentation for the current main branch is also hosted here:

Documentation for zarr_sqlite

SQLiteStore implements the abstract store interface defined by the Zarr Core Specification, which essentially provides a key-value storage interface. It maps this interface directly onto a SQLite table, using string keys and binary BLOB values. Keys correspond to Zarr paths, while values contain either array chunks or Zarr metadata items. SQLite then provides the persistence and transactional semantics underneath the Zarr store.

The file format is described by a specification stored in this project's repository. This document should allow the implementation of SQLiteStore for other programming languages or Zarr libraries.

The version of this library was incremented to v1.0 to reflect the fact that it complies with v1 of the SQLiteStore Specification. However, this is a relatively young library that has not seen a lot of real-world use yet, therefore users should expect the occasional bug, and possibly breaking API changes in future versions, if absolutely necessary.

The database format defined by the Specification is intended to provide a stable basis for interoperability between implementations.