SPI for formats and storage: load any format, store it any way, use it through the standard interfaces #110

The problem

Only the Jackson model in kdbx-database can read and write proper KDBX files. Database, Group and Entry can't hold everything a KDBX file contains, so keeping KeePass data in another form, such as a different in-memory model or a SQL database, either loses data or means reimplementing the KeePass XML schema.

The basic module doesn't solve this. It uses the KDBX encryption layers but writes its own XML (<database>, not <KeePassFile>), so KeePass can't read its files, and it can't read files that KeePass wrote.

More generally, and a long-standing aim: databases in formats other than KDB and KDBX should be usable through the same Database, Group and Entry interfaces. So the goal is:

Load any format, store it any way, and make everything available to users through the standard interfaces.

Proposal: an SPI for formats and storage

Separate three things that are currently bound together in each implementation:

1. Formats read and write a file format: KDBX 4.x, KDBX 3.1, KDB, and later others. They know about encryption, headers and the document schema, and nothing about storage.
2. Storage holds the data: the current Jackson model, a plain in-memory model, a SQL database. It knows nothing about file formats.
3. The API stays Database, Group and Entry, implemented by the storage, which is what applications use.

Formats and storage talk through an SPI: a contract the library calls, not something applications use. It doesn't have to involve ServiceLoader discovery, because the caller chooses the format and the storage. Discovery could be added later.

Content and events

• Data objects carry the content: DatabaseData, GroupData and EntryData (with history), plus value types such as Times, CustomIcon and DeletedObject. 3.x requires Java 17 from 3.1.0, so these are records (with builders where there are many optional fields).
• A sink receives the content in document order: database(DatabaseData), startGroup(GroupData), entry(EntryData), endGroup(), deletedObject(DeletedObject), end().
• A source replays its content into a sink: void writeTo(Sink sink).

So:

• a format reader is a source: it decrypts, parses and pushes into a sink;
• a format writer is a sink: a source pushes into it, and it writes and encrypts;
• storage is both: it's filled by a reader, and replays into a writer;
• converting between formats or storages is source.writeTo(sink).

Common core and format-specific extensions

Formats don't share a schema, so the data objects have two parts:

• A common core that every format and storage understands: UUID, name or title, properties (protected or not), binary properties, times, icon, child groups and entries in order. This is roughly what Database, Group and Entry expose now.
• Format-specific extensions, each a typed value object owned by a format. For KDBX that's everything in the gap table below: AutoType, colours, tags, CustomData, custom icons, Meta settings, history and so on. KDB would have its own (e.g. meta-stream entries), and so would any later format.

Rules for extensions:

• Storage must keep extensions it doesn't understand, unchanged, and hand them back on replay. A KDBX file then round-trips through any storage without loss, even one that knows nothing about KeePass. (An in-memory store just keeps the objects; a SQL store might keep them as a serialized blob, or map the ones it knows to columns.)
• Writing to a different format uses the core and ignores extensions that belong to other formats, perhaps reporting what was dropped.
• Users reach extensions through the standard interfaces, e.g. entry.getExtension(KdbxEntryExtension.class) returning an Optional. This adds no KeePass-specific methods to Entry, and fits the existing supports…() capability methods on Database.

What the formats and storage keep doing

Formats handle file details that aren't content: HeaderHash, the Binaries pool and Ref numbers, the Protected attribute and decrypting protected values in order, date encoding, and the KDBX 4 inner header. None of it appears in the SPI.

The sink has to say what happens with invalid input, such as endGroup without startGroup, or a missing custom icon. Either readers guarantee the schema's rules, or sinks check them.

Considered: derived interfaces

The first version of this issue proposed KeePassEntry extends Entry, KeePassGroup extends Group and KeePassDatabase extends Database, with getters and setters for every KDBX field, and the KDBX reader and writer working against them. It was set aside because:

• every storage would have to implement a full live object graph (parents, searching, the recycle bin) plus about fifty KeePass setters, just so the reader and writer could reach the fields;
• it ties storage to one format, and doesn't extend to KDB or other formats;
• it mixes the application API with the storage contract.

Derived interfaces for applications that want KeePass fields directly could still come later, built on the extensions.

What KDBX needs that the core doesn't hold

This is the specification for the KDBX extensions. It was checked against Reichl's KDBX 4.1 schema (XSD/KDBX.4.1.reichl.xsd), not sample files, which may not contain every element. "Get" and "set" refer to the current Entry, Group and Database interfaces.

Entry (TEntry)

Group (TGroup)

Database (TMeta, TRoot)

Setting UUIDs and times also comes up in #96.

Rules from the schema

• UUIDs are unique, apart from history entries, which have their parent entry's UUID.
• Every CustomIconUUID refers to an icon in Meta/CustomIcons.
• The order of String, Binary and History items, and of child entries and groups, matters.

Questions

• The core: exactly which fields are common to all formats, and which are extensions? E.g. is history core or a KDBX extension? Are times core? KDB has them too.
• History: presumably read-only snapshots with the parent's UUID; the SPI has to say how they're delivered (inside EntryData seems simplest).
• Dates: decided: Instant. From 3.1.0 Entry and Group times are java.time.Instant rather than Date (KDBX times are UTC and KDB times are treated as UTC, so no zone is needed), and the SPI uses Instant too.
• KDBX 3.1: its schema differs in places (Meta/Binaries, HeaderHash), but those are format details, so one KDBX extension should cover both.
• Existing implementations: does the Jackson model become one storage, filled through the SPI, or keep its direct path for speed and also implement the SPI?

Testing

• A test XML file that uses every element and attribute in the KDBX 4.1 schema at least once, validated against the XSD.
• A recording sink that captures every event, to check the reader field by field.
• Round trips of that file through each storage, and through KDBX 4 (and 3.1), compared with the original. A missing field then fails the test, which no sample file can guarantee.

Examples

• Read a KDBX 4.1 file into storage that doesn't use Jackson, use it through Database, Group and Entry, and write it back with nothing lost.
• Convert between formats, e.g. KDB to KDBX.

Order

After #109 (write/read) and the move to Java 17, both in 3.1.0, so the new code uses the new stream handling and records from the start. The SPI is planned for release 3.2.0, as experimental at first, possibly with prototype releases for comment. The design is in FormatsAndStorageSPI.md on branch issue-110-spi.