Skip to content

fix(sdk)!: name the TDF manifest entry manifest.json per spec - #405

Open
pflynn-virtru wants to merge 2 commits into
mainfrom
fix/manifest-json-entry-name
Open

pflynn-virtru wants to merge 2 commits into
mainfrom
fix/manifest-json-entry-name

Conversation

@pflynn-virtru

@pflynn-virtru pflynn-virtru commented Sep 15, 2026 •

Copy link
Copy Markdown
Member

Refs opentdf/platform#3513. Follows opentdf/platform#4049 (go) and unblocks the java cells of opentdf/tests#600.

Warning

Breaking file-format change. Archives written by this SDK now name the manifest entry manifest.json. Readers that look it up by exact name cannot read them until they accept the spec name — including this SDK's own earlier releases.

Problem

The OpenTDF spec names the manifest entry in a .tdf archive manifest.json:

The manifest.json file MUST be in JSON format and reside within the root of the OpenTDF Zip archive.

This SDK writes and reads 0.manifest.json. On the read side TDFReader does an exact containsKey with no fallback, so a TDF produced by any implementation written against the published spec is rejected with tdf doesn't contain a manifest before any schema check runs. SDK.isTDF carries its own copy of the literal.

Every SDK agreed on the same wrong name, so nothing caught it — which is exactly the third-party incompatibility opentdf/platform#3513 reports.

The 0. prefix is a holdover from an early design that anticipated several payload/manifest pairs per archive. That design never shipped.

Change

Write — TDFWriter.TDF_MANIFEST_FILE_NAME is now manifest.json; the non-aligned name moves to a new TDF_MANIFEST_FILE_NAME_OFFSPEC constant that only readers consult.

Read — TDFReader prefers manifest.json and falls back to 0.manifest.json, so archives written by earlier releases keep working. Preference matters rather than first-match: an archive carrying both must not have its conformant entry passed over for the superseded one.

Sniffing — SDK.isTDF accepts either name. It also no longer requires the archive to hold exactly two entries. That check was flagged in opentdf/tests#600 and is worth dropping on its own terms: the spec fixes where the manifest lives, not what else the archive may hold, and the count made isTDF stricter than the TDFReader it screens for — a three-entry archive would be rejected by the sniffer and then read fine by the reader.

0.payload is untouched. It is recorded in the manifest's payload.url, so renaming it would alter manifest contents rather than just archive layout, and neither the spec nor opentdf/platform#4049 fixes it.

scripts/test-mlkem.sh and scripts/test-hybrid-pqc.sh already match the entry by the manifest\.json$ suffix and read back whatever name they find, so they need no change.

Downstream impact

Consumer Site
java-sdk < this release TDFReader.java exact containsKey; SDK.isTDF
web-sdk getManifest(cd, '0.manifest.json') in tdf3/src/tdf.ts, tdf3/src/client/index.ts
client-cpp sdk_constants.h: kTDFManifestFileName
opentdf/docs quickstart tells users to unzip -p example.tdf 0.manifest.json

go (opentdf/platform#4049) and xtest (opentdf/tests#600) are in flight.

Testing

Written test-first. Each behavioral test was watched failing against unmodified production code, and the failure was the expected one:

Test Failure without the change
TDFReaderTest.readsManifestUnderTheSpecName IllegalArgumentException: tdf doesn't contain a manifest
TDFReaderTest.readsThePayloadAlongsideASpecNamedManifest same
TDFReaderTest.prefersTheSpecNameWhenAnArchiveCarriesBoth returned the non-aligned manifest
TDFWriterTest.writesTheManifestUnderTheSpecEntryName entry named 0.manifest.json
SDKTest.testExaminingTDFWithSpecManifestName false
SDKTest.testExaminingTDFWithAnExtraEntry false

prefersTheSpecNameWhenAnArchiveCarriesBoth files a different manifest under each name, so it cannot pass by reading whichever entry the reader happened to pick.

Four tests guard behavior that must not change and pass both before and after: TDFReaderTest.readsManifestUnderTheOffspecName, TDFReaderTest.rejectsAnArchiveWithNoManifestUnderEitherName, SDKTest.testExaminingZipWithNoManifest, SDKTest.testExaminingZipWithNoPayload. The pre-existing SDKTest.testExaminingValidZTDF and testExaminingManifest run against the checked-in sample.txt.tdf, which is a non-aligned-named archive from before this change, so backward compatibility stays under test with a real fixture rather than a synthetic one.

$ mvn -pl sdk -am test -Dtest='TDFReaderTest,TDFWriterTest,SDKTest'
Tests run: 19, Failures: 0, Errors: 0, Skipped: 0

CI

mavenverify is green — the full suite, both the fips and non-fips runs, including TDFRootSignatureTest and the zip/fuzzing suites. Platform Integration and Platform Integration (FIPS) also pass.

I could not run any of that locally: codegen needs buf against the BSR, and this machine is unauthenticated and hard rate-limited (resource_exhausted: too many requests), so mvn install never got past generate-sources. Only the three targeted test classes above ran locally.

xtest is red, as expected

Both java@pull-405 cells fail: 93 failed, 5 passed — the same tally, from the same four causes, that opentdf/platform#4049 reported for its own xct cells.

Cause Count Whose bug
KeyError: There is no item named '0.manifest.json' in the archive 46 xtest — manifest(), validate_manifest_schema()
FileNotFoundError: .../unzipped/0.manifest.json 32 xtest — update_manifest()
CalledProcessError on decrypt 12 readers that predate the spec name
assert b'segment' in ... 3 same — test_tdf_altered_payload_end gets a manifest-lookup failure instead of a segment-integrity message

78 of 93 are xtest's own hardcoded 0.manifest.json, which opentdf/tests#600 fixes.

Of the decrypt failures, 8 are sdk/java/dist/v0.18.0/cli.sh raising tdf doesn't contain a manifest against an archive written by this branch. That is this PR's documented breaking change observed directly, not a defect: v0.18.0's reader predates the fallback added here. Every current reader in the matrix handles it.

xct (main, go@main) passes, confirming the failures are confined to cells that build java from this branch.

This PR should stay draft until opentdf/tests#600 lands; merging sooner leaves xtest red.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Archives now use the specification-aligned manifest.json name when written.
    • The SDK recognizes archives using either the new or legacy manifest name, including archives with extra entries. When both names are present, it reads manifest.json.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 3f5b1b58-2ea0-4563-ab38-f1461c56c884

📥 Commits

Reviewing files that changed from the base of the PR and between 446d8ec and ce5f6f7.

📒 Files selected for processing (3)
  • sdk/src/main/java/io/opentdf/platform/sdk/SDK.java
  • sdk/src/main/java/io/opentdf/platform/sdk/TDFReader.java
  • sdk/src/test/java/io/opentdf/platform/sdk/SDKTest.java

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The SDK now writes manifest.json and reads both the spec and legacy manifest names. The reader prefers the spec name when both are present. SDK.isTDF recognizes archives with either manifest name and 0.payload, including archives with additional entries.

Changes

Manifest compatibility

Layer / File(s) Summary
Spec manifest output
sdk/src/main/java/io/opentdf/platform/sdk/TDFWriter.java, sdk/src/test/java/io/opentdf/platform/sdk/TDFWriterTest.java
TDFWriter now emits manifest.json and exposes 0.manifest.json as the legacy constant. Tests verify the archive entry names.
Manifest reading compatibility
sdk/src/main/java/io/opentdf/platform/sdk/TDFReader.java, sdk/src/test/java/io/opentdf/platform/sdk/TDFReaderTest.java
TDFReader prefers manifest.json and falls back to 0.manifest.json. Tests cover both names, precedence, missing manifests, and payload reading.
TDF archive detection
sdk/src/main/java/io/opentdf/platform/sdk/SDK.java, sdk/src/test/java/io/opentdf/platform/sdk/SDKTest.java
SDK.isTDF accepts either manifest name with 0.payload and allows additional entries. Tests cover valid archives and archives missing either required entry.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: mkleene

Merge Risk: 🟡 Moderate · up to ce5f6

The PR reports widespread failures in the cross-version integration tests, and the Checks workflow includes those tests in its CI result. Update the suite or explicitly accept the failing check before merging; whether branch protection enforces it is not established.

Security Architecture Review

Security architecture risk: 🔵 Low · up to ce5f6

New readers preserve legacy compatibility, and the inspected loading paths retain their existing verification controls. Newly written archives nevertheless require upgraded readers, so mixed-version deployments and reader rollback need care. External consumer behavior remains unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The directly affected exposure is attacker-supplied ZIP input processed by SDK consumers. Supported archive names and recognition expand, but the inspected change grants no new credentials or KAS authority. External callers that route decisions using isTDF were not established by the available repository evidence.

Trust Boundaries and Controls

  • observed — Both manifest names feed the same downstream parsing and loading path; recognition does not substitute for its controls. KAS allowlist enforcement remains conditional on the existing ignoreKasAllowlist configuration, and segment integrity remains checked before each segment is released.

Resilience and Maintainability Implications

  • observed — Duplicate entry names still fail the reader's duplicate-sensitive map construction. Manifest reads obtain fresh entry streams, while payload reading consumes a single stream and emits verified segments incrementally. A later failure can therefore leave an output prefix; the filename change does not add transactional rollback or retry semantics.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 35.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 6 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main breaking change: naming the TDF manifest entry manifest.json according to the specification.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the zipped-up file,
Finds both names waiting in a row.
The newer name comes first to read,
While older archives still can go.
Extra entries cause no alarm,
And manifest.json seals the show.

Comment @coderabbitai help to get the list of available commands.

@pflynn-virtru
pflynn-virtru marked this pull request as ready for review September 15, 2026 19:04
@pflynn-virtru
pflynn-virtru requested review from a team as code owners September 15, 2026 19:04
The OpenTDF spec puts the manifest at the archive root under
`manifest.json`. This SDK wrote and read `0.manifest.json`, an exact
lookup with no fallback, so a TDF produced by any implementation written
against the published spec was rejected outright.

Write: `TDFWriter` emits `manifest.json`.

Read: `TDFReader` accepts either name, preferring `manifest.json` when an
archive carries both, so archives from earlier releases keep working.

`SDK.isTDF` accepts either name too, and no longer requires the archive
to hold exactly two entries -- the spec fixes where the manifest lives,
not what else the archive may hold, and the count check made isTDF
stricter than the reader it screens for.

`0.payload` is unchanged: it is recorded in the manifest's payload URL
field, so renaming it would alter manifest contents rather than just
archive layout.

Refs opentdf/platform#3513
Follows opentdf/platform#4049

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
@pflynn-virtru
pflynn-virtru force-pushed the fix/manifest-json-entry-name branch from 446d8ec to 9503fa7 Compare September 15, 2026 19:06
@pflynn-virtru pflynn-virtru changed the title fix(sdk)!: name the ZTDF manifest entry manifest.json per spec fix(sdk)!: name the TDF manifest entry manifest.json per spec Sep 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

X-Test Failure Report

Reword comments and test descriptions that call the legacy
0.manifest.json entry "off-spec". Wording only; no behavior change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

pflynn-virtru added a commit that referenced this pull request Oct 1, 2026
Refs opentdf/platform#3513. The read half of #405, split out so it can
land on its own.

## Problem

The [OpenTDF spec](https://opentdf.io/spec#tdf-structure) names the
manifest entry in a `.tdf` archive `manifest.json`:

> The `manifest.json` file MUST be in JSON format and reside within the
root of the OpenTDF Zip archive.

This SDK writes and reads `0.manifest.json`. On the read side
`TDFReader` does an exact `containsKey` with no fallback, so a TDF
produced by any implementation written against the published spec is
rejected with `tdf doesn't contain a manifest` before any schema check
runs. `SDK.isTDF` carries its own copy of the literal, so such an
archive is screened out one step earlier still.

The `0.` prefix is a holdover from an early design that anticipated
several payload/manifest pairs per archive. That design never shipped.

## Change

**Read** — `TDFReader` prefers `manifest.json` and falls back to
`0.manifest.json`. Preference matters rather than first-match: an
archive carrying both must not have its conformant entry passed over for
the superseded one.

**Sniffing** — `SDK.isTDF` accepts either name, so it does not reject an
archive the reader can now read. It also no longer requires the archive
to hold *exactly* two entries. That check was scoped out at first as
unrelated to the entry name, but review pointed out it is not: an
archive carrying both manifest names holds three entries, and
`prefersTheSpecNameWhenAnArchiveCarriesBoth` is precisely the case the
reader now handles — so the sniffer would reject what the reader it
screens for accepts. The count was never part of the structure `isTDF`
describes; the spec fixes where the manifest lives, not what else the
archive may hold.

**Duplicate entry names** — a zip may legally list one name twice, and
`Collectors.toMap` turned that into an `IllegalStateException` out of
`TDFReader`'s constructor: outside its declared `throws SDKException,
IOException`, outside what a caller screening untrusted input catches,
and outside the fuzz targets' catch list. Dropping the count check above
made it newly reachable through `isTDF`, so it is rejected here as an
`IllegalArgumentException` alongside the constructor's other
malformed-input paths.

An archive carrying *both* manifest names is still read, not rejected.
Those are two distinct entries and the spec settles which one wins; a
single name listed twice offers no principled choice. Worth being
explicit about the tradeoff, since the root signature covers only
`integrityInformation.segments` — not `policy`, not `keyAccess[].url` —
and no hash covers the entry name: a DEK holder can file two
internally-valid manifests over one ciphertext and have this SDK enforce
one while a `0.`-keyed peer enforces the other. That ambiguity predates
this PR; preferring the spec name relocates which side Java lands on
rather than creating it. Failing closed would be a cross-SDK decision,
not a Java-only one.

**Write is untouched.** `TDFWriter` still emits `0.manifest.json`, so
this release produces byte-identical archives and every existing reader
keeps working. The new `TDF_MANIFEST_FILE_NAME_SPEC` constant is
read-side only. Flipping the writer is the breaking half and stays in
#405.

`0.payload` is untouched. It is recorded in the manifest's
`payload.url`, so renaming it would alter manifest contents rather than
just archive layout, and neither the spec nor opentdf/platform#4049
fixes it.

## Testing

Ported from #405, minus the write-side test:

| Test | Failure without the change |
|---|---|
| `TDFReaderTest.readsManifestUnderTheSpecName` |
`IllegalArgumentException: tdf doesn't contain a manifest` |
| `TDFReaderTest.readsThePayloadAlongsideASpecNamedManifest` | same |
| `TDFReaderTest.prefersTheSpecNameWhenAnArchiveCarriesBoth` | returned
the non-aligned manifest |
| `SDKTest.testExaminingTDFWithSpecManifestName` | `false` |
| `SDKTest.testExaminingTDFWithBothManifestNames` | `false` -- three
entries |
| `SDKTest.testExaminingTDFWithAnExtraEntry` | `false` -- three entries
|
| `TDFReaderTest.rejectsAnArchiveThatListsOneNameTwice` |
`IllegalStateException: Duplicate key 0.payload` |
| `TDFReaderTest.rejectsAnArchiveThatListsTheManifestNameTwice` | same,
for `manifest.json` |

`prefersTheSpecNameWhenAnArchiveCarriesBoth` files a *different*
manifest under each name, so it cannot pass by reading whichever entry
the reader happened to pick.

Guarding behavior that must not change:
`TDFReaderTest.readsManifestUnderTheOffspecName`,
`TDFReaderTest.rejectsAnArchiveWithNoManifestUnderEitherName`,
`SDKTest.testExaminingZipWithNoManifest`,
`SDKTest.testExaminingZipWithNoPayload`, and the pre-existing
`SDKTest.testExaminingValidZTDF` / `testExaminingManifest`, which run
against the checked-in non-aligned-named `sample.txt.tdf` fixture. The
two negative `isTDF` cases hold two entries each, so they failed for the
name they were missing rather than for their entry count even before the
count check came out.

Review of this branch added four tests that pin rules nothing else held
down: `rejectsNearMissManifestEntryNames` (a basename or
case-insensitive lookup passed the suite before, so `sub/manifest.json`
would have been read as the manifest despite the spec's root-only rule),
`SDKTest.testExaminingLargerZipWithNoManifest` (both negatives held two
entries, so "any zip over two entries is a TDF" passed), and the two
duplicate-name cases above.

**Verification:** Local Java execution was unavailable because this
machine has no JDK or Maven. GitHub CI passed `mavenverify`, Platform
Integration (including FIPS), and the full cross-SDK xtest matrix on the
PR head; the previously failing matrix job passed on rerun. The new
tests were also watched failing against unmodified production code on
#405's branch, where the shared code is identical.

## Downstream impact

None. Readers gain a name they accept and `isTDF` gains archive shapes
it recognizes; nothing either previously accepted is taken away, and no
archive this SDK writes changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Compatibility**
* Improved support for TDF archives using the standard `manifest.json`
entry name.
* Continued support for archives using the SDK’s existing
`0.manifest.json` entry name.
* When both manifest names are present, the standard `manifest.json`
entry is used.
* Archives may include additional entries beyond the manifest and
payload.

* **Bug Fixes**
* TDF detection now correctly recognizes valid archives and rejects
those missing a manifest or payload.
* Archives containing duplicate entry names are now rejected with a
clear error.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
Co-authored-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant