fix(sdk)!: name the TDF manifest entry manifest.json per spec - #405
pflynn-virtru wants to merge 2 commits into
Conversation
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (3)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe SDK now writes ChangesManifest compatibility
Priority: ➖ Normal Estimated code review effort: 2 (Simple) | ~10 minutes Change: Bug fix Suggested reviewers: Merge Risk: 🟡 Moderate · up to The PR reports widespread failures in the cross-version integration tests, and the Checks workflow includes those tests in its CI result. Update the suite or explicitly accept the failing check before merging; whether branch protection enforces it is not established. Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to New readers preserve legacy compatibility, and the inspected loading paths retain their existing verification controls. Newly written archives nevertheless require upgraded readers, so mixed-version deployments and reader rollback need care. External consumer behavior remains unverified. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. A rabbit checks the zipped-up file, Comment |
The OpenTDF spec puts the manifest at the archive root under `manifest.json`. This SDK wrote and read `0.manifest.json`, an exact lookup with no fallback, so a TDF produced by any implementation written against the published spec was rejected outright. Write: `TDFWriter` emits `manifest.json`. Read: `TDFReader` accepts either name, preferring `manifest.json` when an archive carries both, so archives from earlier releases keep working. `SDK.isTDF` accepts either name too, and no longer requires the archive to hold exactly two entries -- the spec fixes where the manifest lives, not what else the archive may hold, and the count check made isTDF stricter than the reader it screens for. `0.payload` is unchanged: it is recorded in the manifest's payload URL field, so renaming it would alter manifest contents rather than just archive layout. Refs opentdf/platform#3513 Follows opentdf/platform#4049 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
446d8ec to
9503fa7
Compare
X-Test Failure Report |
Reword comments and test descriptions that call the legacy 0.manifest.json entry "off-spec". Wording only; no behavior change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com>
|
Refs opentdf/platform#3513. The read half of #405, split out so it can land on its own. ## Problem The [OpenTDF spec](https://opentdf.io/spec#tdf-structure) names the manifest entry in a `.tdf` archive `manifest.json`: > The `manifest.json` file MUST be in JSON format and reside within the root of the OpenTDF Zip archive. This SDK writes and reads `0.manifest.json`. On the read side `TDFReader` does an exact `containsKey` with no fallback, so a TDF produced by any implementation written against the published spec is rejected with `tdf doesn't contain a manifest` before any schema check runs. `SDK.isTDF` carries its own copy of the literal, so such an archive is screened out one step earlier still. The `0.` prefix is a holdover from an early design that anticipated several payload/manifest pairs per archive. That design never shipped. ## Change **Read** — `TDFReader` prefers `manifest.json` and falls back to `0.manifest.json`. Preference matters rather than first-match: an archive carrying both must not have its conformant entry passed over for the superseded one. **Sniffing** — `SDK.isTDF` accepts either name, so it does not reject an archive the reader can now read. It also no longer requires the archive to hold *exactly* two entries. That check was scoped out at first as unrelated to the entry name, but review pointed out it is not: an archive carrying both manifest names holds three entries, and `prefersTheSpecNameWhenAnArchiveCarriesBoth` is precisely the case the reader now handles — so the sniffer would reject what the reader it screens for accepts. The count was never part of the structure `isTDF` describes; the spec fixes where the manifest lives, not what else the archive may hold. **Duplicate entry names** — a zip may legally list one name twice, and `Collectors.toMap` turned that into an `IllegalStateException` out of `TDFReader`'s constructor: outside its declared `throws SDKException, IOException`, outside what a caller screening untrusted input catches, and outside the fuzz targets' catch list. Dropping the count check above made it newly reachable through `isTDF`, so it is rejected here as an `IllegalArgumentException` alongside the constructor's other malformed-input paths. An archive carrying *both* manifest names is still read, not rejected. Those are two distinct entries and the spec settles which one wins; a single name listed twice offers no principled choice. Worth being explicit about the tradeoff, since the root signature covers only `integrityInformation.segments` — not `policy`, not `keyAccess[].url` — and no hash covers the entry name: a DEK holder can file two internally-valid manifests over one ciphertext and have this SDK enforce one while a `0.`-keyed peer enforces the other. That ambiguity predates this PR; preferring the spec name relocates which side Java lands on rather than creating it. Failing closed would be a cross-SDK decision, not a Java-only one. **Write is untouched.** `TDFWriter` still emits `0.manifest.json`, so this release produces byte-identical archives and every existing reader keeps working. The new `TDF_MANIFEST_FILE_NAME_SPEC` constant is read-side only. Flipping the writer is the breaking half and stays in #405. `0.payload` is untouched. It is recorded in the manifest's `payload.url`, so renaming it would alter manifest contents rather than just archive layout, and neither the spec nor opentdf/platform#4049 fixes it. ## Testing Ported from #405, minus the write-side test: | Test | Failure without the change | |---|---| | `TDFReaderTest.readsManifestUnderTheSpecName` | `IllegalArgumentException: tdf doesn't contain a manifest` | | `TDFReaderTest.readsThePayloadAlongsideASpecNamedManifest` | same | | `TDFReaderTest.prefersTheSpecNameWhenAnArchiveCarriesBoth` | returned the non-aligned manifest | | `SDKTest.testExaminingTDFWithSpecManifestName` | `false` | | `SDKTest.testExaminingTDFWithBothManifestNames` | `false` -- three entries | | `SDKTest.testExaminingTDFWithAnExtraEntry` | `false` -- three entries | | `TDFReaderTest.rejectsAnArchiveThatListsOneNameTwice` | `IllegalStateException: Duplicate key 0.payload` | | `TDFReaderTest.rejectsAnArchiveThatListsTheManifestNameTwice` | same, for `manifest.json` | `prefersTheSpecNameWhenAnArchiveCarriesBoth` files a *different* manifest under each name, so it cannot pass by reading whichever entry the reader happened to pick. Guarding behavior that must not change: `TDFReaderTest.readsManifestUnderTheOffspecName`, `TDFReaderTest.rejectsAnArchiveWithNoManifestUnderEitherName`, `SDKTest.testExaminingZipWithNoManifest`, `SDKTest.testExaminingZipWithNoPayload`, and the pre-existing `SDKTest.testExaminingValidZTDF` / `testExaminingManifest`, which run against the checked-in non-aligned-named `sample.txt.tdf` fixture. The two negative `isTDF` cases hold two entries each, so they failed for the name they were missing rather than for their entry count even before the count check came out. Review of this branch added four tests that pin rules nothing else held down: `rejectsNearMissManifestEntryNames` (a basename or case-insensitive lookup passed the suite before, so `sub/manifest.json` would have been read as the manifest despite the spec's root-only rule), `SDKTest.testExaminingLargerZipWithNoManifest` (both negatives held two entries, so "any zip over two entries is a TDF" passed), and the two duplicate-name cases above. **Verification:** Local Java execution was unavailable because this machine has no JDK or Maven. GitHub CI passed `mavenverify`, Platform Integration (including FIPS), and the full cross-SDK xtest matrix on the PR head; the previously failing matrix job passed on rerun. The new tests were also watched failing against unmodified production code on #405's branch, where the shared code is identical. ## Downstream impact None. Readers gain a name they accept and `isTDF` gains archive shapes it recognizes; nothing either previously accepted is taken away, and no archive this SDK writes changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Compatibility** * Improved support for TDF archives using the standard `manifest.json` entry name. * Continued support for archives using the SDK’s existing `0.manifest.json` entry name. * When both manifest names are present, the standard `manifest.json` entry is used. * Archives may include additional entries beyond the manifest and payload. * **Bug Fixes** * TDF detection now correctly recognizes valid archives and rejects those missing a manifest or payload. * Archives containing duplicate entry names are now rejected with a clear error. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Paul Flynn <pflynn-virtru@users.noreply.github.com> Co-authored-by: Paul Flynn <pflynn-virtru@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>



Refs opentdf/platform#3513. Follows opentdf/platform#4049 (go) and unblocks the java cells of opentdf/tests#600.
Warning
Breaking file-format change. Archives written by this SDK now name the manifest entry
manifest.json. Readers that look it up by exact name cannot read them until they accept the spec name — including this SDK's own earlier releases.Problem
The OpenTDF spec names the manifest entry in a
.tdfarchivemanifest.json:This SDK writes and reads
0.manifest.json. On the read sideTDFReaderdoes an exactcontainsKeywith no fallback, so a TDF produced by any implementation written against the published spec is rejected withtdf doesn't contain a manifestbefore any schema check runs.SDK.isTDFcarries its own copy of the literal.Every SDK agreed on the same wrong name, so nothing caught it — which is exactly the third-party incompatibility opentdf/platform#3513 reports.
The
0.prefix is a holdover from an early design that anticipated several payload/manifest pairs per archive. That design never shipped.Change
Write —
TDFWriter.TDF_MANIFEST_FILE_NAMEis nowmanifest.json; the non-aligned name moves to a newTDF_MANIFEST_FILE_NAME_OFFSPECconstant that only readers consult.Read —
TDFReaderprefersmanifest.jsonand falls back to0.manifest.json, so archives written by earlier releases keep working. Preference matters rather than first-match: an archive carrying both must not have its conformant entry passed over for the superseded one.Sniffing —
SDK.isTDFaccepts either name. It also no longer requires the archive to hold exactly two entries. That check was flagged in opentdf/tests#600 and is worth dropping on its own terms: the spec fixes where the manifest lives, not what else the archive may hold, and the count madeisTDFstricter than theTDFReaderit screens for — a three-entry archive would be rejected by the sniffer and then read fine by the reader.0.payloadis untouched. It is recorded in the manifest'spayload.url, so renaming it would alter manifest contents rather than just archive layout, and neither the spec nor opentdf/platform#4049 fixes it.scripts/test-mlkem.shandscripts/test-hybrid-pqc.shalready match the entry by themanifest\.json$suffix and read back whatever name they find, so they need no change.Downstream impact
TDFReader.javaexactcontainsKey;SDK.isTDFgetManifest(cd, '0.manifest.json')intdf3/src/tdf.ts,tdf3/src/client/index.tssdk_constants.h: kTDFManifestFileNameunzip -p example.tdf 0.manifest.jsongo (opentdf/platform#4049) and xtest (opentdf/tests#600) are in flight.
Testing
Written test-first. Each behavioral test was watched failing against unmodified production code, and the failure was the expected one:
TDFReaderTest.readsManifestUnderTheSpecNameIllegalArgumentException: tdf doesn't contain a manifestTDFReaderTest.readsThePayloadAlongsideASpecNamedManifestTDFReaderTest.prefersTheSpecNameWhenAnArchiveCarriesBothTDFWriterTest.writesTheManifestUnderTheSpecEntryName0.manifest.jsonSDKTest.testExaminingTDFWithSpecManifestNamefalseSDKTest.testExaminingTDFWithAnExtraEntryfalseprefersTheSpecNameWhenAnArchiveCarriesBothfiles a different manifest under each name, so it cannot pass by reading whichever entry the reader happened to pick.Four tests guard behavior that must not change and pass both before and after:
TDFReaderTest.readsManifestUnderTheOffspecName,TDFReaderTest.rejectsAnArchiveWithNoManifestUnderEitherName,SDKTest.testExaminingZipWithNoManifest,SDKTest.testExaminingZipWithNoPayload. The pre-existingSDKTest.testExaminingValidZTDFandtestExaminingManifestrun against the checked-insample.txt.tdf, which is a non-aligned-named archive from before this change, so backward compatibility stays under test with a real fixture rather than a synthetic one.CI
mavenverifyis green — the full suite, both the fips and non-fips runs, includingTDFRootSignatureTestand the zip/fuzzing suites.Platform IntegrationandPlatform Integration (FIPS)also pass.I could not run any of that locally: codegen needs
bufagainst the BSR, and this machine is unauthenticated and hard rate-limited (resource_exhausted: too many requests), somvn installnever got pastgenerate-sources. Only the three targeted test classes above ran locally.xtest is red, as expected
Both
java@pull-405cells fail: 93 failed, 5 passed — the same tally, from the same four causes, that opentdf/platform#4049 reported for its own xct cells.KeyError: There is no item named '0.manifest.json' in the archivemanifest(),validate_manifest_schema()FileNotFoundError: .../unzipped/0.manifest.jsonupdate_manifest()CalledProcessErroron decryptassert b'segment' in ...test_tdf_altered_payload_endgets a manifest-lookup failure instead of a segment-integrity message78 of 93 are xtest's own hardcoded
0.manifest.json, which opentdf/tests#600 fixes.Of the decrypt failures, 8 are
sdk/java/dist/v0.18.0/cli.shraisingtdf doesn't contain a manifestagainst an archive written by this branch. That is this PR's documented breaking change observed directly, not a defect: v0.18.0's reader predates the fallback added here. Every current reader in the matrix handles it.xct (main, go@main)passes, confirming the failures are confined to cells that build java from this branch.This PR should stay draft until opentdf/tests#600 lands; merging sooner leaves xtest red.
🤖 Generated with Claude Code
Summary by CodeRabbit
manifest.jsonname when written.manifest.json.