Expose recent committed ledger prefixes by achamayou · Pull Request #8214 · microsoft/CCF

Amaury Chamayou (achamayou) · 2026-08-26T15:17:48Z

Summary

Add opt-in include_committed_prefix=true discovery for committed ledger entries not yet available in canonical .committed files.
Serve immutable .committed_prefix resources with 307 Temporary Redirect, Cache-Control: no-store, range requests, ETags, and representation digests.
Preserve physical ledger boundaries and keep prefix files excluded from committed-only discovery and recovery.
Materialise prefixes without stalling the ledger writer: the chunk header and positions table are built under the ledger state lock, then the entry bytes are read with pread() on a duplicated descriptor with no ledger lock held. pread() never touches the writer's FILE* position, and the range is stable because Ledger::truncate never regresses past committed_idx. Committed bytes are always flushed (committable entries and completed files fflush), so a violation surfaces as a short read (HTTP 500), never as incorrect bytes.
Ignore .committed_prefix files in ccf.ledger.Ledger directory discovery, matching the host; they remain readable individually with ccf.ledger.LedgerChunk.
Add host and Python tests, endpoint e2e coverage, a 64 KiB-50 MiB benchmark matrix, OpenAPI updates, and operator documentation.

Design discussion: #8207

Testing

ledger_test (including a case with non-committable entries inside the served range and an unflushed tail after it) and files_cleanup_test
Python SDK test suite, including Ledger ignoring .committed_prefix files
ledger_bench: pread() materialisation runs at 0.59-1.08x of a plain read of the same file across 64 KiB-50 MiB
schema_test committed-prefix-download scenario, pinned with a high snapshot_tx_interval so a due snapshot cannot promote the source file to .committed mid-test
Node OpenAPI validation
Full logging sample build

Add opt-in discovery and immutable download endpoints for recent committed ledger entries that have not yet been published as canonical .committed files. Include host and SDK support, unit and end-to-end coverage, benchmarks, OpenAPI, and operator documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add the release note for draft PR #8214. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds opt-in access to recent committed ledger prefixes while preserving canonical ledger and recovery behavior.

Changes:

Adds prefix discovery, immutable downloads, ranges, digests, and metadata.
Extends host and Python ledger parsing.
Adds tests, benchmarks, OpenAPI definitions, and operator documentation.

Custom instructions used:

.github/copilot-instructions.md
.github/instructions/changelog.instructions.md
.github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`CHANGELOG.md`	Records the new capability.
`include/ccf/http_consts.h`	Adds the chunk-kind header.
`src/host/ledger.h`	Materializes committed prefixes.
`src/host/ledger_filenames.h`	Recognizes and excludes prefix files.
`src/host/test/ledger.cpp`	Tests prefix creation and recovery exclusion.
`src/host/test/ledger_bench.cpp`	Benchmarks prefix materialization.
`src/host/test/files_cleanup_test.cpp`	Tests strict prefix filenames.
`src/node/rpc/file_serving_handlers.h`	Implements discovery and download endpoints.
`src/node/rpc/ledger_interface.h`	Extends the ledger subsystem interface.
`src/node/rpc/ledger_subsystem.h`	Connects RPC and host ledger operations.
`src/node/rpc/node_frontend.h`	Bumps the API version.
`python/src/ccf/ledger.py`	Parses committed-prefix filenames.
`python/tests/test_ledger.py`	Tests Python filename handling.
`tests/e2e_operations.py`	Adds endpoint integration coverage.
`tests/schema.py`	Registers the new e2e scenario.
`doc/operations/configuration.rst`	Documents operator feature gating.
`doc/operations/ledger_snapshot.rst`	Documents prefix semantics and usage.
`doc/schemas/node_openapi.json`	Updates the generated API schema.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Reject zero-length suffix ranges, cover opt-in redirects across lagging and late-joining nodes, exclude valid committed-prefix files from read-only ledger lookup, and fix the clang-tidy callable warning. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mmitted-ledger-chunks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…mmitted-ledger-chunks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

ccf::pal::Mutex -> ccf::ds::Mutex, TimeBoundLogger -> ccf::ds::TimeBoundLogger, asynchost ledger filename helpers -> ccf::ledger, and the subsystem type lookup uses the abstract interface. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…n Ledger - Materialise the chunk header and positions table under the ledger state lock, then read the entry bytes with pread() on a duplicated descriptor without holding any ledger lock, so serving a prefix does not stall the ledger writer. - Ignore .committed_prefix files in ccf.ledger.Ledger directory discovery, matching the host. - Pin snapshot_tx_interval in the committed-prefix e2e scenario so a due snapshot cannot promote the source file to .committed mid-test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve CHANGELOG conflict by moving the #8214 entry to the unpublished 7.0.18 section, and drop the LedgerFile::file_lock guard from prepare_completed_chunk since #8414 removed that lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Explain that clients tracking `since` must not advance it past a committed prefix, and that committed-prefix URLs are not immutable across a recovery from a ledger missing some of their transactions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Only the header and positions table of a committed prefix are now materialised under the ledger state lock, rather than a buffer for the whole prefix. Entry bytes are read afterwards with pread(), only for the requested byte range, and only once that range has been validated. The whole prefix is still read when a digest of the full representation is requested. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-10-01T16:28:25Z

Description

Comparing 5 available runs from this branch (#8214) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 3,103 tx/s ▬ 0%"]
  axis b1["Basic Blocking 20ms: 15,338 tx/s ▬ 0%"]
  axis b2["Basic Blocking 2ms: 49,880 tx/s ▬ +2%"]
  axis b3["Basic JS: 16,049 tx/s ▬ 0%"]
  axis b4["Historical Queries: 970,316 tx/s ▬ +7%"]
  axis b5["L…g Certificate Blocking: 28,844 tx/s ▬ 0%"]
  axis b6["Logging JWT Blocking: 15,320 tx/s ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.29, 100.25, 110.98, 103.83, 149.34, 100.59, 100.25}
  curve stddev1_high["main EWMA + 1 std dev"]{100.14, 100.13, 105.49, 101.91, 124.67, 100.29, 100.12}
  curve stddev1_low["main EWMA - 1 std dev"]{99.86, 99.87, 94.51, 98.09, 75.33, 99.71, 99.88}
  curve stddev2_low["main EWMA - 2 std dev"]{99.71, 99.75, 89.02, 96.17, 50.66, 99.41, 99.75}
  curve branch_0["#8214 (4 runs earlier)"]{99.83, 99.88, 103.59, 101.36, 104.00, 100.30, 99.93}
  curve branch_1["#8214 (3 runs earlier)"]{100.16, 99.79, 90.39, 95.06, 100.37, 99.63, 99.89}
  curve branch_2["#8214 (2 runs earlier)"]{100.02, 99.99, 90.73, 96.42, 100.61, 100.05, 99.88}
  curve branch_3["#8214 (1 run earlier)"]{100.11, 99.80, 103.40, 100.52, 105.53, 99.95, 99.90}
  curve branch_4["#8214"]{100.19, 99.88, 102.13, 100.03, 106.52, 99.89, 100.03}
  graticule polygon
  max 184
  min 16
  ticks 0
  showLegend false

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 98 ms ▬ 0%"]
  axis b1["Basic Blocking 20ms: 19 ms ▬ 0%"]
  axis b2["Basic Blocking 2ms: 5 ms ▬ -6%"]
  axis b3["Basic JS: 19 ms ▬ -1%"]
  axis b4["Historical Queries: 32 ms ▬ -36%"]
  axis b5["Logging Certificate Blocking: 19 ms ▬ 0%"]
  axis b6["Logging JWT Blocking: 19 ms ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 115.01, 105.36, 397.83, 100.00, 100.00}
  curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 107.50, 102.68, 248.92, 100.00, 100.00}
  curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 92.50, 97.32, 0.00, 100.00, 100.00}
  curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 84.99, 94.64, 0.00, 100.00, 100.00}
  curve branch_0["#8214 (4 runs earlier)"]{100.00, 100.00, 93.79, 99.46, 63.82, 100.00, 100.00}
  curve branch_1["#8214 (3 runs earlier)"]{100.00, 100.00, 112.54, 104.69, 63.82, 100.00, 100.00}
  curve branch_2["#8214 (2 runs earlier)"]{100.00, 100.00, 112.54, 104.69, 67.81, 100.00, 100.00}
  curve branch_3["#8214 (1 run earlier)"]{100.00, 100.00, 93.79, 99.46, 61.82, 100.00, 100.00}
  curve branch_4["#8214"]{100.00, 100.00, 93.79, 99.46, 63.82, 100.00, 100.00}
  graticule polygon
  max 538
  ticks 0
  showLegend false

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking 100ms: 88.3 MiB ▬ 0%"]
  axis b1["Basic Blocking 20ms: 88 MiB ▼ 2%"]
  axis b2["Basic Blocking 2ms: 91 MiB ▬ -1%"]
  axis b3["Basic JS: 97.2 MiB ▲ 1%"]
  axis b4["Historical Queries: 146 MiB ▲ 1%"]
  axis b5["Logging Certificate Blocking: 114 MiB ▬ 0%"]
  axis b6["Logging JWT Blocking: 88.7 MiB ▬ +1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{102.02, 102.51, 102.99, 101.01, 101.48, 101.83, 102.83}
  curve stddev1_high["main EWMA + 1 std dev"]{101.01, 101.26, 101.50, 100.50, 100.74, 100.91, 101.41}
  curve stddev1_low["main EWMA - 1 std dev"]{98.99, 98.74, 98.50, 99.50, 99.26, 99.09, 98.59}
  curve stddev2_low["main EWMA - 2 std dev"]{97.98, 97.49, 97.01, 98.99, 98.52, 98.17, 97.17}
  curve branch_0["#8214 (4 runs earlier)"]{100.29, 98.10, 99.57, 100.36, 99.62, 98.87, 99.45}
  curve branch_1["#8214 (3 runs earlier)"]{99.15, 99.98, 100.17, 99.86, 99.67, 101.76, 98.83}
  curve branch_2["#8214 (2 runs earlier)"]{100.73, 101.42, 101.45, 101.00, 99.77, 99.00, 100.55}
  curve branch_3["#8214 (1 run earlier)"]{98.98, 98.63, 98.48, 102.38, 100.64, 100.07, 100.31}
  curve branch_4["#8214"]{99.78, 98.04, 99.29, 100.63, 100.96, 99.73, 100.98}
  graticule polygon
  max 107
  min 93
  ticks 0
  showLegend false

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(9){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CCF c…n context lifecycle: 21,435 ops/s ▬ 0%"]
  axis b1["CCF fresh JS invocation: 19,667 ops/s ▬ +1%"]
  axis b2["CHAMP get: 65,270,740 ops/s ▬ +2%"]
  axis b3["CHAMP put: 8,238,134 ops/s ▬ +2%"]
  axis b4["KV deserialisation: 2,727,769 ops/s ▬ -1%"]
  axis b5["KV serialisation: 2,435,460 ops/s ▬ -1%"]
  axis b6["KV s…t deserialisation: 6,177 ops/s ▬ -1%"]
  axis b7["KV snapshot serialisation: 4,695 ops/s ▼ 3%"]
  axis b8["Q…S s…d c…t lifecycle: 26,386 ops/s ▬ +1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{102.76, 102.96, 103.95, 106.17, 103.12, 103.56, 103.66, 104.97, 102.69}
  curve stddev1_high["main EWMA + 1 std dev"]{101.38, 101.48, 101.97, 103.08, 101.56, 101.78, 101.83, 102.48, 101.35}
  curve stddev1_low["main EWMA - 1 std dev"]{98.62, 98.52, 98.03, 96.92, 98.44, 98.22, 98.17, 97.52, 98.65}
  curve stddev2_low["main EWMA - 2 std dev"]{97.24, 97.04, 96.05, 93.83, 96.88, 96.44, 96.34, 95.03, 97.31}
  curve branch_0["#8214 (4 runs earlier)"]{100.22, 99.93, 102.09, 103.29, 99.05, 98.91, 98.43, 96.75, 99.04}
  curve branch_1["#8214 (3 runs earlier)"]{100.53, 99.17, 99.29, 100.46, 96.67, 100.37, 100.26, 96.04, 101.23}
  curve branch_2["#8214 (2 runs earlier)"]{100.70, 100.98, 99.17, 100.31, 102.73, 98.91, 101.60, 97.07, 99.88}
  curve branch_3["#8214 (1 run earlier)"]{101.26, 101.01, 102.09, 99.92, 100.73, 99.64, 99.67, 95.85, 100.76}
  curve branch_4["#8214"]{100.48, 100.78, 101.70, 102.32, 98.78, 99.40, 98.98, 97.49, 100.84}
  graticule polygon
  max 111
  min 89
  ticks 0
  showLegend false

- Build committed prefix names with the same helper used to parse them, accept only canonical names, and use one named range type throughout. - Share the redirect-to-another-node logic between the next node and primary paths of the ledger chunk locator. - Own the reader's duplicated descriptor with a small RAII type, and report read failures as nullopt rather than exceptions. - Remove a redundant check in get_file_from_idx, move test-only full reads into the tests, and rename a shadowed variable. - Clarify that a committed prefix starts at the requested sequence number, use one sentence per line in that section, simplify the SDK range validation, and restore an unrelated trailing blank line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) · 2026-10-02T01:22:30Z

Fixed-revision benchmark results

I do not see a statistically supported performance regression at the tested revision. The initial 2 ms blocking-throughput warning was not reproduced. However, CI assigned the two revisions to different mixes of faster and slower machines, so the pooled results must not be interpreted as proof of a PR improvement either.

Collected 12 new successful measurements per revision, from 2026-10-01 21:54 to 2026-10-02 01:01 UTC, without changing either revision:

PR head: 87004a2, tested as merge commit 4569597.
Main control: f656489, the PR's exact base.

Only the new attempts 2 through 13 are included below, not the earlier exploratory results. All measurements used the existing virtual benchmark jobs, WORKER_THREADS=2, and the gha-vmss-d16av7-ci pool. Every result was retained for this analysis; it is not limited to the five PR runs retained by the radar chart.

Primary metric: 2 ms blocking throughput

The pooled arithmetic means are 46,502 tx/s on main and 49,669 tx/s on the PR. The prespecified unpaired, two-sided Welch comparison on log throughput gives a +6.83% geometric-mean ratio change, with a pointwise 95% interval of +1.84% to +12.06% and a p-value of 0.0090. An exact rank-test sensitivity check gives 0.0145.

That apparent improvement is confounded by machine allocation. I checked the actual Machine name in the CI log for every job, rather than treating the changing runner registration names as different machines:

Machine	Main samples	PR samples	Main mean, tx/s	PR mean, tx/s	Arithmetic-mean change
`a1ebf7acc000000`	8	2	44,691	44,229	-1.04%
`31279283c000000`	3	8	49,942	50,470	+1.06%
`bc584fccc000000`	1	1	50,661	51,511	+1.68%
`809780fec000000`	0	1	N/A	52,303	No matched control

Main received eight measurements on the slower machine; the PR received only two. Within the three machines that ran both revisions, the observed differences are much smaller.

An exploratory, post-hoc machine-adjusted analysis of those shared machines, using 11 PR and 12 main measurements, gives +0.34%, with an approximate 95% interval of -3.26% to +4.08%. This uses a log-throughput model with machine fixed effects and machine-clustered standard errors, with a small-sample correction and two degrees of freedom. A separate exact permutation check stratified by machine gives a two-sided p-value of 0.545, over 14,850 possible within-machine label allocations.

A 5% slowdown is outside that adjusted interval under this model, but this is conditional evidence, not a hardware-independent guarantee: only three machines have both revisions, the adjustment was post-hoc, and one PR-only machine cannot enter the matched comparison. The pooled coefficients of variation were 5.3% on the PR and 5.8% on main, above the 4% planning assumption, so the original approximate 80% power estimate should not be treated as achieved automatically.

Other performance metrics

No adverse change passes a Holm-corrected, two-sided 5% significance threshold across the 36 secondary metrics: the other 29 charted metrics plus seven p99 latency measurements. In particular, neither the earlier 20 ms memory warning nor the broad KV/JS slowdown is reproduced.

Selected results are below. Changes and intervals are inferred from log ratios; positive is better for throughput/rate, negative for RSS/latency. These are pointwise 95% intervals, not simultaneous intervals across the metric family.

Metric	PR/main change	Pointwise 95% interval
20 ms blocking throughput	+0.030%	-0.117% to +0.178%
20 ms blocking RSS	-0.059%	-0.868% to +0.757%
Basic JS throughput	+0.939%	-0.906% to +2.818%
KV serialisation rate	-0.560%	-2.015% to +0.916%
KV deserialisation rate	-0.181%	-1.550% to +1.207%
KV snapshot serialisation rate	-1.358%	-4.113% to +1.475%
KV snapshot deserialisation rate	-0.944%	-2.146% to +0.273%
2 ms blocking p99 latency	-5.132%	-11.004% to +1.128%

The secondary tests above are pooled exploratory comparisons and share the machine-allocation caveat. Latencies are per-run p50/p99 values, with millisecond quantisation; identical rounded values are not evidence of zero underlying variance. Workflow attempts, not individual requests, are the measurement units.

Scope: these results cover the standard virtual bench-ab workloads, not SNP or performance under concurrent committed-prefix downloads. A balanced crossover, where each machine measures both commits, would be needed for a stronger causal improvement or equivalence claim.

All 12 new 2 ms throughput measurements and CI sources

Each measurement links to its CI attempt. Machine aliases refer to the prefixes in the machine table above.

Round	PR tx/s (machine)	Main tx/s (machine)
1	49,553.9 (`312`)	44,664.9 (`a1`)
2	51,511.1 (`bc`)	45,333.8 (`a1`)
3	52,302.9 (`809`)	49,999.2 (`312`)
4	50,594.5 (`312`)	50,661.1 (`bc`)
5	51,215.6 (`312`)	44,221.6 (`a1`)
6	44,147.3 (`a1`)	50,089.1 (`312`)
7	44,310.1 (`a1`)	49,738.6 (`312`)
8	51,154.5 (`312`)	45,215.7 (`a1`)
9	50,499.6 (`312`)	43,886.1 (`a1`)
10	49,880.3 (`312`)	45,290.4 (`a1`)
11	50,271.6 (`312`)	44,792.0 (`a1`)
12	50,588.1 (`312`)	44,126.1 (`a1`)

Custom instructions used

.github/instructions/reviewing.instructions.md and .github/skills/testing/SKILL.md.

Co-authored-by: maxtropets <maxtropets@users.noreply.github.com> Co-authored-by: Amaury Chamayou <amchamay@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the published 7.0.18 notes intact and synchronise the Python SDK version with the new release section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) and others added 4 commits August 26, 2026 16:17

Document committed ledger prefix API

7954233

Add the release note for draft PR #8214. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into achamayou-document-committed-ledger-chunks

7b62e9b

Merge branch 'main' into achamayou-document-committed-ledger-chunks

12516b6

Amaury Chamayou (achamayou) requested a balanced review from Copilot August 27, 2026 09:26

Copilot started reviewing on behalf of Amaury Chamayou (achamayou) August 27, 2026 09:26 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Comment thread src/node/rpc/file_serving_handlers.h Outdated

Comment thread tests/schema.py

Amaury Chamayou (achamayou) and others added 10 commits August 27, 2026 11:25

Merge branch 'main' into achamayou-document-committed-ledger-chunks

fcf3c16

Merge remote-tracking branch 'origin/main' into achamayou-document-co…

d234654

…mmitted-ledger-chunks

Fix committed prefix HTTP examples

0dc4cbf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into achamayou-document-co…

44d4e55

…mmitted-ledger-chunks

Move committed prefix changelog entry to 7.0.17

4aeafa5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Format committed prefix handler lambda

dac12d4

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into achamayou-document-committed-ledger-chunks

4f0fd23

Amaury Chamayou (achamayou) marked this pull request as ready for review September 21, 2026 09:58

Amaury Chamayou (achamayou) requested a review from a team as a code owner September 21, 2026 09:58

Merge branch 'main' into achamayou-document-committed-ledger-chunks

5eb6f78

Amaury Chamayou (achamayou) added the run-long-test Run Long Test job label Sep 21, 2026

Merge branch 'main' into achamayou-document-committed-ledger-chunks

64d1c17

Andrea Piccione (andpiccione) reviewed Sep 22, 2026

View reviewed changes

Comment thread doc/operations/ledger_snapshot.rst Outdated

Comment thread doc/operations/ledger_snapshot.rst Outdated

Comment thread src/host/ledger.h Outdated

Comment thread src/node/rpc/file_serving_handlers.h Outdated

Amaury Chamayou (achamayou) mentioned this pull request Sep 22, 2026

Annotate host ledger locking and remove redundant LedgerFile::file_lock and Ledger::read_cache_lock #8414

Merged

Amaury Chamayou (achamayou) and others added 5 commits September 24, 2026 09:16

Merge branch 'main' into achamayou-document-committed-ledger-chunks

2e25d4e

Merge branch 'main' into achamayou-document-committed-ledger-chunks

ba1b202

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) added the bench-ab label Oct 1, 2026

Amaury Chamayou (achamayou) and others added 3 commits October 1, 2026 17:43

Merge branch 'main' into achamayou-document-committed-ledger-chunks

87004a2

Remove cognitive complexity suppressions from unit tests (#8482)

22ab766

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) and others added 2 commits October 2, 2026 13:54

Update CI badge to reflect main branch (#8492)

49fc5c7

COSE key in CCF (#8480)

13caf3b

Co-authored-by: maxtropets <maxtropets@users.noreply.github.com> Co-authored-by: Amaury Chamayou <amchamay@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Amaury Chamayou (achamayou) removed run-long-test Run Long Test job bench-ab labels Oct 2, 2026

Merge main and move #8214 changelog to 7.0.19

b8e2d54

Keep the published 7.0.18 notes intact and synchronise the Python SDK version with the new release section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Navigation Menu

Expose recent committed ledger prefixes - #8214

Expose recent committed ledger prefixes#8214
Amaury Chamayou (achamayou) wants to merge 27 commits into
mainfrom
achamayou-document-committed-ledger-chunks

Amaury Chamayou (achamayou) commented Aug 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

Amaury Chamayou (achamayou) commented Oct 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Amaury Chamayou (achamayou) commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput (tx/s)

Latency (ms)

Memory (bytes)

Rate (ops/s)

Uh oh!

Amaury Chamayou (achamayou) commented Oct 2, 2026

Fixed-revision benchmark results

Primary metric: 2 ms blocking throughput

Other performance metrics

Custom instructions used

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Amaury Chamayou (achamayou) commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading