Skip to content

fix(txpool): expire validity transactions after published flashblocks - #5477

Open
fahimahmedx wants to merge 6 commits into
mainfrom
fahim/fix-flashblock-tx-expiry
Open

fahimahmedx wants to merge 6 commits into
mainfrom
fahim/fix-flashblock-tx-expiry

Conversation

@fahimahmedx

@fahimahmedx fahimahmedx commented Oct 2, 2026 •

Copy link
Copy Markdown

Summary

  • Expire validity transactions at their inclusive flashblock-index deadline immediately after the flashblock is successfully published, instead of waiting until the full block seals.
  • Coordinate publication and txpool admission so an expired same-nonce transaction cannot continue to block an unbumped replacement; retain block-only deadlines until their last valid block commits and avoid eviction on failed publication.
  • Cover protocol and sidecar admission, expiry ordering, and same-nonce reuse with pool tests and an in-process builder/RPC/WebSocket regression test.

Review follow-ups

Addresses all three comments from Brian Bland on #5436:

  • Replace the bare (u64, u64) expiry keys with a documented ExpiryPosition struct, ordered by block number and then flashblock index (comment).
  • Tighten the regression test's 30-second waits to 1 second for the first flashblock and 3 seconds for the remainder of the configured 2-second block build (comment).
  • Drain block deadlines inclusively at the committed head using the next-block cutoff, correcting the off-by-one boundary described in fix(txpool): evict validity txs once their last valid block is committed #5333. Test the canonical-maintenance boundary and maximum block number without overflow (comment).

GitHub review follow-up: reset the orphaned publication watermark on canonical reorgs, while preserving it during ordinary commits that may lag newer flashblock publication. Regression tests cover protocol/sidecar admission and expiry after lower-height or lower-index rebuilds, and preserve active deadline protection across delayed ordinary commits (comment).

Verification

  • cargo test -p base-execution-txpool --lib (298 passed, 3 ignored)
  • cargo test -p base-builder-core --test transaction_events -- --test-threads=1 (3 passed)
  • Same-nonce flashblock regression repeated 5 additional times at the initial PR head (all passed)
  • New canonical-reorg regression fails without the fix and passes with it
  • cargo clippy -p base-execution-txpool -p base-builder-core --lib --tests -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

The end-to-end test waits until flashblock 1 is published, submits an unbumped same-nonce transaction, and verifies it is included in the current L2 block before that block seals. It does not assert a particular later flashblock index.

Replaces #5436 with the head branch hosted in base/base at fahim/fix-flashblock-tx-expiry rather than in the fork.

Generated with Toshi

@fahimahmedx
fahimahmedx requested a review from BrianBland October 2, 2026 15:51
@cb-heimdall

Copy link
Copy Markdown
Collaborator

🟡 Heimdall Review Status

Requirement Status More Info
Reviews 🟡 0/1
Denominator calculation
Show calculation
1 if user is bot 0
1 if user is external 0
2 if repo is sensitive 0
From .codeflow.yml 1
Additional review requirements
Show calculation
Max 0
0
From CODEOWNERS 0
Global minimum 0
Max 1
1
1 if commit is unverified 0
Sum 1

@depot-code-access

depot-code-access Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

✅ All benchmarks green — 14 within ±2% (deterministic instruction counts). View run

Benchmark details (14)
Benchmark Base (target) Head (this PR) Δ instructions
batch_queue/drain/drain_cached_span_batches 242,027 242,027 +0.0%
batch_transaction/encode_in_place/encode_in_place 4,199,759 4,199,759 +0.0%
batch_transaction/temporary_frame_buffers/temporary_frame_buffers 8,408,350 8,408,350 +0.0%
flashblock_decode/decode/brotli 3,296,484 3,296,484 +0.0%
flashblock_decode/decode/plain_json 2,280,194 2,280,194 +0.0%
flz/compress_len/real_contract_call 43,148 43,148 +0.0%
flz/compress_len/synthetic_0 38,205 38,205 +0.0%
flz/compress_len/synthetic_1 54,682 54,682 +0.0%
flz/compress_len/synthetic_2 147,976 147,976 +0.0%
flz/data_gas 43,059 43,059 +0.0%
flz/tx_estimated_size 43,056 43,056 +0.0%
frame_parse/decode/single_4kib 1,031 1,031 +0.0%
frame_parse/parse_frames/few_large 1,053,062 1,053,062 +0.0%
frame_parse/parse_frames/many_small 154,763 154,763 +0.0%

Comment thread crates/execution/txpool/src/block_expiry.rs
Comment on lines +925 to +928
let _admission_guard = self.protocol_admission_lock.lock();
let result = publish()?;
let expired = self.expire_published_locked(block_number, flashblock_index);
Ok((result, expired))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The protocol_admission_lock is now held for the entire duration of the publish closure (websocket serialization + send), whereas before this PR publish() ran without it. During this window, all protocol and sidecar pool admission (add_validated_protocol_transaction, add_validated_sidecar_transaction, add_transaction_and_subscribe) blocks on the same mutex.

If the websocket publish experiences backpressure or latency (e.g., slow consumer, large flashblock serialization), pool admission stalls for all submitters until it completes. This is the intended tradeoff for atomicity, but worth confirming that the publish path cannot block long enough to cause transaction submission timeouts or RPC deadline violations under production load.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lock intentionally serializes admission with publication and expiry so an observed deadline frees the same-nonce slot before admission proceeds. WebSocketPublisher::publish (crates/builder/publish/src/publisher.rs) performs synchronous JSON serialization, a ring-buffer update, and tokio broadcast::Sender::send, not a socket send. Per-client network sends run separately in BroadcastLoop; replay snapshots release the ring-buffer read lock before awaiting sends. Slow consumers therefore do not backpressure this admission lock. Serialization still contributes to lock hold time; this inspection does not establish production latency bounds, but it does not identify a timeout regression requiring a code change.

🤖 Reply generated with Toshi

Comment thread crates/builder/core/src/flashblocks/payload.rs
@depot-code-access

depot-code-access Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Acceptance Tests: ✅ Passed

3/3 checks passed · Checks & reports

Acceptance report
  • smoke / chain-identity — passed: chain identity matched
    • Expected: 84538453
    • Observed: 84538453
  • smoke / production — passed: head advanced
    • Expected: {"minimum_blocks":3}
    • Observed: {"delta":3,"from":10,"last":13,"required_delta":3}
  • smoke / validator-converges — passed: heads converged on a common canonical block
    • Expected: {"common_height_hash":true,"max_lag_blocks":5}
    • Observed: {"common_height":13,"hash":"0xfa27ddb064d9a68f100e1862c6cf96884e546b9797e71ba34eaefd09d8d0d3cd","high":13,"lag":0,"low":13,"maximum_lag":5}

}) {
return 0;
}
*published = Some(position);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

published_flashblock is set to Some(position) here but is never reset to None when a new block starts (e.g., in on_canonical_state_change). This works today because expire_published_locked uses a monotonic >= guard at line 443 and ensure_not_published_expired uses is_batch_expired which correctly handles cross-block semantics (a stale published position from block N won't falsely reject transactions targeting block N+1).

However, the stale published_flashblock from a previous block means ensure_not_published_expired performs a non-trivial is_batch_expired evaluation on every admission for the rest of the node's lifetime, even when no flashblocks are being built (e.g., sequencer downtime). More importantly, the implicit invariant that the stale position is always safe relies on is_batch_expired's block-vs-flashblock interaction. An explicit reset in on_canonical_state_change (e.g., *self.published_flashblock.write() = None; under the admission lock) would make this simpler to reason about and remove the coupling.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The retained position is intentional across ordinary block transitions: canonical maintenance can arrive after flashblocks for the next block have already been published. An unconditional reset would erase that newer deadline protection and admit already-expired submissions.

I did reproduce a correctness issue on reorgs: a position from the old branch rejects otherwise-valid rebuilt-block deadlines and suppresses expiry at lower block/index positions. The follow-up resets the watermark only for PoolUpdateKind::Reorg, while holding the admission lock. Regression tests cover protocol and sidecar admission/expiry after both a lower-height rebuild and a same-height flashblock-index restart, plus preservation of newer publication state during an ordinary commit. The reorg regression fails before the fix.

No performance bottleneck was demonstrated for retaining the position during ordinary commits, so I have not added a broader reset or a performance refactor.

🤖 Reply generated with Toshi

Resolve the overlap with #5333 by keeping inclusive expiry in BlockExpiryIndex and the canonical-maintenance boundary regression. Refresh the stale tx-forwarding README digest inherited from main after verifying its existing summary.
Keep main’s updated tx-forwarding docs-index summary and matching generated indexes. Preserve the flashblock expiry and reorg handling unchanged.
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Review Summary

This PR correctly coordinates flashblock publication with pool admission and deadline eviction, addressing the race where an expired same-nonce transaction could block its unbumped replacement. The design is sound: publish_and_expire atomically publishes the flashblock and sweeps expired deadlines under the same protocol_admission_lock, and ensure_not_published_expired gates admission against the watermark. The ExpiryPosition struct is a clear improvement over the previous bare tuple.

Findings from prior review passes (already commented inline):

  1. protocol_admission_lock held during websocket publish — pool admission stalls if the publish path experiences backpressure.
  2. Permanently rejected txs deferred to after publish — cancelled builds leave known-expired transactions in the pool until the next build cycle.
  3. remove_from_position visibility — pub on an internal helper.
  4. published_flashblock lifetime — partially addressed by the reorg reset in ca0306dc; PoolUpdateKind::Commit intentionally preserves the watermark (tested).

Additional observations (not warranting inline comments):

  • drain_before (block_expiry.rs:94) is also pub but only called internally by drain_expired and drain_published — same pattern as remove_from_position. Consider making both fn (private).
  • The off-by-one correction in drain_expired (now using committed_block directly instead of committed_block + 1 as the boundary) is correctly tested, including the u64::MAX overflow edge case.
  • Lock ordering is consistent throughout: publish_guard → protocol_admission_lock → published_flashblock/block_expiry/nonce_pool. No deadlock risk.
  • Error propagation from publish_and_expire via ? matches the pre-existing behavior — websocket publish failures already aborted the flashblock loop before this PR. No new block-production halt surface.
  • The FlashblockExpiry trait design cleanly separates the pool's publication coordination from the builder, with a closure-based API that preserves the pool-unchanged invariant on publish failure.

No critical or high-severity findings. The existing inline comments cover the substantive review points.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants