Non-validator crashes 6-12x/day: panic 'too many blocks to request' in nv_stream_forward_client_blocks (binary 102/103) #160

Summary

Since the binary index 102 rollout (2026-07-24), our Mainnet non-validator crashes 6–12×/day with a panic in nv_stream_forward_client_blocks:

hl-node @@ tokio_spawn_forever crashing process name=nv_stream_forward_client_blocks
Some("crashing process because tokio_spawn_forever ended, name=nv_stream_forward_client_blocks err=tokio_spawn_forever error task 53 panicked with message \"too many blocks to request\"")
hl-node @@ client_block_reader error, ending task: Bounded TrySend error: closed @@ [tcp_stream.peer_addr(): Ok(63.143.43.202:4001)]

The visor then reports visor child in bad state, restarting @@ [child_stuck: true] and restarts the child. Each cycle costs ~15–25 minutes of stream output (state-sync + client-block backfill), i.e. scattered gaps in node_fills / node_order_statuses / node_raw_book_diffs.

Setup

• hl-node 27b9f0db1df4eb807dddbb1c44ef28d4485e916f | 2026-07-25 13:31:14 +0800 (visor-managed, currently binary index 103)
• Mainnet non-validator: hl-visor run-non-validator --write-fills --write-order-statuses --write-raw-book-diffs --write-misc-events --replica-cmds-style recent-actions --serve-info
• Ubuntu 24.04.4 guest VM: 16 vCPU, 40 GB RAM, dedicated 590G volume (cache='none' io='native'; fsync latency 2–5ms)
• Gossip ports 4001/4002 reachable; --serve-evm-rpc NOT enabled

Observations

1. The panic appears to fire during post-restart catch-up when the EVM client-block backfill span exceeds an internal cap ("too many blocks to request"), which crashes the child — making the next restart's backfill span even larger. Two crashes 60–80 min apart are common once it starts.
2. The Bounded TrySend error: closed lines from client_block_reader (peers 63.143.43.0/24:4001) look like downstream symptoms of the forwarder task dying, not the cause.
3. We do not consume EVM data on this node at all — is there a supported way for a non-validator to opt out of client-block forwarding, or to raise/paginate the backfill request cap?
4. Disk/CPU are exonerated on our side (sar: await 5–33ms, steal <2%, iowait <1.4% at crash times). We rule out the local I/O regression we did have earlier (fixed).

Questions

• Is this panic known, and is a fix queued for an upcoming binary index?
• Is there a recommended mitigation (gossip config overrides, EVM state reset, request-cap tuning) for non-validators hitting this?

Happy to provide longer redacted journal excerpts.