You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.
Motivation
This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.
Design
After fetching batch N, spawn the DB write as a background task
Immediately begin fetching batch N+1
Before writing batch N+1, await completion of the batch N write
Errors in either fetch or write should still propagate correctly
Key considerations:
The reorder buffer (BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder step
The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
SSE publishing (via HeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timing
References
tidx implementation: sync/engine.rs — tick_realtime fetches batch N+1 while writing batch N
Activity