Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
193cc2e
feat: add direct async iteration on Dataset and KeyValueStore
Pijukatel Oct 1, 2026
2deace2
feat: make KeyValueStore record iteration overridable by storage clients
Pijukatel Oct 1, 2026
3c03eae
perf: read key-value store entries in bulk in the SQL and Redis clients
Pijukatel Oct 1, 2026
125bba2
test: drop the test for records deleted during key-value store iteration
Pijukatel Oct 2, 2026
668c170
refactor: decode None values in one place in the Redis key-value stor…
Pijukatel Oct 2, 2026
0e255a5
docs: describe the bulk iterate_entries overrides in docstrings
Pijukatel Oct 2, 2026
6c0cc3f
refactor: reject a decoding Redis client instead of suppressing the t…
Pijukatel Oct 2, 2026
95a5154
refactor: narrow Redis replies without a cast
Pijukatel Oct 2, 2026
a07f81a
docs: scope the raw-bytes requirement to the Redis key-value store cl…
Pijukatel Oct 2, 2026
1f32875
fix: bound memory of the SQL key-value store iteration with keyset pages
Pijukatel Oct 2, 2026
f809fe7
fix: retry the batched Redis value read and test the batching helper
Pijukatel Oct 2, 2026
c0c334e
test: give the batching helper test module a unique basename
Pijukatel Oct 2, 2026
f7db136
fix: skip Redis records deleted mid-iteration without a missing-value…
vdusek Oct 6, 2026
5918899
docs: correct memory and round-trip claims in KVS iteration docstrings
vdusek Oct 6, 2026
eb6a2f3
test: cover SQL iterate_entries limit ending inside a later page
vdusek Oct 6, 2026
a967540
test: rename batching test helpers and add missing test docstrings
vdusek Oct 6, 2026
db0339a
feat: make KeyValueStore async iteration yield keys like a dict
Pijukatel Oct 6, 2026
0b9af81
refactor: return the key iterator from KeyValueStore.__aiter__ like D…
vdusek Oct 6, 2026
4905199
docs: fix RedisStorageClient and KVS iteration docstrings
vdusek Oct 6, 2026
1c66c3b
test: record Redis and SQL batch calls with wrapping mocks
vdusek Oct 6, 2026
2c3dd78
test: cover records deleted mid-iteration in memory and SQL KVS clients
vdusek Oct 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: describe the bulk iterate_entries overrides in docstrings
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV
  • Loading branch information
Pijukatel and claude committed Oct 2, 2026
commit 0e255a5de1ded433a5e076a0f49326f105f537c3
9 changes: 6 additions & 3 deletions src/crawlee/storage_clients/_redis/_key_value_store_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,12 @@ async def iterate_entries(
exclusive_start_key: str | None = None,
limit: int | None = None,
) -> AsyncIterator[KeyValueStoreRecord]:
# Fetch the values in batches with a single HMGET per batch, instead of two round trips per record as the
# default implementation does. The batches are bounded by the record sizes known from the metadata, so a store
# with large values does not load too many of them at once.
"""Iterate over all the existing records in the key-value store, including their values.

The values are fetched in batches with a single HMGET call per batch, instead of two round trips per record
as the default implementation does. The batches are bounded by the record sizes known from the metadata, so
a store with large values does not load too many of them at once.
"""
batch: list[KeyValueStoreRecordMetadata] = []
batch_size = 0

Expand Down
7 changes: 5 additions & 2 deletions src/crawlee/storage_clients/_sql/_key_value_store_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -296,8 +296,11 @@ async def iterate_entries(
exclusive_start_key: str | None = None,
limit: int | None = None,
) -> AsyncIterator[KeyValueStoreRecord]:
# Read the values together with the keys in a single streamed query, instead of one query per record as the
# default implementation does. Streaming keeps a single row in memory at a time.
"""Iterate over all the existing records in the key-value store, including their values.

The values are read together with the keys in a single streamed query, instead of one query per record as
the default implementation does. Streaming keeps a single row in memory at a time.
"""
stmt = (
select(
self._ITEM_TABLE.key,
Expand Down
Loading