Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
193cc2e
feat: add direct async iteration on Dataset and KeyValueStore
Pijukatel Oct 1, 2026
2deace2
feat: make KeyValueStore record iteration overridable by storage clients
Pijukatel Oct 1, 2026
3c03eae
perf: read key-value store entries in bulk in the SQL and Redis clients
Pijukatel Oct 1, 2026
125bba2
test: drop the test for records deleted during key-value store iteration
Pijukatel Oct 2, 2026
668c170
refactor: decode None values in one place in the Redis key-value stor…
Pijukatel Oct 2, 2026
0e255a5
docs: describe the bulk iterate_entries overrides in docstrings
Pijukatel Oct 2, 2026
6c0cc3f
refactor: reject a decoding Redis client instead of suppressing the t…
Pijukatel Oct 2, 2026
95a5154
refactor: narrow Redis replies without a cast
Pijukatel Oct 2, 2026
a07f81a
docs: scope the raw-bytes requirement to the Redis key-value store cl…
Pijukatel Oct 2, 2026
1f32875
fix: bound memory of the SQL key-value store iteration with keyset pages
Pijukatel Oct 2, 2026
f809fe7
fix: retry the batched Redis value read and test the batching helper
Pijukatel Oct 2, 2026
c0c334e
test: give the batching helper test module a unique basename
Pijukatel Oct 2, 2026
f7db136
fix: skip Redis records deleted mid-iteration without a missing-value…
vdusek Oct 6, 2026
5918899
docs: correct memory and round-trip claims in KVS iteration docstrings
vdusek Oct 6, 2026
eb6a2f3
test: cover SQL iterate_entries limit ending inside a later page
vdusek Oct 6, 2026
a967540
test: rename batching test helpers and add missing test docstrings
vdusek Oct 6, 2026
db0339a
feat: make KeyValueStore async iteration yield keys like a dict
Pijukatel Oct 6, 2026
0b9af81
refactor: return the key iterator from KeyValueStore.__aiter__ like D…
vdusek Oct 6, 2026
4905199
docs: fix RedisStorageClient and KVS iteration docstrings
vdusek Oct 6, 2026
1c66c3b
test: record Redis and SQL batch calls with wrapping mocks
vdusek Oct 6, 2026
2c3dd78
test: cover records deleted mid-iteration in memory and SQL KVS clients
vdusek Oct 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: scope the raw-bytes requirement to the Redis key-value store cl…
…ient

Datasets use RedisJSON and request queues parse JSON strings, so both
work with a decoding Redis client. Only key-value store values are
binary.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wg6jZgyQp7XkvVvxuueJyV
  • Loading branch information
Pijukatel and claude committed Oct 2, 2026
commit a07f81aa92306cf3e72653f6b2aea24d881af853
3 changes: 3 additions & 0 deletions src/crawlee/storage_clients/_redis/_key_value_store_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ class RedisKeyValueStoreClient(KeyValueStoreClient, RedisClientMixin):

All operations are atomic through Redis hash operations and pipeline transactions. The client supports
concurrent access through Redis's built-in atomic operations for hash fields.

Values are stored as raw bytes, so the Redis client must not decode responses. A Redis client created with
`decode_responses=True` is rejected when a value is read.
"""

_DEFAULT_NAME = 'default'
Expand Down
5 changes: 3 additions & 2 deletions src/crawlee/storage_clients/_redis/_storage_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,9 @@ class RedisStorageClient(StorageClient):
to a Redis database v8.0+. Each storage type uses Redis-specific data structures and key patterns for
efficient storage and retrieval.

The client accepts either a Redis connection string or a pre-configured Redis client instance. The Redis client
must return raw bytes, which is the default; a client created with `decode_responses=True` is not supported.
The client accepts either a Redis connection string or a pre-configured Redis client instance. Key-value stores
hold binary values, so their client needs a Redis client that returns raw bytes, which is the default; a client
created with `decode_responses=True` works for datasets and request queues only.
Exactly one of these parameters must be provided during initialization.

Storage types use the following Redis data structures:
Expand Down
7 changes: 4 additions & 3 deletions src/crawlee/storage_clients/_redis/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,16 @@ def expect_bytes(value: bytes | str | None) -> bytes | None:
"""Narrow a Redis reply to raw bytes, rejecting a client that decodes responses.

redis-py types every reply as `bytes | str | None`, because a client created with `decode_responses=True` returns
strings. The storage clients store binary values and need the raw bytes back, so such a client is not supported.
strings. The key-value store client stores binary values and needs the raw bytes back, so it rejects such
a client.

Raises:
TypeError: If the reply is a string, i.e. the Redis client decodes responses.
"""
if isinstance(value, str):
raise TypeError(
'The Redis client returned a decoded string instead of raw bytes. The Redis storage client requires '
'a Redis client created without `decode_responses=True`.'
'The Redis client returned a decoded string instead of raw bytes. The Redis key-value store client '
'requires a Redis client created without `decode_responses=True`.'
)
return value

Expand Down
Loading