Skip to content

Commit 16b3305

Browse files
eavanvalkenburgCopilot
eavanvalkenburg
and
Copilot
authored
Python: Add Redis HASH and JSON vector stores (microsoft#8156)
* Python: Add Redis HASH and JSON vector stores Add RedisCollection and RedisStore with native HASH/JSON codecs, portable filter support, dense vector search, lifecycle management, and shared connection settings. Include public documentation, examples, and Redis integration coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Python: Fix Redis vector test typing Narrow optional deserialize results, patch the owned client close method, and keep dictionary and model collection variables distinct across the five CI test type checkers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Python: Address Redis vector connector review Revalidate index existence and schema before nonempty operations, ignore noncanonical foreign index names, honor explicit falsey overrides, clarify the Redis CI baseline, and assert independent Param sentinel identity. Add unit and real Redis HASH/JSON regressions without changing native L2 semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Python: Validate Redis vector client configuration Require strict UTF-8 encoding and database zero for vector clients. Centralize persisted namespace encoding and parsing without changing names, document supported settings, and add unit and live Unicode regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent ae0d17d commit 16b3305

12 files changed

Lines changed: 2550 additions & 45 deletions

File tree

‎.github/workflows/python-integration-tests.yml‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,49 @@ jobs:
475475
path: ./python/pytest.xml
476476
if-no-files-found: ignore
477477

478+
python-tests-redis:
479+
name: Python Integration Tests - Redis
480+
runs-on: ubuntu-latest
481+
timeout-minutes: 20
482+
permissions:
483+
contents: read
484+
services:
485+
redis:
486+
image: redis:8.0.3
487+
ports:
488+
- 6379:6379
489+
options: >-
490+
--health-cmd "redis-cli ping"
491+
--health-interval 5s
492+
--health-timeout 3s
493+
--health-retries 10
494+
env:
495+
REDIS_VECTOR_TEST_URL: redis://localhost:6379
496+
defaults:
497+
run:
498+
working-directory: python
499+
steps:
500+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
501+
with:
502+
ref: ${{ inputs.checkout-ref }}
503+
persist-credentials: false
504+
- name: Set up python and install the project
505+
uses: ./.github/actions/python-setup
506+
with:
507+
python-version: ${{ env.UV_PYTHON }}
508+
os: ${{ runner.os }}
509+
- name: Test Redis HASH and JSON vector collections
510+
run: >
511+
uv run --no-sync pytest packages/redis/tests/test_vector_store.py
512+
-m integration --timeout=120 --junitxml=pytest-redis.xml
513+
- name: Upload test results
514+
if: always()
515+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
516+
with:
517+
name: test-results-redis
518+
path: ./python/pytest-redis.xml
519+
if-no-files-found: ignore
520+
478521
# Integration test trend report (aggregates per-job JUnit XML results)
479522
python-integration-test-report:
480523
name: Integration Test Report
@@ -491,6 +534,7 @@ jobs:
491534
python-tests-foundry-hosting,
492535
python-tests-cosmos,
493536
python-tests-github-copilot,
537+
python-tests-redis,
494538
]
495539
runs-on: ubuntu-latest
496540
defaults:
@@ -554,7 +598,8 @@ jobs:
554598
python-tests-foundry,
555599
python-tests-foundry-hosting,
556600
python-tests-cosmos,
557-
python-tests-github-copilot
601+
python-tests-github-copilot,
602+
python-tests-redis,
558603
]
559604
steps:
560605
- name: Fail workflow if tests failed

‎.github/workflows/python-merge-tests.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ jobs:
3939
foundryChanged: ${{ steps.filter.outputs.foundry }}
4040
foundryHostingChanged: ${{ steps.filter.outputs.foundry_hosting }}
4141
cosmosChanged: ${{ steps.filter.outputs.cosmos }}
42+
redisChanged: ${{ steps.filter.outputs.redis }}
4243
githubCopilotChanged: ${{ steps.filter.outputs.github_copilot }}
4344
steps:
4445
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -84,6 +85,9 @@ jobs:
8485
- 'python/packages/foundry_hosting/**'
8586
cosmos:
8687
- 'python/packages/azure-cosmos/**'
88+
redis:
89+
- 'python/packages/redis/**'
90+
- 'python/packages/core/agent_framework/redis/**'
8791
github_copilot:
8892
- 'python/packages/github_copilot/**'
8993
# run only if 'python' files were changed
@@ -638,6 +642,53 @@ jobs:
638642
path: ./python/pytest.xml
639643
if-no-files-found: ignore
640644

645+
python-tests-redis:
646+
name: Python Tests - Redis Integration
647+
needs: paths-filter
648+
if: >
649+
github.event_name != 'pull_request' &&
650+
needs.paths-filter.outputs.pythonChanges == 'true' &&
651+
(github.event_name != 'merge_group' ||
652+
needs.paths-filter.outputs.redisChanged == 'true' ||
653+
needs.paths-filter.outputs.coreChanged == 'true')
654+
runs-on: ubuntu-latest
655+
timeout-minutes: 20
656+
permissions:
657+
contents: read
658+
services:
659+
redis:
660+
image: redis:8.0.3
661+
ports:
662+
- 6379:6379
663+
options: >-
664+
--health-cmd "redis-cli ping"
665+
--health-interval 5s
666+
--health-timeout 3s
667+
--health-retries 10
668+
env:
669+
REDIS_VECTOR_TEST_URL: redis://localhost:6379
670+
defaults:
671+
run:
672+
working-directory: python
673+
steps:
674+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
675+
- name: Set up python and install the project
676+
uses: ./.github/actions/python-setup
677+
with:
678+
python-version: ${{ env.UV_PYTHON }}
679+
os: ${{ runner.os }}
680+
- name: Test Redis HASH and JSON vector collections
681+
run: >
682+
uv run --no-sync pytest packages/redis/tests/test_vector_store.py
683+
-m integration --timeout=120 --junitxml=pytest-redis.xml
684+
- name: Upload test results
685+
if: always()
686+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
687+
with:
688+
name: test-results-redis
689+
path: ./python/pytest-redis.xml
690+
if-no-files-found: ignore
691+
641692
# Integration test trend report (aggregates per-job JUnit XML results)
642693
python-integration-test-report:
643694
name: Integration Test Report
@@ -654,6 +705,7 @@ jobs:
654705
python-tests-foundry-hosting,
655706
python-tests-cosmos,
656707
python-tests-github-copilot,
708+
python-tests-redis,
657709
]
658710
runs-on: ubuntu-latest
659711
defaults:
@@ -715,6 +767,7 @@ jobs:
715767
python-tests-foundry-hosting,
716768
python-tests-cosmos,
717769
python-tests-github-copilot,
770+
python-tests-redis,
718771
]
719772
steps:
720773
- name: Fail workflow if tests failed

‎python/packages/core/agent_framework/redis/__init__.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,19 @@
66
- ``agent-framework-redis``
77
88
Supported classes:
9+
- RedisCollection
910
- RedisContextProvider
1011
- RedisHistoryProvider
12+
- RedisSettings
13+
- RedisStore
1114
"""
1215

1316
import importlib
1417
from typing import Any
1518

1619
IMPORT_PATH = "agent_framework_redis"
1720
PACKAGE_NAME = "agent-framework-redis"
18-
_IMPORTS = ["RedisContextProvider", "RedisHistoryProvider"]
21+
_IMPORTS = ["RedisCollection", "RedisContextProvider", "RedisHistoryProvider", "RedisSettings", "RedisStore"]
1922

2023

2124
def __getattr__(name: str) -> Any:

‎python/packages/core/agent_framework/redis/__init__.pyi‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,17 @@
11
# Copyright (c) Microsoft. All rights reserved.
22

33
from agent_framework_redis import (
4+
RedisCollection,
45
RedisContextProvider,
56
RedisHistoryProvider,
7+
RedisSettings,
8+
RedisStore,
69
)
710

811
__all__ = [
12+
"RedisCollection",
913
"RedisContextProvider",
1014
"RedisHistoryProvider",
15+
"RedisSettings",
16+
"RedisStore",
1117
]

‎python/packages/redis/AGENTS.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,33 @@ Redis-based storage for agent threads and context.
66

77
- **`RedisHistoryProvider`** - Persistent chat history provider using Redis
88
- **`RedisContextProvider`** - Context provider with Redis-backed retrieval
9+
- **`RedisSettings`** - TypedDict connection settings for vector stores, resolved with core `load_settings` from
10+
explicit URL overrides, an optional .env file, or `REDIS_URL`. URLs use `SecretString` to mask credentials.
11+
- **`RedisCollection` / `RedisStore`** - Experimental generic vector storage and search over native HASH and JSON
12+
documents. `_vector_store.py` contains settings, connection/codec/schema/filter helpers, the collection, and
13+
the store, separate from the preexisting history/context providers. `_create_client` shares AF settings resolution
14+
and requires database 0, strict UTF-8 encoding, RESP2, and binary responses for URL-created and borrowed clients.
15+
`_RedisNamespaceNames` owns the persisted index/document prefix format and canonical index-name parsing;
16+
`RedisCollection._prepare_filter` validates support and translates portable filters to native Redis queries.
17+
Requires Redis Search with INDEXMISSING/INDEXEMPTY support.
18+
Nonempty CRUD/search operations recheck index existence and schema. These checks observe completed external
19+
lifecycle changes but are not atomic with subsequent operations.
20+
See README for the per-type native filter restrictions, HASH null rejection, and distance units.
21+
22+
## Vector connector tests
23+
24+
From `python/`, run the connector's deterministic unit tests with:
25+
26+
```bash
27+
uv run --no-sync pytest packages/redis/tests/test_vector_store.py packages/redis/tests/test_vector_settings.py -m "not integration"
28+
```
29+
30+
Set `REDIS_VECTOR_TEST_URL` to an explicitly disposable Search/JSON instance
31+
and select `-m integration` to run integration tests in `tests/test_vector_store.py`.
32+
Automated live tests use Redis 8.0.3, pinned in both CI workflows. A configured but insufficient server fails;
33+
only an unset URL skips them. Tests create unique namespaces and clean only
34+
their own indexes/keys, never `FLUSHALL`. Coverage includes both formats and
35+
1000-record batches with two 1536-dimensional vectors without embedding API calls.
936

1037
## Usage
1138

0 commit comments

Comments
 (0)