Skip to content

Add Redis-backed checkpoint storage for distributed workflow persistence #2401

Open
Feature
#9253
Open
Add Redis-backed checkpoint storage for distributed workflow persistence#2401
Feature
#9253
Assignees
peibekwe
Labels
.NETUsage: [Issues, PRs], Target: .NetpythonUsage: [Issues, PRs], Target: PythonquestionUsage: [Issues, PRs], Target: all issues or PRs where additional context is needed from the authorworkflowsUsage: [Issues, PRs], Target: Workflows

Description

@dotnetspark
Yadel Lopez (dotnetspark)
opened on Nov 23, 2025

Problem

The Agent Framework currently supports only two checkpoint storage implementations:

  • In-memory (ephemeral, not durable)
  • Flat file (local-only, not scalable)

These options are useful for development and simple scenarios, but they limit adoption in production environments. Workflows often need to be resumed across processes, pods, or machines, which requires a distributed and durable checkpoint store.

Proposal

Introduce a RedisCheckpointStorage implementation that persists workflow checkpoints in Redis. This will enable:

  • Distributed resilience: Agents can resume workflows across multiple hosts or containers.
  • Durability with TTL: Checkpoints can be retained for configurable lifetimes, preventing stale state.
  • Scalability: Redis is widely used in cloud-native architectures and integrates well with Kubernetes, Azure Cache for Redis, and other managed services.
  • Cross-language parity: Implementations will be provided in both Python and C#, ensuring developers can use Redis-backed persistence regardless of language choice.

Implementation Plan

  • Python
    • Create RedisCheckpointStorage class, mirroring InMemoryCheckpointStorage and FileCheckpointStorage.
    • Use aioredis for async operations.
    • Support configurable options: Redis URL, key prefix, TTL.
  • C#
    • Create RedisCheckpointStore implementing ICheckpointStore.
    • Integrate with CheckpointManager for workflow runs.
    • Use StackExchange.Redis for connection pooling and resilience.
  • Shared Features
    • Serialize checkpoints as JSON for readability and debugging.
    • Optional support for binary formats (MessagePack/Protobuf) for performance.
    • Add unit and integration tests (docker-compose Redis).
    • Provide documentation and sample workflows demonstrating checkpoint persistence and resumption.

Impact

This enhancement will:

  • Expand checkpoint storage options beyond local-only solutions.
  • Improve production-readiness of the framework.
  • Encourage community adoption by aligning with a widely used, battle-tested backend.
  • Lay groundwork for additional backends (e.g., Azure Blob Storage, PostgreSQL, SQLite) via the same pluggable pattern.

Activity

  1. Mark Wallace (markwallace-microsoft)
    added
    .NETUsage: [Issues, PRs], Target: .Net
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 23, 2025
  2. github-actions
    changed the title [-]Add Redis-backed checkpoint storage for distributed workflow persistence[/-] [+].NET: Add Redis-backed checkpoint storage for distributed workflow persistence[/+] on Nov 23, 2025
  3. github-actions
    changed the title [-].NET: Add Redis-backed checkpoint storage for distributed workflow persistence[/-] [+]Python: Add Redis-backed checkpoint storage for distributed workflow persistence[/+] on Nov 23, 2025
  4. Yadel Lopez (dotnetspark)
    changed the title [-]Python: Add Redis-backed checkpoint storage for distributed workflow persistence[/-] [+]Add Redis-backed checkpoint storage for distributed workflow persistence[/+] on Nov 23, 2025
  5. Mark Wallace (markwallace-microsoft)
    removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 28, 2025
  6. Mark Wallace (markwallace-microsoft)
    assigned
    Ben Thomas (alliscode)
    on Nov 28, 2025
  7. Mark Wallace (markwallace-microsoft)
    added the
    Feature
    issue type on Nov 28, 2025
  8. Mark Wallace (markwallace-microsoft)
    added this to Agent Frameworkon Nov 28, 2025
  9. Eduard van Valkenburg (eavanvalkenburg)
    added
    questionUsage: [Issues, PRs], Target: all issues or PRs where additional context is needed from the author
    on Jun 2, 2026
  10. Evan Mattson (moonbox3)
    added
    workflowsUsage: [Issues, PRs], Target: Workflows
    on Jun 18, 2026
  11. chetantoshniwal
    assigned
    Peter Ibekwe (peibekwe)
    and unassigned
    Ben Thomas (alliscode)
    on Aug 18, 2026

  12. AwaisSandhu10 commented on Oct 5, 2026

    @AwaisSandhu10
    AwaisSandhu10
    on Oct 5, 2026

    Hi, is the Python side of this still open to contribution? I'd be interested in implementing the Redis checkpoint storage and adding the corresponding tests, following the existing Cosmos implementation.

    I saw the earlier .NET attempt in #2799 was closed as outdated. I'd start with Python and can take on the .NET side afterward if that's still needed.

  13. Murat7Ay commented on Oct 10, 2026

    @Murat7Ay
    Murat Ay (Murat7Ay)
    on Oct 10, 2026

    I'd like to pick up the .NET side of this, since #2799 was closed as outdated.

    Plan: a new Microsoft.Agents.AI.Redis package (alpha) with RedisCheckpointStore : JsonCheckpointStore on StackExchange.Redis. It keeps a per-session sorted set scored by a commit counter, so RetrieveIndexAsync keeps commit order even for checkpoints in the same second. Writes are atomic through one Lua script, keys are hash-tagged so it works on Redis Cluster, and there is an optional per-session TTL. It comes with unit tests, tests against a real Redis that skip when none is available (like the Cosmos emulator tests), and a 03-workflows/Checkpoint/CheckpointWithRedis sample. It also addresses the Copilot comments from #2799.

    Peter Ibekwe (@peibekwe), is a separate Microsoft.Agents.AI.Redis package the right home, or would you prefer it in Microsoft.Agents.AI.Valkey? I've opened it as a draft PR: #9253. Happy to adjust the location or scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

.NETUsage: [Issues, PRs], Target: .NetpythonUsage: [Issues, PRs], Target: PythonquestionUsage: [Issues, PRs], Target: all issues or PRs where additional context is needed from the authorworkflowsUsage: [Issues, PRs], Target: Workflows

Type

Feature

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions