Skip to content

Python: Bound harness file memory with quotas and retention - #8970

Open
quifox (quifox) wants to merge 3 commits into
microsoft:mainfrom
quifox:feat/file-memory-quota
Open

Python: Bound harness file memory with quotas and retention#8970
quifox (quifox) wants to merge 3 commits into
microsoft:mainfrom
quifox:feat/file-memory-quota

Conversation

@quifox

@quifox quifox (quifox) commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation & Context

The hosted claw sample persists file memory without capacity or lifecycle controls. Repeated authorized writes and abandoned scopes can exhaust shared disk space. This change adds opt-in SDK limits and explicit retention management, and configures finite limits in the hosted sample.

Description & Review Guide

  • What are the major changes? Add UTF-8 byte, regular-file count, and per-root byte limits to the builtin stores. Add an experimental filesystem retention manager with shared capacity accounting, per-memory idle expiry, periodic cleanup, process locks, atomic replacements, and request-lifetime protection. Integrate it into the hosted sample, including full stream consumption and close paths.
  • What is the impact of these changes? Omitted SDK limits remain unlimited; providers without a retention manager retain existing behavior. Legacy unregistered files count toward capacity but are not automatically enrolled for expiry. Disabling TTL stops new expiry decisions while completing deletions already marked for cleanup. Normal restarts preserve durable deadlines; explicitly resuming a disabled TTL grants remaining registered, completed records a fresh lifetime. Managed retention requires the builtin filesystem backend and cooperating writers on one host; external backends retain their existing contract. Quota and interrupted-write failures are reported without identity values or local paths.
  • What do you want reviewers to focus on? Filesystem accounting and manifest I/O run in worker threads under the manager lock, including during cancellation. Review concurrent accounting, interrupted update/deletion recovery, TTL transitions, and stream protection. The sample's initial limits (1 MiB/file, 100 regular files and 10 MiB/root, 256 MiB shared, 30-day TTL) are configurable starting points. Local validation with core 1.20.0, filelock 3.32.7, and msgspec 0.22.0: core unit tests (8,394 passed, 21 skipped, 2 expected failures), five core test type checkers, source Pyright, changed sample type checks, Ruff, all pre-commit hooks including Bandit, wheel/sdist build, and lockfile verification. A focused runtime probe passed with the minimum filelock version on an earlier snapshot. Native Windows/macOS execution, live Azure deployment, network filesystems, and power-loss durability were not validated.

Related Issue

Fixes #8900

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

@quifox
Python: Bound harness file memory with quotas and retention
4385ebd 
Add opt-in quotas, shared retention management, and server-owned cleanup
with request lifetime protection. Preserve SDK defaults and cover limits,
TTL transitions, concurrent operations, interruptions, and hosted streams.
Copilot AI balanced review requested due to automatic review settings October 2, 2026 11:08
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 11:08 — with GitHub Actions Active
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 11:08 — with GitHub Actions Active
@agent-framework-automation agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python labels Oct 2, 2026
Copilot AI reviewed Oct 2, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Managed description cleanup, disabled-TTL deletion recovery, quota error handling, and synchronous accounting need correction.

Review effort: Balanced
Findings: 4 Medium severity · 1 Low severity

Open (5)
What changed in this PR

Adds bounded file-memory storage and experimental retention management to the Python SDK and hosted claw sample.

Changes:

  • Adds per-file, file-count, root, and shared quotas.
  • Adds TTL retention, GC, locking, repair, and request protection.
  • Configures hosted defaults and extensive lifecycle tests.
File Description
python/​uv.lock Locks the new filelock dependency.
python/​packages/​core/​pyproject.toml Declares filelock.
python/​packages/​core/​AGENTS.md Documents quotas and retention.
python/​packages/​core/​agent_framework/​__init__.py Exports the retention manager.
python/​packages/​core/​agent_framework/​__init__.pyi Types the new export.
_harness/​_file_store_limits.py Implements quota validation and accounting.
_harness/​_file_access.py Enforces built-in store limits.
_harness/​_file_memory.py Integrates managed retention operations.
_harness/​_file_memory_retention.py Implements retention, GC, and shared quotas.
claw_step04_production_ready/​agent.py Accepts managed memory providers.
claw_step04_production_ready/​hosted.py Configures quotas and request protection.
claw_step04_production_ready/​README.md Documents hosted capacity and retention.
test_harness_file_store_limits.py Tests store quotas.
test_harness_file_memory.py Tests quota failure behavior.
test_harness_file_memory_retention.py Tests retention and recovery.
test_harness_hosted_memory_lifetime.py Tests hosted stream protection.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread python/packages/core/agent_framework/_harness/_file_memory.py Outdated
Comment thread python/packages/core/agent_framework/_harness/_file_memory_retention.py
Comment thread python/packages/core/agent_framework/_harness/_file_memory_retention.py Outdated
Comment thread python/packages/core/agent_framework/_harness/_file_memory_retention.py
Comment thread python/packages/core/agent_framework/_harness/_file_memory_retention.py
@quifox
Python: Address file-memory retention review findings
d67a8b3 
Complete confirmed deletions while TTL is disabled, clear omitted descriptions
on overwrite, and report list/quota errors consistently. Move filesystem
accounting and lifecycle I/O off the event loop while retaining coordination
through cancellation. Cover legacy limits, shared descriptions, deletion
recovery, quota diagnostics, and request cancellation with regression tests.
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 13:00 — with GitHub Actions Active
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 13:05 — with GitHub Actions Active
@quifox
Python: Sync file-memory retention with main
945a9ae 
Preserve the 1.20.0 release metadata and updated msgspec bounds while retaining
the filelock dependency used by managed retention. Verify the merged lockfile
and rerun core validation with the resolved dependencies.
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 13:47 — with GitHub Actions Active
@quifox
quifox (quifox) deployed to github-app-auth October 2, 2026 13:50 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
github-app-auth — 945a9aed Deployed Oct 2, 2026 by quifox via add_label #24296
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) Awaiting requested review from eavanvalkenburg eavanvalkenburg is a code owner
@SergeyMenshykh SergeyMenshykh Awaiting requested review from SergeyMenshykh SergeyMenshykh is a code owner
@rogerbarreto Roger Barreto (rogerbarreto) Awaiting requested review from rogerbarreto rogerbarreto is a code owner
@westey-m westey (westey-m) Awaiting requested review from westey-m westey-m is a code owner
@moonbox3 Evan Mattson (moonbox3) Awaiting requested review from moonbox3 moonbox3 is a code owner
@jpalvarezl Jose Alvarez (jpalvarezl) Awaiting requested review from jpalvarezl jpalvarezl is a code owner
@sophia-ramsey sophia-ramsey Awaiting requested review from sophia-ramsey sophia-ramsey is a code owner
@TaoChenOSU Tao Chen (TaoChenOSU) Awaiting requested review from TaoChenOSU TaoChenOSU is a code owner
@giles17 Giles Odigwe (giles17) Awaiting requested review from giles17 giles17 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Python: Bound hosted harness file memory with quotas and retention

2 participants

@quifox