Python: [Bug]: BackgroundAgentsProvider retains per-session runtime (tasks, child sessions) forever #7385

Description

1. Problem

BackgroundAgentsProvider keeps non-serializable per-session runtime in self._runtime: dict[str, _RuntimeState], keyed by session_id (_get_runtime, ~307-312). Entries are inserted on first use and never removed — there is no eviction, no TTL, no hook for the host to signal "this session is over".

The provider is designed for exactly the deployment where this hurts: a single agent (hence a single provider instance) serving many sessions concurrently, each session's state carried by its own AgentSession. Any long-lived host that follows that pattern leaks one _RuntimeState per session that ever started a background task, for the process lifetime.

2. What each entry retains

• background_sessions — full child AgentSession objects, one per task, each carrying the background agent's entire message transcript for that task. A grounded background run realistically retains hundreds of KB (measured in our deployment: 80 KB of message text retains ~39.5 KiB of heap; real tasks run 200 KB–2 MB).
• in_flight_tasks — asyncio.Task objects. A completed task's entry is released only by background_agents_clear_completed_task, i.e. only if the model follows the instruction to call it — there is no structural cleanup. Tasks still running when a session is abandoned are never cancelled: orphan work that keeps executing (and keeps calling tools) with results nobody will read.

3. Secondary hazard: naive eviction is unsafe

in_flight_tasks is the only strong reference to the running asyncio.Tasks. If a host works around the leak by dropping _runtime entries (or the provider instance), a pending task becomes garbage-collectable mid-flight ("Task was destroyed but it is pending") while its side effects half-complete. Any fix needs cancellation + await semantics, not just deletion — which is why this belongs upstream rather than in each host.

Code Sample

Error Messages / Stack Traces

Package Versions

agent-framework-core: 1.11.0

Python Version

No response

Additional Context

4. Requested end state

One of, in preference order:

1. An eviction API with lifecycle semantics — e.g. async BackgroundAgentsProvider.release_session(session_id, *, cancel_running=True) that cancels + awaits in-flight tasks, drops child sessions, and removes the entry; hosts call it when a session ends or from their own LRU/TTL policy.
2. A bounded per-session registry (LRU with a documented cap) with the same cancellation semantics on eviction, on by default.
3. At minimum: structural release of a task's background_sessions entry when the task is finalized or cleared, so retention no longer depends on the model calling background_agents_clear_completed_task.

5. Reproduction sketch

provider = BackgroundAgentsProvider([worker_agent])
agent = create_harness_agent(client=client, background_agents=[worker_agent])
for i in range(N):                       # N distinct sessions, e.g. web conversations
    session = agent.create_session()
    # one run that calls background_agents_start_task + wait + get_results
    await agent.run(prompt, session=session)
# provider._runtime now holds N _RuntimeState entries, each with the child session's
# full transcript; nothing ever removes them.

Origin

Observed via create_harness_agent(background_agents=[...]) on a shared-agent web host (one agent per model, all conversations). Our runtime-registered sibling provider (ats.subagents, issue #232) ported the same _runtime pattern and has the same defect, tracked separately on our side; we would prefer to adopt an upstream mechanism over maintaining a bespoke eviction path in both places.