Skip to content

[BUG] aiagent: two-phase write proposal silently lost when the model restates the confirmation copy without invoking the tool — a later 'confirm' never persists #3369

Description

@SummerSweety

Describe the issue

In the AI assistant chat (agent), the update_* tools (e.g. update_alert_rule) use a two-phase propose/confirm flow: the first tool call computes a change set and returns an approval interrupt (a pending proposal is persisted to the message extra and to Redis), and a later user reply "confirm" is resolved deterministically by replaying the tool's apply leg (router_ai_interrupt.go tryResumePending), which only triggers when the previous message carries extra.pending.

Problem: the model can produce the confirmation copy ("About to modify ... reply 'confirm' to apply") in plain reply text without actually invoking the tool. In that case no pending proposal is created, so when the user replies "confirm", tryResumePending finds prevMsg.Extra.Pending == nil and silently falls back to the normal agent flow. The change is never persisted, yet the agent is free to fabricate a "confirmed and applied" receipt because the runtime gives it no grounding. The user believes the change succeeded when it did not.

Observed in a production fork with update_alert_rule:

  1. User asks to change an alert rule threshold.
  2. The model replies with the exact confirmation copy ("以上改动尚未写入。回复「确认」立即生效…") but never calls update_alert_rule → no pending proposal (the conversation transcript shows zero tool calls).
  3. User replies "确认".
  4. No pending → agent flow resumes; the model claims the change was applied, and may even fabricate the "✅ 已确认并写入" receipt.
  5. The rule's threshold stays unchanged; the user must repeat the request multiple times until one turn actually calls the tool.

Also observed: when the model does call the tool but passes the full baked expression (e.g. metric{...} < 20000) as the prom_ql argument of update_alert_rule, rebuildBakedPromQL treats it as the base and appends the current operator+threshold, producing an invalid query like metric{...} < 20000 < 40000 that is then shown to the user as the proposed change.

Expected behavior

A user reply "confirm" should either deterministically apply a real pending proposal, or be rejected with a clear message that there is nothing pending — never silently fall into the agent flow where the model can report a success that did not happen. Invalid/self-contradictory tool arguments should be rejected by the tool rather than concatenated into a bad expression.

Steps to reproduce

  1. Start an agent chat in n9e with the AI assistant feature enabled.
  2. Ask the assistant to modify an existing alert rule (e.g. change its threshold).
  3. Observe that the model may print the standard proposal confirmation copy without an actual tool call (verify via the conversation transcript / pending state).
  4. Reply with the confirmation word.
  5. The rule is not changed, and depending on the model the assistant may still say it applied.

Environment

  • nightingale version: current main (v9.1.1 series)
  • AI assistant chat with tool-use enabled; update_alert_rule two-phase write gate
  • LLM: observed with a DeepSeek-class chat model; likely model-agnostic

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions