补充 Agent 可观测性:加入 OrcaReplay(运行录制与离线回放) - #13
Open
xizhuomengcontin wants to merge 1 commit into
Open
xizhuomengcontin wants to merge 1 commit into
xizhuomengcontin wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
改了什么
在
### Agent可观测性小节里加了一条:OrcaReplay。为什么这一节需要它
这一节现在只有一条「从 AI Agent 到模型推理:端到端 AI 可观测实践」,讲的是链路追踪、指标、成本这类能力——它们回答的是「这次运行花了多少钱、卡在哪一步」。
缺的一侧是「能不能把这一次运行原样拿回来」。多数 coding agent 的行为无法重放:再跑一遍得到的是另一个失败,而 provider 侧的原始请求在终端关闭后就消失了。
OrcaReplay 补的是这一段:它架在 agent 与模型 provider 之间,把一次运行的真实字节流(每一轮请求、流式响应、工具调用与结果)录成本地 trace 文件,之后
orca replay last可以在完全断网、不调用模型、不花钱的前提下按字节重放这次运行;orca compare则从任意 checkpoint 把这次录像 fork 到另一个模型上重跑,让「换个模型会不会做对」变成一个变量受控的实验。它不要求改动 agent 代码(走 base URL / fetch hook),Apache-2.0,Node CLI。对于一个列好了「怎么看 Agent」的路线图来说,「怎么看一次已经失败的 Agent 运行」是相邻但独立的一步,放在这里最合适。
Disclosure: I maintain OrcaReplay.