Agent Systems · quality infrastructure
Agent Observability & Evaluation Flywheel
Impact
Engineers can inspect selected answers and manual replays while offline golden cases prevent known regressions from merging.
- Answer quality checks
- Regression tests in CI
- Engineer-reviewed releases
01 · Problem & operating context
A plausible answer could still be wrong in the way that matters.
A Teams answer can read well while missing the expected answer, drawing from the wrong context, or exposing a tool-path problem. Without a trace and a way to compare it, the team cannot tell whether it is an isolated case or the beginning of a release regression.
The job was to make a selected answer explainable, catch known failures before merge, and leave a human enough context to decide whether a change is ready.
02 · My role & ownership
I built a review path around real cases.
I instrumented eligible Teams-answer scoring, connected expert references as the comparison input for that scoring, kept manual replay available, and added offline golden cases to the CI merge path.
The wider runtime and release platform are shared context. My work makes the handoff from an observed answer to a controlled engineering decision inspectable.
- Designed
- Eligible answer scoring with expert-reference comparison, plus separate replay and approval.
- Built
- Offline golden-case regression checks and CI merge gate.
- Connected
- Eligible Teams answers, expert references, and engineer-reviewed changes.
Connected eligible answer scoring to expert-reference comparison while separating replay and release approval.
Built the offline golden-case CI merge gate and evaluation workflow.
Connected Teams answers, expert references, and engineer-reviewed changes.
The flywheel operates within the wider agent platform and its shared release process.
03 · System architecture
Online answers and offline regression gates answer different questions.
Eligible Teams answers enter the online scoring path, where expert references provide the comparison input. The comparison and trace show where a selected production answer diverged from the expected result.
Offline golden cases run separately in CI, where a known regression can block a merge. When a case needs explanation, an engineer can replay it manually, inspect the trace and comparison, then return a proposed change to review before release.
- Score
- Eligible Teams answers are scored against their evaluation criteria.
- Compare
- Expert references support eligible Teams-answer scoring; offline golden cases are CI regressions.
- Decide
- Manual replay and engineer review gate a proposed change before release.
04 · Decisions & trade-offs
The comparison method should match the decision.
Expert references belong with eligible answer scoring because they supply the standard for that comparison. Offline golden cases belong in CI because they protect a known failure from reaching merge. Combining them into one score would hide which question a failure had actually answered.
Manual replay stays deliberate because it is a deeper investigation, not a background ritual for every trace. That costs engineer time, but it gives a trace-level failure enough context to support a targeted change.
Score eligible answers with expert references, keep manual replay on demand, and use offline golden cases as the CI merge gate.
Treat every trace as one automatic evaluation and release signal.
Each evidence source supports a different review decision and should remain inspectable.
Engineers must choose and maintain the right path for each evaluation need.
05 · Failure modes & recovery
A failing case becomes a review question, not an automatic change.
A failed golden case blocks the CI merge gate. An eligible answer score, informed by its expert reference, gives the engineer a specific response to inspect, but it does not silently become a release decision.
The engineer can replay the relevant case, make a targeted change, and re-run the appropriate gate. Approval remains explicit, so a failure improves the next decision instead of triggering an automatic repair.
- Failure signal
- An eligible answer score with its expert-reference comparison, or an offline golden case, identifies a problem.
- Attribution
- The evidence path shows which control surfaced the issue.
- Response
- Replay the case manually when needed and prepare a targeted change for review.
- Authority
- An engineer approves the change before deployment.
- Recovery check
- The relevant answer score with its reference comparison, or CI golden case, is rechecked.
06 · Evidence & boundaries
Evidence & boundaries
- Verified result
- Eligible answer scoring with expert-reference comparison, manual replay, CI merge gates, and approval remain distinct.
- Contribution scope
- I instrumented the evaluation path for eligible Teams answers, expert references, manual replay, and offline golden-case CI merge gates.
- Metrics not established by public material
- No public score lift, false-positive rate, or release-frequency delta is established.