Agent Systems · production operations
Automated Incident Diagnosis
Impact
Responders can move from an alert to a traceable investigation, then decide the next operational step with the actual tool results in view.
- One default investigation
- Live tool context
- Engineer-owned changes
01 · Problem & operating context
An incident alert was only the beginning of the investigation.
A PagerDuty alert gave the on-call engineer a starting point, not an explanation. The relevant logs, current metrics, code changes, and operating knowledge lived in different systems, and they did not all have the same freshness or authority.
Before the harness, the first response meant rebuilding that context by hand. I wanted a path that made its sources and limits visible, so the responder could review a diagnosis instead of repeatedly assembling the first evidence trail.
02 · My role & ownership
I made one Unified investigation the normal path.
I built the diagnosis harness, the Unified path and its migration, plus the Studio adapters that use it. The default flow selects context and calls bounded read-only tools before it assembles a response for REST, streaming, and Studio.
The earlier parallel-investigator design remains available for cases that need it, but it is now a fallback rather than the standard route. Shared clients and the wider operational platform remain team context.
- Built
- Diagnosis harness, Unified path and migration, and Studio adapters.
- Designed
- Default Unified investigation with parallel investigators retained as fallback.
- Connected
- Bounded read-only tools and shared operational context.
Implemented the diagnosis harness, Unified path, migration, and Studio adapters.
Kept the Unified loop as the default and legacy parallel investigators as a fallback.
Connected bounded read-only tools and shared operational context.
Shared Studio, Teams, IDE, and MCP surfaces provide the surrounding platform context.
03 · System architecture
The same loop carries the request from alert to answer.
Requests from REST, streaming, and Studio enter the same default loop. It chooses incident context, reads the relevant operational tools within their allowed scope, and assembles a response around the results returned by those tools.
Streaming can show progress while the investigation is still running; the final verdict is a different state. That distinction lets a responder tell what has been checked from what the completed investigation concludes.
- Inputs
- PagerDuty incident signal, live operational tools, and indexed knowledge.
- State
- Unified context, bounded reads, and streamed progress.
- Output
- A review-ready verdict with a separate approval boundary for changes.
04 · Decisions & trade-offs
I used knowledge to orient the search, not to certify the answer.
Indexed operational knowledge is useful for understanding a service and choosing where to look. It is not the same as current proof. A prompt-only or retrieval-only diagnosis would have been simpler to demonstrate, but it would blur historical context with live service state.
The response therefore keeps evidence references alongside the claim and preserves the difference between streamed work and the final verdict. That adds interfaces and tests, but it prevents a fluent answer from being mistaken for an operational decision.
Keep context and operational proof separate, then validate the claim before returning it.
Let one prompt or retrieval result produce the diagnosis directly.
Different sources have different authority and freshness; the response needs to show what actually supports it.
More stages, typed evidence, and validation work make the path slower to build and operate.
05 · Failure modes & recovery
Thin evidence remains part of the answer.
The loop only reads from bounded tools. If a result is stale, missing, or contradictory, the response keeps that gap visible and can ask for the next piece of evidence instead of smoothing it into a confident story.
A recommendation can still make the next investigation step clear, but an engineer decides whether any production change is appropriate. The system stops at a review boundary rather than turning a diagnosis into an automatic action.
- Failure signal
- A tool result is missing, stale, contradictory, or insufficient for the claim.
- Attribution
- The evidence and validation stages keep the gap attached to the claim instead of treating it as a model failure alone.
- Response
- Stop or return a bounded recommendation with the evidence gap visible.
- Authority
- An engineer remains the authority for any change to a system.
- Recovery check
- Evidence assertions, traces, and evaluator checks make the decision path reviewable.
06 · Evidence & boundaries
Evidence & boundaries
- Verified result
- Unified investigation, streamed progress, final verdict, and human approval remain separate and reviewable.
- Contribution scope
- I implemented the harness, Unified path and migration, plus Studio adapters for the shared diagnosis surface.
- Metrics not established by public material
- No public latency, concurrency, success-rate, cost, or MTTR figure is established here.