WW

Work case study · Agent Systems

Agent Systems · production operations

Automated Incident Diagnosis

Impact

Responders can move from an alert to a traceable investigation, then decide the next operational step with the actual tool results in view.

  • One default investigation
  • Live tool context
  • Engineer-owned changes

01 · Problem & operating context

An incident alert was only the beginning of the investigation.

A PagerDuty alert gave the on-call engineer a starting point, not an explanation. The relevant logs, current metrics, code changes, and operating knowledge lived in different systems, and they did not all have the same freshness or authority.

Before the harness, the first response meant rebuilding that context by hand. I wanted a path that made its sources and limits visible, so the responder could review a diagnosis instead of repeatedly assembling the first evidence trail.

02 · My role & ownership

I made one Unified investigation the normal path.

I built the diagnosis harness, the Unified path and its migration, plus the Studio adapters that use it. The default flow selects context and calls bounded read-only tools before it assembles a response for REST, streaming, and Studio.

The earlier parallel-investigator design remains available for cases that need it, but it is now a fallback rather than the standard route. Shared clients and the wider operational platform remain team context.

Built
Diagnosis harness, Unified path and migration, and Studio adapters.
Designed
Default Unified investigation with parallel investigators retained as fallback.
Connected
Bounded read-only tools and shared operational context.
Owned

Implemented the diagnosis harness, Unified path, migration, and Studio adapters.

Designed

Kept the Unified loop as the default and legacy parallel investigators as a fallback.

Contributed

Connected bounded read-only tools and shared operational context.

Team context

Shared Studio, Teams, IDE, and MCP surfaces provide the surrounding platform context.

03 · System architecture

The same loop carries the request from alert to answer.

Requests from REST, streaming, and Studio enter the same default loop. It chooses incident context, reads the relevant operational tools within their allowed scope, and assembles a response around the results returned by those tools.

Streaming can show progress while the investigation is still running; the final verdict is a different state. That distinction lets a responder tell what has been checked from what the completed investigation concludes.

Inputs
PagerDuty incident signal, live operational tools, and indexed knowledge.
State
Unified context, bounded reads, and streamed progress.
Output
A review-ready verdict with a separate approval boundary for changes.
ONE UNIFIED INVESTIGATION LOOP

04 · Decisions & trade-offs

I used knowledge to orient the search, not to certify the answer.

Indexed operational knowledge is useful for understanding a service and choosing where to look. It is not the same as current proof. A prompt-only or retrieval-only diagnosis would have been simpler to demonstrate, but it would blur historical context with live service state.

The response therefore keeps evidence references alongside the claim and preserves the difference between streamed work and the final verdict. That adds interfaces and tests, but it prevents a fluent answer from being mistaken for an operational decision.

Chosen approach

Keep context and operational proof separate, then validate the claim before returning it.

Rejected alternative

Let one prompt or retrieval result produce the diagnosis directly.

Why

Different sources have different authority and freshness; the response needs to show what actually supports it.

Cost introduced

More stages, typed evidence, and validation work make the path slower to build and operate.

TWO AUTHORITIES, ONE GATE

05 · Failure modes & recovery

Thin evidence remains part of the answer.

The loop only reads from bounded tools. If a result is stale, missing, or contradictory, the response keeps that gap visible and can ask for the next piece of evidence instead of smoothing it into a confident story.

A recommendation can still make the next investigation step clear, but an engineer decides whether any production change is appropriate. The system stops at a review boundary rather than turning a diagnosis into an automatic action.

Failure signal
A tool result is missing, stale, contradictory, or insufficient for the claim.
Attribution
The evidence and validation stages keep the gap attached to the claim instead of treating it as a model failure alone.
Response
Stop or return a bounded recommendation with the evidence gap visible.
Authority
An engineer remains the authority for any change to a system.
Recovery check
Evidence assertions, traces, and evaluator checks make the decision path reviewable.
FAIL CLOSED, RECOVER WITH PROOF

06 · Evidence & boundaries

Evidence & boundaries

Verified result
Unified investigation, streamed progress, final verdict, and human approval remain separate and reviewable.
Contribution scope
I implemented the harness, Unified path and migration, plus Studio adapters for the shared diagnosis surface.
Metrics not established by public material
No public latency, concurrency, success-rate, cost, or MTTR figure is established here.
TOTAL VIEWS—
TODAY'S VIEWS—