Six steps from a goal to a better generation.
Nothing here is magic. It is a loop that refuses to stop at the first guess.
- 01
Declare the job
Name the goal, hand over the tools, point at an eval pack. No prompt writing.
- 02
Forge an architecture
SMITH proposes the agent graph — router, tools, memory, guardrails — and writes it as code.
- 03
Run the eval
Every case executes in a real session. Accuracy, cost, latency and reliability are captured per run.
- 04
Read the failure
Traces are sorted into a failure taxonomy instead of a vague 'it didn't work'.
- 05
Mutate and re-score
Prompt, tool, memory or orchestrator gets patched, then the pack runs again until the score moves.
- 06
Ship the report card
A before/after table you can hand to anyone: what improved, by how much, at what cost.
What the forge actually reads
Failures are sorted into classes, each with the patch that historically fixes it.
Wrong tool selected
34%Router rewrite
Schema drift on output
26%Prompt + validator
Missing memory of prior line
18%Memory window
Tool timeout
13%Retry + fallback tool
Hallucinated citation
9%Grounding check