Testing Whether the Extraction Captured the Expertise

Every extraction produces a knowledge artifact — a decision tree, a runbook, a structured guide. The question that most practitioners never ask is: does this artifact actually transfer the expertise, or does it transfer a description of the expertise? The difference is not semantic. An expert who reviews the artifact and says "yes, this is accurate" has not validated transfer. Transfer is validated when a non-expert uses the artifact to perform the task at a higher level than they could without it.

1. Artifact Review (Expert Validation) Present the knowledge artifact to the expert who provided it and ask structured review questions: "Is this accurate?" "What is missing?" "What would you change?" "Is there a scenario where following this would produce the wrong result?" This is not the validation of transfer — it is the validation of accuracy. Both are necessary, but they are different tests.
2. Novice Test (Transfer Validation) Present the knowledge artifact to a non-expert who performs the task domain at a lower level. Give them a set of scenarios. Measure their performance without the artifact, then with the artifact. The delta is the transfer measure. If the artifact produces no measurable improvement in novice performance, the extraction failed to capture the actionable layer of the expertise.
3. Edge Case Stress Testing Design test scenarios specifically at the boundary conditions of the extracted knowledge — the cases the expert described as "difficult," the anomalies from the extraction sessions, the scenarios where "it depends." If the artifact fails at its own identified edge cases, the extraction is incomplete at exactly the points where expertise matters most.
4. Failure Mode Inventory Run the artifact against the failure modes the expert identified during extraction. Does following the artifact prevent the common mistakes the expert described? Does it help the user recognize the early signals the expert said novices miss? A knowledge artifact that does not defend against known failure modes has failed a basic quality check.