Config Validation & Drift Detection

Configuration drift is the silent killer of AI-assisted workflows. It happens when AGENTS.md files diverge from the codebase reality — rules that reference deleted files, commands that no longer work, conventions that the team abandoned months ago. Stale rules are worse than no rules: they teach Codex patterns that conflict with the current codebase, producing output that looks correct but violates actual team practice.

Config validation should be automated and continuous. A CI check that parses every AGENTS.md file, verifies that referenced commands exist, checks that file paths are valid, and flags rules that conflict across hierarchy levels catches drift before it causes problems. Treat AGENTS.md files with the same rigor as infrastructure-as-code: validate on every PR, review changes in diff, and test the merged output.

#!/bin/bash
# Validate all AGENTS.md files in the repository
set -euo pipefail

echo "=== AGENTS.md Validation ===" 

# Find all config files
files=$(find . -name "AGENTS.md" -o -name "AGENTS.override.md" | sort)
echo "Found $(echo "$files" | wc -l) config files"

# Check each file for common issues
for f in $files; do
  echo "\nChecking: $f"
  
  # Check for referenced commands that don't exist
  grep -oP '`\K[^`]+' "$f" | while read -r cmd; do
    binary=$(echo "$cmd" | awk '{print $1}')
    if ! command -v "$binary" &>/dev/null && \
       ! grep -q "$binary" package.json 2>/dev/null; then
      echo "  WARN: Command '$binary' not found in PATH or package.json"
    fi
  done
  
  # Check file size (warn if over 200 lines)
  lines=$(wc -l < "$f")
  if [ "$lines" -gt 200 ]; then
    echo "  WARN: $lines lines (recommended: <200)"
  fi
done

echo "\n=== Validation complete ==="

1. Create the validation script Write a script that finds all AGENTS.md files, checks referenced commands, validates file paths, and reports size. Run it in CI on every PR that touches an AGENTS.md file.
2. Add to PR template Include "AGENTS.md updated if conventions changed?" in your PR checklist. Make it a habit, not an afterthought.
3. Schedule quarterly audits Set a calendar reminder to review all AGENTS.md files quarterly. Delete stale rules, update changed conventions, and verify the hierarchy still matches the codebase structure.