Does the compiler actually work?

We constructed 10 evaluation cases from realistic B2B SaaS company knowledge: Slack threads, Notion policy pages, GitHub post-mortems, Intercom tickets. Cases are graded easy / medium / hard based on input length, ambiguity, and the presence of conflicting or implicit rules.

For each case we hand-wrote a reference SKILL.md capturing what an ideal compiler would produce, then run the Skillforge pipeline (Claude Haiku for fact extraction, Claude Opus for skill synthesis) against the raw input. The generated skill is graded by a separate Claude Opus pass on four axes:

• Faithfulness: every claim grounded in the source; hard rules carried verbatim.
• Executability: an agent can act on the skill alone; decisions are precise.
• Format compliance: follows Anthropic's exact Agent Skills spec.
• Conciseness: every section earns its place; no redundancy.

Each axis 1-5; case total max 20. Grades: A ≥ 17, B 13-16, C 9-12, D < 9.

The eval runner is open source under /evals in the repo. To reproduce these results yourself:

git clone https://github.com/AhmedTariqCS/skillforge
cd skillforge
npm install
export ANTHROPIC_API_KEY=sk-ant-...
npm run eval
# Results written to evals/results.json
# Per-case generated SKILL.md files written to evals/cases/*/generated.md

Full-suite cost: about $2.40 across all 10 cases on the current models. Per-case cost is logged in the results file.

We report three things on every run:

• Per-axis means — where the pipeline is strong and where it isn't, by category.

• Per-case grade — a holistic A/B/C/D rating for each case so we can spot regressions.

• Cost per skill — both extraction and synthesis. Real $ numbers, not estimates.

• Single-document only. The full Skillforge product synthesizes across multiple sources; that's not in this benchmark yet.

• The grader is itself a Claude model. Spot-checked vs human graders on 30 outputs and found average disagreement of 0.4 points across axes — acceptable for a relative benchmark, not gospel.

• 10 cases is small. We're growing toward 50, and we accept community-contributed cases via PR.

• The cases are well-structured by design (real but readable). Production data is messier; expect lower scores when running against raw exports.

• Multi-source synthesis benchmark. When two sources disagree, can we produce a skill that handles the disagreement correctly?

• Eval expansion to 50 cases across 20 domains. Public leaderboard for community-contributed pipelines.

• Continuous re-evaluation: every Claude model upgrade triggers a full benchmark re-run. Results published here on every commit.