skillhawk

Do skills actually help? We measured.

Across four skills, the same task on the same model scored higher with the skill installed in three cases and equal in one (the base model already knew database migrations). This is a small sample: one task per skill, 2 runs per condition.

Results

SkillWithout skillWith skillLift
deploy-runbook10%100%+90 pts
commit-messages50%88%+38 pts
dockerfile-hardening83%100%+17 pts
database-migrations100%100%+0 pts

Method

Claude Code with Claude Haiku, run 2026-10-29. Each task ran twice in an empty folder and twice with the skill installed in .claude/skills/. Each answer is graded by fixed checks in the skill's evals/evals.json; there is no AI judge. Lift is the with-skill score minus the without-skill score. Rerun: npx skillhawk eval <skill-dir> --model haiku --runs 3.

Limit. These runs used the author's normal Claude Code setup, so personal instructions and memory were loaded in both conditions. skillhawk eval now runs locked down, so reruns may differ slightly.

Source: reports/2026-10-lift-evals.md in the repository.