Experiments

13 experiments tracked

description-with-examples

negative

ProjectB change: Append 2-3 concrete example prompts to each skill description field. E.g., CSS description gets "Examples: 'add dark mode', 'make the sidebar responsive', 'add hover animations'". Same domain terms, different format.

2026-03-31-17.1%
10 promptsskills 8/10refs 7/10signals 25/26

gotchas-vs-rules-framing

negative

ProjectB change: Add a "Common failures" section to each SKILL.md body that reframes the key conventions as failure modes. Control keeps the current neutral "This project uses custom X conventions" framing.

2026-03-31-45.9%
9 promptsskills 7/9refs 5/9signals 19/26

gotchas-vs-rules-rerun

negative

ProjectB change: Add "Common Failures" section to each SKILL.md body reframing key conventions as failure modes. Same treatment as original experiment.

2026-03-31-22.9%
10 promptsskills 4/10refs 3/10signals 20/26

pure-routing-skill-body

positive

ProjectB change: Rewrite all three SKILL.md bodies to pure routing format. Remove inline context lines (e.g., "Layouts use a data-rack attribute system"). Keep only numbered steps pointing to references. All domain knowledge stays exclusively in reference files.

2026-03-31+12.9%
8 promptsskills 8/8refs 7/8signals 26/26

reference-toc-header

positive

ProjectB change: Add a ToC header to all 11 reference files listing sections and key concepts. Keep all content identical below the ToC.

2026-03-31+10.3%
10 promptsskills 5/10refs 4/10signals 19/26

descriptive-reference-names

positive

ProjectB change: Rename all 11 reference files to generic names (css/ref-1.md through ref-4.md, html/ref-1.md through ref-4.md, js/ref-1.md through ref-4.md). Update SKILL.md pointers. Keep file contents and SKILL.md descriptions identical.

2026-03-30+8.3%
9 promptsskills 6/9refs 5/9signals 20/26

hook-prompt-scanning

neutral

ProjectB change: Add a UserPromptSubmit hook (prompt-scanner.ts) that pattern-matches the prompt against keyword sets per domain and outputs additionalContext with specific reference file paths to read.

2026-03-30+2.6%
9 promptsskills 7/9refs 7/9signals 18/26

keyword-dense-descriptions

neutral

ProjectB change: Rewrite all three skill descriptions from natural language prose to keyword-dense format (action verbs + domain nouns, no grammar). Keep skill body and references identical.

2026-03-30-2.4%
10 promptsskills 7/10refs 6/10signals 15/26

paths-auto-activation

negative

ProjectB change: Add paths: frontmatter to all three skills mapping file extensions to skill activation (css: *.css, html: *.html, javascript: *.js).

2026-03-30-8.6%
9 promptsskills 6/9refs 5/9signals 20/26

short-vs-long-descriptions

positive

ProjectB change: Shorten all three skill descriptions to generic ~30-36 char versions that remove domain terms (layout, animation, theming, forms, dialogs, events, state, fetch).

2026-03-30+9.4%
9 promptsskills 6/9refs 4/9signals 22/26

skill-index-read-instruction

positive

ProjectB change: Replace skill-index placeholder in CLAUDE.md with a full docs index listing all 11 reference files grouped by domain, plus the instruction "Before starting any task, identify which docs below are relevant and read them first."

2026-03-30+8.1%
9 promptsskills 8/9refs 2/9signals 16/26

forced-eval-hook-v4

positive

ProjectB change: Add `skill-forced-eval-hook.sh` that outputs forced evaluation instructions on every prompt, requiring Claude to reason about each skill YES/NO and activate via Skill tool before proceeding.

2026-03-18+8.6%
8 promptsskills 2/8refs 1/8signals 23/26

example

neutral

Added skill-index hints to CLAUDE.md

2026-03-10
3 promptsskills 2/3refs 2/3signals 20/26