paths-auto-activation
differences detectedControl
No changes (baseline)
Treatment
ProjectB change: Add paths: frontmatter to all three skills mapping file extensions to skill activation (css: *.css, html: *.html, javascript: *.js).
Skills
6/9
Refs
5/9
Tools
2/9
Signals
20/26
Grading
Control
88.6%
31/35
Treatment
80.0%
28/35
Delta
-8.6%
9 prompts graded
Insights
paths: auto-activation loaded more skills but read worse references. Treatment activated javascript skill via *.js glob but read event-handling.md instead of task-relevant references. More activation != better targeting.
Prompt #4 (dialog) went from 100% to 0%. paths: triggered HTML skill when touching index.html, but Claude read best-practices.md instead of dialog-patterns.md. Broad file-based activation doesn't steer reference selection.
Prompt #6 (animation) dropped from 100% to 33%. CSS skill activated correctly via *.css glob, but Claude read theming.md instead of animation-patterns.md. Auto-activation gets the SKILL.md body loaded, but the SKILL.md body already lists all references equally.
paths: triggers skill loading, not reference selection. This is the fundamental limitation. It ensures the SKILL.md is in context, but SKILL.md lists all references neutrally. Without additional steering toward the *right* reference, broad activation can actually dilute focus.
Per-Prompt Results
#2 Add a dark mode toggle button that switches between light and dark themes
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 39 | 45 | ≠ |
| Duration | 73.6s | 87.6s | ≠ |
| Skills | css | html, css, javascript | ≠ |
| Refs | css/theming.md | css/theming.md; html/best-practices.md; javascript/event-handling.md | ≠ |
| Tools | Agent(1), Bash(3), Edit(3), Read(9), Skill(1), Write(1) | Agent(1), Bash(3), Edit(3), Glob(1), Read(9), Skill(3), Write(1) | ≠ |
| Signals | 2 | 2 | = |
Control signals: data-coat, --ink-
Treatment signals: data-coat, --ink-
#3 Add form validation to the contact form so empty fields show error messages
control timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | timed out | 27 | --- |
| Duration | --- | 47.8s | --- |
| Skills | --- | none | --- |
| Refs | --- | none | --- |
| Tools | --- | Agent(1), Bash(2), Edit(3), Read(6) | --- |
| Signals | --- | 0 | --- |
#4 Create a confirmation dialog that appears when the user clicks delete on a contact
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 35 | timed out | --- |
| Duration | 92.8s | --- | --- |
| Skills | html | --- | --- |
| Refs | html/dialog-patterns.md | --- | --- |
| Tools | Agent(1), Bash(2), Edit(5), Read(7), Skill(1) | --- | --- |
| Signals | 4 | --- | --- |
Control signals: forge-trigger, data-hatch-id, hatch-trigger, hatch-body
#5 Add click-to-sort functionality to the contact table columns
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 35 | 25 | ≠ |
| Duration | 59.1s | 43.0s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Bash(1), Edit(5), Glob(3), Read(6) | Agent(1), Bash(1), Edit(3), Read(6) | ≠ |
| Signals | 0 | 0 | = |
#6 Add a fade-in animation when new contacts appear in the table
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 17 | 13 | ≠ |
| Duration | 42.5s | 36.5s | ≠ |
| Skills | css | none | ≠ |
| Refs | css/animation-patterns.md | none | ≠ |
| Tools | Edit(3), Glob(1), Read(3), Skill(1) | Edit(3), Glob(1), Read(2) | ≠ |
| Signals | 2 | 0 | ≠ |
Control signals: data-zap, --pulse
#7 Add a search input that filters the contact table in real-time as the user types
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 27 | 33 | ≠ |
| Duration | 46.9s | 57.0s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Bash(1), Edit(4), Read(6) | Agent(1), Bash(4), Edit(4), Read(6) | ≠ |
| Signals | 0 | 0 | = |
#8 Fetch contacts from a /api/contacts endpoint and display them in the table on page load
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 7 | 17 | ≠ |
| Duration | 10.2s | 25.3s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Glob(1), Read(2) | Agent(1), Bash(2), Read(4) | ≠ |
| Signals | 0 | 0 | = |
#9 Add a comment at the top of each file explaining what it does
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 15 | 15 | = |
| Duration | 14.1s | 19.6s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(3) | Edit(3), Glob(1), Read(3) | = |
| Signals | 0 | 0 | = |
#10 Rename the project title in index.html from Contact Manager to Address Book
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 9 | 9 | = |
| Duration | 18.9s | 21.3s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(1), Glob(1), Grep(1), Read(1) | Edit(1), Glob(1), Grep(1), Read(1) | = |
| Signals | 0 | 0 | = |
Totals
Control
Sessions
8
Prompts
8
Events
184
Skills: css, html
Tools: Agent(4), Bash(7), Edit(24), Glob(7), Grep(1), Read(37), Skill(3), Write(1)
Treatment
Sessions
8
Prompts
8
Events
184
Skills: html, css, javascript
Tools: Agent(5), Bash(12), Edit(20), Glob(4), Grep(1), Read(37), Skill(3), Write(1)
Verification Signals
| Signal | Control | Treatment | Proves |
|---|---|---|---|
| data-zap | ● | ○ | CSS animation-patterns |
| --pulse | ● | ○ | |
| data-coat | ● | ● | CSS theming |
| --ink- | ● | ● | |
| forge-trigger | ● | ○ | HTML form-patterns |
| data-hatch-id | ● | ○ | HTML dialog-patterns |
| hatch-trigger | ● | ○ | |
| hatch-body | ● | ○ |
Conclusion
skills differed in 3/9 prompts; subskill refs differed in 4/9 prompts; 6/26 verification signals differed