reference-toc-header
differences detectedControl
No changes (baseline)
Treatment
ProjectB change: Add a ToC header to all 11 reference files listing sections and key concepts. Keep all content identical below the ToC.
Skills
5/10
Refs
4/10
Tools
2/10
Signals
19/26
Grading
Control
69.2%
27/39
Treatment
79.5%
31/39
Delta
+10.3%
10 prompts graded
Insights
Treatment loaded more skills and read more references. The ToC listing section headers (including signal names like "data-slab-id", "zap()") may have made references more discoverable, leading to more reads. B activated javascript skill that A didn't.
More reads didn't consistently improve signal adoption. Prompts #2 (dark mode) and #4 (dialog) degraded despite more files read. Extra reads may dilute focus on the most relevant reference.
Multi-skill prompts benefited most. #5 (table sort: +67%) and #7 (search filter: +80%) are cross-domain tasks that benefit from broader reading. The ToC encouraged Claude to explore more references.
Net effect is a wash (+1.4%). ToC helps complex prompts, hurts simple ones. Not a universal improvement. Value depends on task complexity.
Per-Prompt Results
#1 Add a two-column layout to the contact manager with a sidebar for filters
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 43 | timed out | --- |
| Duration | 134.2s | --- | --- |
| Skills | html | --- | --- |
| Refs | html/best-practices.md,form-patterns.md | --- | --- |
| Tools | Agent(1), Bash(1), Edit(5), Glob(3), Read(9), Skill(1) | --- | --- |
| Signals | 3 | --- | --- |
Control signals: data-forge-id, flux-pod, forge-trigger
#2 Add a dark mode toggle button that switches between light and dark themes
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 33 | 37 | ≠ |
| Duration | 66.8s | 46.1s | ≠ |
| Skills | css | html | ≠ |
| Refs | css/theming.md | html/best-practices.md | ≠ |
| Tools | Agent(1), Bash(1), Edit(3), Glob(1), Read(7), Skill(1), Write(1) | Agent(1), Bash(2), Edit(4), Read(9), Skill(1) | ≠ |
| Signals | 2 | 0 | ≠ |
Control signals: data-coat, --ink-
#3 Add form validation to the contact form so empty fields show error messages
control timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | timed out | 31 | --- |
| Duration | --- | 78.8s | --- |
| Skills | --- | javascript, css, html | --- |
| Refs | --- | javascript/event-handling.md; html/form-patterns.md; css/best-practices.md | --- |
| Tools | --- | Edit(2), Glob(2), Read(6), Skill(3), Write(2) | --- |
| Signals | --- | 5 | --- |
Treatment signals: data-forge-id, flux-pod, zap(), on_x_y, forge-trigger
#4 Create a confirmation dialog that appears when the user clicks delete on a contact
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 45 | 29 | ≠ |
| Duration | 70.5s | 50.4s | ≠ |
| Skills | css | none | ≠ |
| Refs | css/best-practices.md,theming.md | none | ≠ |
| Tools | Agent(1), Bash(3), Edit(3), Glob(3), Read(10), Skill(1) | Agent(1), Bash(3), Edit(1), Read(8) | ≠ |
| Signals | 2 | 0 | ≠ |
Control signals: data-coat, --ink-
#5 Add click-to-sort functionality to the contact table columns
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 25 | 29 | ≠ |
| Duration | 47.5s | 83.4s | ≠ |
| Skills | none | javascript | ≠ |
| Refs | none | javascript/event-handling.md,state-management.md | ≠ |
| Tools | Agent(1), Bash(1), Edit(3), Read(6) | Edit(5), Glob(2), Read(5), Skill(1), Write(1) | ≠ |
| Signals | 0 | 2 | ≠ |
Treatment signals: zap(), on_x_y
#6 Add a fade-in animation when new contacts appear in the table
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 17 | 19 | ≠ |
| Duration | 47.0s | 59.8s | ≠ |
| Skills | css | css | = |
| Refs | css/animation-patterns.md | css/animation-patterns.md | = |
| Tools | Edit(3), Glob(1), Read(3), Skill(1) | Edit(4), Glob(1), Read(3), Skill(1) | ≠ |
| Signals | 2 | 2 | = |
Control signals: data-zap, --pulse
Treatment signals: data-zap, --pulse
#7 Add a search input that filters the contact table in real-time as the user types
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 41 | 25 | ≠ |
| Duration | 72.9s | 100.3s | ≠ |
| Skills | html | html | = |
| Refs | html/table-patterns.md,best-practices.md,form-patterns.md | html/form-patterns.md,table-patterns.md; javascript/state-management.md; css/layout-patterns.md | ≠ |
| Tools | Agent(1), Edit(5), Glob(3), Read(9), Skill(1) | Edit(3), Glob(1), Read(7), Skill(1) | ≠ |
| Signals | 0 | 4 | ≠ |
Treatment signals: data-rack, flux-pod, data-slab-id, --seam
#8 Fetch contacts from a /api/contacts endpoint and display them in the table on page load
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 21 | 7 | ≠ |
| Duration | 21.2s | 6.3s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Bash(2), Read(6) | Glob(1), Read(2) | ≠ |
| Signals | 0 | 0 | = |
#9 Add a comment at the top of each file explaining what it does
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 15 | 15 | = |
| Duration | 12.7s | 14.6s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(3) | Edit(3), Glob(1), Read(3) | = |
| Signals | 0 | 0 | = |
#10 Rename the project title in index.html from Contact Manager to Address Book
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 9 | 9 | = |
| Duration | 13.8s | 15.8s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(1), Glob(1), Grep(1), Read(1) | Edit(1), Glob(1), Grep(1), Read(1) | = |
| Signals | 0 | 0 | = |
Totals
Control
Sessions
9
Prompts
9
Events
249
Skills: html, css
Tools: Agent(6), Bash(8), Edit(26), Glob(13), Grep(1), Read(54), Skill(5), Write(1)
Treatment
Sessions
9
Prompts
9
Events
201
Skills: html, javascript, css
Tools: Agent(2), Bash(5), Edit(23), Glob(9), Grep(1), Read(44), Skill(7), Write(3)
Verification Signals
| Signal | Control | Treatment | Proves |
|---|---|---|---|
| data-rack | ○ | ● | CSS layout-patterns |
| --seam | ○ | ● | |
| data-zap | ● | ● | CSS animation-patterns |
| --pulse | ● | ● | |
| data-coat | ● | ○ | CSS theming |
| --ink- | ● | ○ | |
| data-forge-id | ● | ● | HTML form-patterns |
| flux-pod | ● | ● | |
| forge-trigger | ● | ● | |
| data-slab-id | ○ | ● | HTML table-patterns |
| zap() | ○ | ● | JS event-handling |
| on_x_y | ○ | ● |
Conclusion
skills differed in 5/10 prompts; subskill refs differed in 6/10 prompts; 7/26 verification signals differed