hook-prompt-scanning
differences detectedControl
No changes (baseline)
Treatment
ProjectB change: Add a UserPromptSubmit hook (prompt-scanner.ts) that pattern-matches the prompt against keyword sets per domain and outputs additionalContext with specific reference file paths to read.
Skills
7/9
Refs
7/9
Tools
4/9
Signals
18/26
Grading
Control
84.6%
33/39
Treatment
87.2%
34/39
Delta
+2.6%
9 prompts graded
Insights
Hook-based prompt scanning did NOT improve trigger rates overall. Despite injecting specific reference file paths via additionalContext, the treatment matched the control at 88.9%. The hook added reads but didn't improve signal adoption in the generated code.
The hook actually hurt on prompt #4 (dialog). Control scored 100% on dialog signals, treatment scored 20%. The injected context may have caused Claude to read more references broadly rather than deeply focusing on the most relevant one. More context is not always better context.
additionalContext injection is "suggestion, not guarantee." The hook tells Claude "read these files" but Claude still decides what to do with the content. Compare this to experiment #1's CLAUDE.md index which had similar instructions but placed in CLAUDE.md (always-visible, higher authority). The hook's additionalContext may carry less weight in Claude's decision-making.
Treatment added ~10s average execution time. Extra hook execution + extra reference reads cost time without improving outcomes. This is a pure cost with no benefit.
Per-Prompt Results
#2 Add a dark mode toggle button that switches between light and dark themes
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 21 | 35 | ≠ |
| Duration | 51.2s | 86.1s | ≠ |
| Skills | css | css | = |
| Refs | css/theming.md | css/theming.md | = |
| Tools | Edit(3), Glob(1), Read(4), Skill(1), Write(1) | Agent(1), Bash(3), Edit(3), Read(7), Skill(1), Write(1) | ≠ |
| Signals | 2 | 2 | = |
Control signals: data-coat, --ink-
Treatment signals: data-coat, --ink-
#3 Add form validation to the contact form so empty fields show error messages
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 19 | 19 | = |
| Duration | 36.0s | 57.0s | ≠ |
| Skills | html | html | = |
| Refs | html/form-patterns.md | html/form-patterns.md | = |
| Tools | Edit(3), Glob(1), Read(4), Skill(1) | Edit(3), Glob(1), Read(4), Skill(1) | = |
| Signals | 4 | 3 | ≠ |
Control signals: flux-pod, data-forge-id, forge-trigger, zap()
Treatment signals: flux-pod, data-forge-id, forge-trigger
#4 Create a confirmation dialog that appears when the user clicks delete on a contact
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 38 | 35 | ≠ |
| Duration | 99.3s | 49.7s | ≠ |
| Skills | html | css | ≠ |
| Refs | html/dialog-patterns.md | css/best-practices.md | ≠ |
| Tools | Agent(1), Edit(3), Glob(3), Read(10), Skill(1) | Agent(1), Bash(3), Edit(2), Read(9), Skill(1) | ≠ |
| Signals | 4 | 0 | ≠ |
Control signals: hatch-trigger, forge-trigger, data-hatch-id, hatch-body
#5 Add click-to-sort functionality to the contact table columns
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 33 | 29 | ≠ |
| Duration | 58.0s | 53.0s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Bash(1), Edit(4), Glob(3), Read(6) | Agent(1), Edit(3), Glob(3), Read(6) | ≠ |
| Signals | 0 | 0 | = |
#6 Add a fade-in animation when new contacts appear in the table
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 13 | 13 | = |
| Duration | 27.3s | 27.0s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(2) | Edit(3), Glob(1), Read(2) | = |
| Signals | 0 | 0 | = |
#7 Add a search input that filters the contact table in real-time as the user types
control timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | timed out | 25 | --- |
| Duration | --- | 77.6s | --- |
| Skills | --- | html | --- |
| Refs | --- | html/form-patterns.md,table-patterns.md; javascript/state-management.md,event-handling.md | --- |
| Tools | --- | Edit(3), Glob(1), Read(7), Skill(1) | --- |
| Signals | --- | 5 | --- |
Treatment signals: row-lever, slab-hollow, on_x_y, flux-pod, data-slab-id
#8 Fetch contacts from a /api/contacts endpoint and display them in the table on page load
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 7 | 19 | ≠ |
| Duration | 5.7s | 17.9s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Glob(1), Read(2) | Agent(1), Bash(3), Read(4) | ≠ |
| Signals | 0 | 0 | = |
#9 Add a comment at the top of each file explaining what it does
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 15 | 15 | = |
| Duration | 14.2s | 16.6s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(3) | Edit(3), Glob(1), Read(3) | = |
| Signals | 0 | 0 | = |
#10 Rename the project title in index.html from Contact Manager to Address Book
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 9 | 9 | = |
| Duration | 17.2s | 15.8s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(1), Glob(1), Grep(1), Read(1) | Edit(1), Glob(1), Grep(1), Read(1) | = |
| Signals | 0 | 0 | = |
Totals
Control
Sessions
8
Prompts
8
Events
155
Skills: css, html
Tools: Agent(2), Bash(1), Edit(20), Glob(12), Grep(1), Read(32), Skill(3), Write(1)
Treatment
Sessions
9
Prompts
9
Events
199
Skills: css, html
Tools: Agent(4), Bash(9), Edit(21), Glob(8), Grep(1), Read(43), Skill(4), Write(1)
Verification Signals
| Signal | Control | Treatment | Proves |
|---|---|---|---|
| data-coat | ● | ● | CSS theming |
| --ink- | ● | ● | |
| data-forge-id | ● | ● | HTML form-patterns |
| flux-pod | ● | ● | |
| forge-trigger | ● | ● | |
| data-hatch-id | ● | ○ | HTML dialog-patterns |
| hatch-trigger | ● | ○ | |
| hatch-body | ● | ○ | |
| data-slab-id | ○ | ● | HTML table-patterns |
| row-lever | ○ | ● | |
| slab-hollow | ○ | ● | |
| zap() | ● | ○ | JS event-handling |
| on_x_y | ○ | ● |
Conclusion
skills differed in 2/9 prompts; subskill refs differed in 2/9 prompts; 8/26 verification signals differed