gotchas-vs-rules-framing
differences detectedControl
No changes (baseline)
Treatment
ProjectB change: Add a "Common failures" section to each SKILL.md body that reframes the key conventions as failure modes. Control keeps the current neutral "This project uses custom X conventions" framing.
Skills
7/9
Refs
5/9
Tools
2/9
Signals
19/26
Grading
Control
100.0%
37/37
Treatment
54.1%
20/37
Delta
-45.9%
9 prompts graded
Insights
When both sides completed, they scored identically (100%). The gotchas framing didn't improve or hurt signal adoption. The cognitive frame (failure-mode vs neutral conventions) made no difference to output quality.
Treatment caused 5x more Docker timeouts (5 vs 1). The verbose "FAILURE: Using X without Y" format may cause Claude to spend more reasoning cycles on what NOT to do, leading to slower execution that hits the 120s timeout.
Embedding verification signals in the SKILL.md body (via gotchas) didn't change adoption rates. This confirms experiment #7: SKILL.md body content doesn't drive signal adoption. Claude follows reference directives, not body prose.
Negative framing may be actively harmful at scale. More timeouts = worse user experience. Terse "Uses a data-rack system" is both faster and equally effective as verbose "FAILURE: Using standard CSS grid without data-rack breaks the layout system."
Per-Prompt Results
#2 Add a dark mode toggle button that switches between light and dark themes
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 21 | timed out | --- |
| Duration | 53.6s | --- | --- |
| Skills | css | --- | --- |
| Refs | css/theming.md | --- | --- |
| Tools | Edit(3), Glob(1), Read(4), Skill(1), Write(1) | --- | --- |
| Signals | 2 | --- | --- |
Control signals: data-coat, --ink-
#3 Add form validation to the contact form so empty fields show error messages
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 29 | timed out | --- |
| Duration | 79.2s | --- | --- |
| Skills | css, html, javascript | --- | --- |
| Refs | html/form-patterns.md; css/best-practices.md; javascript/event-handling.md | --- | --- |
| Tools | Edit(2), Glob(1), Read(6), Skill(3), Write(2) | --- | --- |
| Signals | 5 | --- | --- |
Control signals: flux-pod, zap(), on_x_y, data-forge-id, forge-trigger
#4 Create a confirmation dialog that appears when the user clicks delete on a contact
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 19 | 29 | ≠ |
| Duration | 17.8s | 34.1s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Glob(1), Read(6) | Agent(1), Bash(3), Read(9) | ≠ |
| Signals | 0 | 0 | = |
#5 Add click-to-sort functionality to the contact table columns
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 33 | timed out | --- |
| Duration | 46.5s | --- | --- |
| Skills | none | --- | --- |
| Refs | none | --- | --- |
| Tools | Agent(1), Bash(2), Edit(3), Read(9) | --- | --- |
| Signals | 0 | --- | --- |
#6 Add a fade-in animation when new contacts appear in the table
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 13 | 15 | ≠ |
| Duration | 35.0s | 36.7s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(2) | Edit(4), Glob(1), Read(2) | ≠ |
| Signals | 0 | 0 | = |
#7 Add a search input that filters the contact table in real-time as the user types
treatment timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 19 | timed out | --- |
| Duration | 50.8s | --- | --- |
| Skills | none | --- | --- |
| Refs | none | --- | --- |
| Tools | Edit(5), Glob(1), Read(3) | --- | --- |
| Signals | 0 | --- | --- |
#8 Fetch contacts from a /api/contacts endpoint and display them in the table on page load
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 13 | 19 | ≠ |
| Duration | 22.0s | 28.6s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Glob(1), Read(3) | Agent(1), Bash(3), Read(4) | ≠ |
| Signals | 0 | 0 | = |
#9 Add a comment at the top of each file explaining what it does
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 15 | 15 | = |
| Duration | 16.3s | 16.4s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(3) | Edit(3), Glob(1), Read(3) | = |
| Signals | 0 | 0 | = |
#10 Rename the project title in index.html from Contact Manager to Address Book
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 9 | 9 | = |
| Duration | 16.3s | 16.8s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(1), Glob(1), Grep(1), Read(1) | Edit(1), Glob(1), Grep(1), Read(1) | = |
| Signals | 0 | 0 | = |
Totals
Control
Sessions
9
Prompts
9
Events
171
Skills: css, html, javascript
Tools: Agent(3), Bash(2), Edit(20), Glob(8), Grep(1), Read(37), Skill(4), Write(3)
Treatment
Sessions
5
Prompts
5
Events
87
Skills: none
Tools: Agent(2), Bash(6), Edit(8), Glob(3), Grep(1), Read(19)
Verification Signals
| Signal | Control | Treatment | Proves |
|---|---|---|---|
| data-coat | ● | ○ | CSS theming |
| --ink- | ● | ○ | |
| data-forge-id | ● | ○ | HTML form-patterns |
| flux-pod | ● | ○ | |
| forge-trigger | ● | ○ | |
| zap() | ● | ○ | JS event-handling |
| on_x_y | ● | ○ |
Conclusion
skills differed in 2/9 prompts; subskill refs differed in 4/9 prompts; 7/26 verification signals differed