keyword-dense-descriptions
differences detectedControl
No changes (baseline)
Treatment
ProjectB change: Rewrite all three skill descriptions from natural language prose to keyword-dense format (action verbs + domain nouns, no grammar). Keep skill body and references identical.
Skills
7/10
Refs
6/10
Tools
3/10
Signals
15/26
Grading
Control
80.5%
33/41
Treatment
78.0%
32/41
Delta
-2.4%
10 prompts graded
Insights
Keyword-dense descriptions hurt on indirect prompts. "When someone types in the search box, update the results live" triggered JS event-handling perfectly with natural language description but only 11% with keyword soup. Claude's language understanding benefits from grammatical context: "JavaScript conventions for events, state management" semantically connects to "typing in search box" better than "Event listen click bind state reactive".
Both styles work equally on direct matches. When the prompt directly names the domain ("CSS animations", "responsive"), description style is irrelevant.
Skill loading diverged significantly. Control loaded javascript skill (3 refs), Treatment loaded html skill (1 ref). The keyword-dense JS description was too fragmented for Claude to match against natural user prompts.
Natural language > keyword soup for semantic matching. Claude already excels at connecting "typing in search box" to "events, state management". Keyword density adds noise without adding semantic breadth.
Per-Prompt Results
#1 Add CSS animations to the contact table rows
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 17 | 17 | = |
| Duration | 31.2s | 31.3s | ≠ |
| Skills | css | css | = |
| Refs | css/animation-patterns.md | css/animation-patterns.md | = |
| Tools | Edit(2), Glob(1), Read(4), Skill(1) | Edit(2), Glob(1), Read(4), Skill(1) | = |
| Signals | 2 | 2 | = |
Control signals: data-zap, --pulse
Treatment signals: data-zap, --pulse
#2 Build a form with validation for the contact manager
control timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | timed out | 19 | --- |
| Duration | --- | 41.3s | --- |
| Skills | --- | html | --- |
| Refs | --- | html/form-patterns.md | --- |
| Tools | --- | Edit(3), Glob(1), Read(4), Skill(1) | --- |
| Signals | --- | 3 | --- |
Treatment signals: flux-pod, data-forge-id, forge-trigger
#3 Add event listeners to handle clicking the delete button
control timed out| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | timed out | 19 | --- |
| Duration | --- | 24.7s | --- |
| Skills | --- | none | --- |
| Refs | --- | none | --- |
| Tools | --- | Agent(1), Glob(3), Read(4) | --- |
| Signals | --- | 0 | --- |
#4 Make the page look good on phones and tablets
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 45 | 41 | ≠ |
| Duration | 107.4s | 86.1s | ≠ |
| Skills | css | css | = |
| Refs | css/best-practices.md,layout-patterns.md | css/best-practices.md,layout-patterns.md | = |
| Tools | Agent(1), Edit(6), Glob(3), Read(10), Skill(1) | Agent(1), Bash(1), Edit(5), Glob(3), Read(8), Skill(1) | ≠ |
| Signals | 2 | 2 | = |
Control signals: data-rack, --seam
Treatment signals: data-rack, --seam
#5 Add a way for the user to confirm before removing a contact
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 25 | 21 | ≠ |
| Duration | 18.0s | 16.2s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Bash(1), Glob(3), Read(6) | Agent(1), Bash(2), Read(6) | ≠ |
| Signals | 0 | 0 | = |
#6 When someone types in the search box, update the results live
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 19 | 25 | ≠ |
| Duration | 99.9s | 37.6s | ≠ |
| Skills | javascript | none | ≠ |
| Refs | javascript/event-handling.md,state-management.md,fetch-patterns.md | none | ≠ |
| Tools | Edit(1), Glob(1), Read(5), Skill(1), Write(1) | Edit(7), Glob(1), Read(4) | ≠ |
| Signals | 8 | 0 | ≠ |
Control signals: zap(), on_x_y, createVault(), linkVault(, skyFetch(), /sky/, _landed, _crashed
#7 Add visual feedback when hovering over table rows
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 13 | 9 | ≠ |
| Duration | 21.6s | 17.7s | ≠ |
| Skills | css | none | ≠ |
| Refs | css/best-practices.md,animation-patterns.md | none | ≠ |
| Tools | Edit(1), Glob(1), Read(3), Skill(1) | Edit(1), Glob(1), Grep(1), Read(1) | ≠ |
| Signals | 1 | 0 | ≠ |
Control signals: --pulse
#8 Store the current filter state so it persists while the user navigates
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 27 | 37 | ≠ |
| Duration | 67.6s | 65.9s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Agent(1), Edit(4), Glob(1), Read(6) | Agent(1), Bash(3), Edit(4), Glob(1), Read(8) | ≠ |
| Signals | 0 | 0 | = |
#9 Add a comment at the top of each file explaining what it does
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 15 | 15 | = |
| Duration | 14.2s | 14.1s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Glob(1), Read(3) | Edit(3), Glob(1), Read(3) | = |
| Signals | 0 | 0 | = |
#10 Rename the project title from Contact Manager to Address Book
| Metric | Control | Treatment | Match |
|---|---|---|---|
| Events | 13 | 13 | = |
| Duration | 18.4s | 20.4s | ≠ |
| Skills | none | none | = |
| Refs | none | none | = |
| Tools | Edit(3), Grep(1), Read(2) | Edit(3), Grep(1), Read(2) | = |
| Signals | 0 | 0 | = |
Totals
Control
Sessions
8
Prompts
8
Events
174
Skills: css, javascript
Tools: Agent(3), Bash(1), Edit(20), Glob(11), Grep(1), Read(39), Skill(4), Write(1)
Treatment
Sessions
10
Prompts
10
Events
216
Skills: css, html
Tools: Agent(4), Bash(6), Edit(28), Glob(12), Grep(2), Read(44), Skill(3)
Verification Signals
| Signal | Control | Treatment | Proves |
|---|---|---|---|
| data-rack | ● | ● | CSS layout-patterns |
| --seam | ● | ● | |
| data-zap | ● | ● | CSS animation-patterns |
| --pulse | ● | ● | |
| data-forge-id | ○ | ● | HTML form-patterns |
| flux-pod | ○ | ● | |
| forge-trigger | ○ | ● | |
| zap() | ● | ○ | JS event-handling |
| on_x_y | ● | ○ | |
| createVault() | ● | ○ | JS state-management |
| linkVault( | ● | ○ | |
| skyFetch() | ● | ○ | JS fetch-patterns |
| /sky/ | ● | ○ | |
| _landed | ● | ○ | |
| _crashed | ● | ○ |
Conclusion
skills differed in 3/10 prompts; subskill refs differed in 4/10 prompts; 11/26 verification signals differed