feat: add BDD + Pyramid/Trophy test strategy across BMM workflows
Integrate comprehensive BDD best practices and pyramid/trophy test strategy: **Agents:** - TEA: Now "Master Test Architect + BDD Specialist" with 6 core BDD principles - DEV: Enhanced with BDD test implementation awareness (Given/When/Then) - SM: Champions test-first with ATDD workflow in menu **Workflows:** - Epic Creation: Step 4.5 identifies E2E vs unit test scenarios during planning - ATDD: Complete pyramid/trophy strategy with Gherkin enforcement for ALL Playwright - Test Design: Strategic test placement with decision tree - Dev Story: Step 4.5 validates BDD tests exist before implementation **Knowledge Base:** - New bdd-standards.md with ONE When rule, declarative style, entity naming - Indexed in tea-index.csv for TEA agent knowledge lookup **Key Principle:** ALL Playwright tests use Gherkin/BDD format - the only difference between E2E (root tests/) and Frontend (frontend/tests/) is whether they use real or mocked backends. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Vincent Brouillet
parent
c748f0f6cc
commit
6e282840d9
|
|
@ -0,0 +1,566 @@
|
|||
# Testing BDD + Pyramid/Trophy Implementation
|
||||
|
||||
This guide walks through testing all BDD best practices and pyramid/trophy test strategy changes across BMM workflows.
|
||||
|
||||
## Quick Validation Checklist
|
||||
|
||||
- [ ] SM agent has ATDD workflow in menu
|
||||
- [ ] Epic creation identifies E2E vs unit test scenarios
|
||||
- [ ] ATDD workflow applies pyramid/trophy decision tree
|
||||
- [ ] ATDD enforces Gherkin format for ALL Playwright tests
|
||||
- [ ] Dev story validates BDD tests exist before implementation
|
||||
- [ ] Dev story enforces Gherkin format for ALL Playwright tests
|
||||
- [ ] TEA agent can load BDD standards knowledge
|
||||
- [ ] Test Design workflow applies pyramid/trophy strategy
|
||||
|
||||
## Test 1: SM Agent Menu Includes ATDD
|
||||
|
||||
**Objective:** Verify Scrum Master agent now offers ATDD workflow for test-first approach.
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# 1. Activate SM agent
|
||||
/bmm:agents:sm
|
||||
|
||||
# 2. Look for ATDD in the menu output
|
||||
# Expected: You should see "*atdd" in the menu with description
|
||||
# "Define BDD acceptance tests BEFORE implementation (test-first approach)"
|
||||
```
|
||||
|
||||
### Expected Result:
|
||||
```
|
||||
Available Commands:
|
||||
...
|
||||
*create-story-context - Generate Story Context XML...
|
||||
*atdd - Define BDD acceptance tests BEFORE implementation (test-first approach)
|
||||
*dev-story - Execute story implementation...
|
||||
...
|
||||
```
|
||||
|
||||
### Validation:
|
||||
- ATDD appears in SM agent menu
|
||||
- Description emphasizes test-first approach
|
||||
- Menu item positioned between story-context and dev-story
|
||||
|
||||
---
|
||||
|
||||
## Test 2: Epic Creation - Test Strategy Planning (Step 4.5)
|
||||
|
||||
**Objective:** Verify epic creation workflow now categorizes acceptance criteria into E2E vs unit tests.
|
||||
|
||||
### Setup:
|
||||
Create a sample epic with mixed acceptance criteria:
|
||||
- User-facing ACs (should become E2E candidates)
|
||||
- Technical ACs (should become unit tests)
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# 1. Run epic creation workflow (or manually test Step 4.5)
|
||||
/bmm:workflows:create-epics-and-stories
|
||||
|
||||
# 2. When you reach Step 4.5, provide sample ACs like:
|
||||
# - "User can register with email and password"
|
||||
# - "User receives confirmation email after registration"
|
||||
# - "Registration validates email format"
|
||||
# - "Backend rate-limits registration attempts"
|
||||
# - "Frontend displays password strength indicator"
|
||||
```
|
||||
|
||||
### Expected Behavior:
|
||||
|
||||
The workflow should analyze each AC using the decision tree:
|
||||
|
||||
**AC: "User can register with email and password"**
|
||||
```
|
||||
1. User-facing? YES
|
||||
2. Needs FE+BE integration? YES
|
||||
3. Critical gate? YES
|
||||
→ E2E BDD Test (Playwright+Gherkin, real backend)
|
||||
```
|
||||
|
||||
**AC: "Registration validates email format"**
|
||||
```
|
||||
1. User-facing? YES
|
||||
2. Needs FE+BE integration? NO (frontend only)
|
||||
3. Critical gate? NO
|
||||
→ Frontend BDD Test (Playwright+Gherkin, mocked backend) OR Frontend Unit Test
|
||||
```
|
||||
|
||||
**AC: "Backend rate-limits registration attempts"**
|
||||
```
|
||||
1. User-facing? NO (technical)
|
||||
→ Backend Unit Test
|
||||
```
|
||||
|
||||
### Expected Output:
|
||||
|
||||
```markdown
|
||||
## Epic 1 Test Strategy
|
||||
|
||||
**E2E BDD Test Scenarios (2):**
|
||||
1. User registration happy path (email → confirmation → login)
|
||||
2. User receives confirmation email after registration
|
||||
|
||||
**Frontend BDD Test Scenarios (1):**
|
||||
1. Registration form validates email format (mocked backend)
|
||||
|
||||
**Unit Test Coverage (5):**
|
||||
- Backend: Rate limiting (3 tests)
|
||||
- Backend: Email validation edge cases (2 tests)
|
||||
- Frontend: Password strength indicator states (2 tests)
|
||||
|
||||
**Test Distribution:**
|
||||
- E2E BDD: 2 tests (critical paths)
|
||||
- Frontend BDD: 1 test (frontend integration)
|
||||
- Unit: 7 tests (variations, edge cases)
|
||||
```
|
||||
|
||||
### Validation:
|
||||
- Step 4.5 exists and runs
|
||||
- Decision tree correctly categorizes user-facing vs technical ACs
|
||||
- E2E BDD reserved for critical full-stack flows
|
||||
- Frontend BDD mentioned with Gherkin format
|
||||
- Unit tests recommended for variations/edge cases
|
||||
- Test counts provided (target: 5-10 E2E per epic)
|
||||
|
||||
---
|
||||
|
||||
## Test 3: ATDD Workflow - Pyramid/Trophy Strategy (Step 2)
|
||||
|
||||
**Objective:** Verify ATDD workflow applies pyramid/trophy decision tree and enforces Gherkin format for ALL Playwright tests.
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# 1. Run ATDD workflow
|
||||
/bmm:workflows:atdd
|
||||
|
||||
# 2. When prompted for story, provide one with user-facing ACs like:
|
||||
# Story: User Login
|
||||
# AC1: User can log in with valid email and password
|
||||
# AC2: User sees error message for invalid credentials
|
||||
# AC3: User account locks after 5 failed attempts
|
||||
# AC4: User can reset forgotten password
|
||||
```
|
||||
|
||||
### Expected Behavior:
|
||||
|
||||
**Step 2 should apply the 3-question decision tree:**
|
||||
|
||||
**AC1: "User can log in with valid email and password"**
|
||||
```
|
||||
Question 1: User-facing? YES
|
||||
Question 2: Needs FE+BE integration? YES
|
||||
Question 3: Critical gate? YES
|
||||
→ E2E BDD Test (root tests/ directory)
|
||||
Format: Playwright with Gherkin (Given/When/Then)
|
||||
```
|
||||
|
||||
**AC2: "User sees error message for invalid credentials"**
|
||||
```
|
||||
Question 1: User-facing? YES
|
||||
Question 2: Needs FE+BE integration? YES
|
||||
Question 3: Critical gate? NO (variation, not critical path)
|
||||
→ Frontend BDD Test (frontend/tests/ directory)
|
||||
Format: Playwright with Gherkin (Given/When/Then)
|
||||
OR Backend Unit Test (faster)
|
||||
```
|
||||
|
||||
**AC3: "User account locks after 5 failed attempts"**
|
||||
```
|
||||
Question 1: User-facing? YES (affects user)
|
||||
Question 2: Needs FE+BE integration? YES
|
||||
Question 3: Critical gate? NO (security feature, but not critical happy path)
|
||||
→ Backend Unit Test (preferred - faster, more edge cases to cover)
|
||||
```
|
||||
|
||||
### Expected Output:
|
||||
|
||||
```markdown
|
||||
## Test Strategy
|
||||
|
||||
**E2E BDD Tests (1) - Playwright with Gherkin:**
|
||||
|
||||
### Test: User Login Happy Path
|
||||
**Location:** `tests/features/auth/login.feature`
|
||||
**Format:** Gherkin/BDD (Given/When/Then)
|
||||
|
||||
Scenario: User logs in with valid credentials
|
||||
Given user has registered account with email "user@example.com"
|
||||
When user logs in with email "user@example.com" and password "validPass123"
|
||||
Then user sees their dashboard
|
||||
|
||||
**Frontend BDD Tests (1) - Playwright with Gherkin:**
|
||||
|
||||
### Test: Invalid Credentials Error Message
|
||||
**Location:** `frontend/tests/features/auth/login-errors.feature`
|
||||
**Format:** Gherkin/BDD (Given/When/Then)
|
||||
|
||||
Scenario: User sees error for invalid credentials
|
||||
Given user is on login page
|
||||
When user enters invalid credentials
|
||||
Then user sees error message "Invalid email or password"
|
||||
|
||||
**Backend Unit Tests (3):**
|
||||
- Account locking after 5 failed attempts
|
||||
- Account locking timer (30 minutes)
|
||||
- Concurrent login attempt handling
|
||||
```
|
||||
|
||||
### Key Validations:
|
||||
|
||||
**CRITICAL: Gherkin Format Enforcement**
|
||||
- E2E BDD section says "Playwright with Gherkin"
|
||||
- Frontend BDD section says "Playwright with Gherkin"
|
||||
- Both sections specify "Format: Gherkin/BDD (Given/When/Then)"
|
||||
- Workflow states: "ALL Playwright tests use Gherkin/BDD format"
|
||||
- Clarifies: "The ONLY difference: E2E uses real backend, frontend uses mocked backend"
|
||||
|
||||
**Pyramid/Trophy Strategy**
|
||||
- Only 1 E2E test (critical happy path)
|
||||
- Edge cases (locking, errors) at unit level
|
||||
- Decision tree applied to each AC
|
||||
- No duplicate coverage
|
||||
|
||||
**BDD Principles**
|
||||
- ONE When per scenario
|
||||
- Declarative language (user "logs in", not "clicks button")
|
||||
- Specific entity names ("user@example.com")
|
||||
|
||||
---
|
||||
|
||||
## Test 4: Dev Story - BDD Test Validation (Step 4.5)
|
||||
|
||||
**Objective:** Verify dev-story workflow checks for BDD tests BEFORE implementation and enforces Gherkin format.
|
||||
|
||||
### Setup:
|
||||
You'll need a story file without E2E BDD tests defined.
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# 1. Create a story with user-facing ACs but NO E2E tests
|
||||
# 2. Run dev-story workflow
|
||||
/bmm:workflows:dev-story
|
||||
|
||||
# Provide story path when prompted
|
||||
```
|
||||
|
||||
### Expected Behavior at Step 4.5:
|
||||
|
||||
**If E2E BDD tests are MISSING:**
|
||||
```
|
||||
⚠️ **BDD TESTS NOT DEFINED**
|
||||
|
||||
This story has user-facing acceptance criteria but no E2E BDD tests defined.
|
||||
|
||||
**RECOMMENDED APPROACH (Test-First):**
|
||||
Run `*atdd` workflow to define BDD acceptance tests BEFORE implementation.
|
||||
- E2E BDD tests act as acceptance gates (if they pass, ship to production)
|
||||
- Tests replace manual testing
|
||||
- Following pyramid/trophy strategy prevents test explosion
|
||||
|
||||
**OPTIONS:**
|
||||
1. **HALT** - Run ATDD workflow first (recommended for test-first approach)
|
||||
2. **CONTINUE** - Implement now, add tests during implementation (less ideal)
|
||||
3. **SKIP** - Story has no user-facing behavior requiring E2E tests
|
||||
|
||||
Choose approach:
|
||||
```
|
||||
|
||||
**If E2E BDD tests EXIST:**
|
||||
```
|
||||
✅ **E2E BDD TESTS FOUND**
|
||||
|
||||
Test-first approach detected! E2E BDD acceptance tests are defined.
|
||||
|
||||
**Your implementation will be validated against:**
|
||||
- E2E BDD tests in root `tests/` directory (acceptance gates)
|
||||
- Unit tests you create during implementation (edge cases, variations)
|
||||
|
||||
Proceeding with implementation...
|
||||
```
|
||||
|
||||
**If story is TECHNICAL (no user-facing ACs):**
|
||||
```
|
||||
ℹ️ **TECHNICAL STORY - NO E2E TESTS REQUIRED**
|
||||
|
||||
This story has only technical acceptance criteria (not user-facing).
|
||||
Unit tests will be sufficient for validation.
|
||||
|
||||
Proceeding with implementation...
|
||||
```
|
||||
|
||||
### Validation:
|
||||
- Step 4.5 executes before implementation
|
||||
- Warns when E2E tests missing for user-facing ACs
|
||||
- Offers HALT option to run ATDD first
|
||||
- Confirms when E2E tests found
|
||||
- Recognizes technical stories don't need E2E
|
||||
|
||||
---
|
||||
|
||||
## Test 5: Dev Story - Test Authoring with Gherkin (Step 6)
|
||||
|
||||
**Objective:** Verify dev-story Step 6 guides developers to use Gherkin format for ALL Playwright tests.
|
||||
|
||||
### Expected Guidance in Step 6:
|
||||
|
||||
```markdown
|
||||
**CRITICAL: ALL PLAYWRIGHT TESTS USE GHERKIN/BDD FORMAT**
|
||||
- Both root `tests/` and `frontend/tests/` use Playwright with Gherkin
|
||||
- Same BDD principles apply to both (ONE When, declarative, specific entities)
|
||||
- The ONLY difference: E2E uses real backend, frontend uses mocked backend
|
||||
|
||||
**E2E BDD Tests (root `tests/` directory) - Playwright with Gherkin**:
|
||||
- **Format**: Playwright with Gherkin/BDD (Given/When/Then)
|
||||
- ONLY if defined in ATDD workflow (should already exist if needed)
|
||||
- ONLY for user-facing acceptance criteria requiring full stack integration
|
||||
|
||||
**Frontend BDD Tests (`frontend/tests/` directory) - Playwright with Gherkin**:
|
||||
- **Format**: Playwright with Gherkin/BDD (Given/When/Then)
|
||||
- Frontend component integration tests
|
||||
- UI workflows that don't need real backend (mocked API)
|
||||
- Visual validation
|
||||
```
|
||||
|
||||
### Test Scenario:
|
||||
Developer needs to add a frontend integration test for form validation.
|
||||
|
||||
### Expected Developer Action:
|
||||
```gherkin
|
||||
# File: frontend/tests/features/forms/registration-validation.feature
|
||||
|
||||
Feature: Registration Form Validation
|
||||
|
||||
Scenario: Email format validation
|
||||
Given user is on registration page
|
||||
When user enters invalid email "notanemail"
|
||||
And user clicks submit
|
||||
Then user sees error "Please enter a valid email address"
|
||||
```
|
||||
|
||||
**NOT this (pure Playwright without Gherkin):**
|
||||
```typescript
|
||||
// WRONG - Missing Gherkin format
|
||||
test('validates email format', async ({ page }) => {
|
||||
await page.goto('/register');
|
||||
await page.fill('[name="email"]', 'notanemail');
|
||||
await page.click('button[type="submit"]');
|
||||
await expect(page.locator('.error')).toHaveText('Please enter a valid email address');
|
||||
});
|
||||
```
|
||||
|
||||
### Validation:
|
||||
- Step 6 emphasizes ALL Playwright tests use Gherkin
|
||||
- Clear distinction: real backend (E2E) vs mocked (frontend)
|
||||
- Format specification for both E2E and Frontend sections
|
||||
- BDD principles referenced (ONE When, declarative)
|
||||
|
||||
---
|
||||
|
||||
## Test 6: TEA Agent - BDD Knowledge Loading
|
||||
|
||||
**Objective:** Verify TEA agent can load BDD standards knowledge fragment.
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# 1. Activate TEA agent
|
||||
/bmm:agents:tea
|
||||
|
||||
# 2. Ask TEA to load BDD knowledge
|
||||
# You can ask: "What are the BDD standards?"
|
||||
# Or: "Load bdd-standards knowledge"
|
||||
# Or: "Tell me about the one-when-per-scenario rule"
|
||||
```
|
||||
|
||||
### Expected Behavior:
|
||||
|
||||
TEA agent should:
|
||||
1. Search its knowledge index (`tea-index.csv`)
|
||||
2. Find the `bdd-standards` entry
|
||||
3. Load `testarch/knowledge/bdd-standards.md`
|
||||
4. Respond with BDD principles
|
||||
|
||||
### Expected Response Content:
|
||||
|
||||
Should include:
|
||||
- **One When Per Scenario** rule and examples
|
||||
- **Declarative vs Imperative** style guide with comparison table
|
||||
- **Specific Entity Names** when 2+ entities exist
|
||||
- **Combinatorial Explosion Prevention** strategies
|
||||
- Examples of CORRECT vs INCORRECT test scenarios
|
||||
|
||||
### Validation:
|
||||
- `bdd-standards` entry exists in `testarch/tea-index.csv`
|
||||
- Knowledge file exists at `testarch/knowledge/bdd-standards.md`
|
||||
- TEA agent can discover and load the knowledge
|
||||
- Content includes all 6 core BDD principles
|
||||
|
||||
---
|
||||
|
||||
## Test 7: Test Design Workflow - Pyramid/Trophy Strategy
|
||||
|
||||
**Objective:** Verify test-design workflow applies pyramid/trophy strategy with Gherkin format enforcement.
|
||||
|
||||
### Steps:
|
||||
|
||||
```bash
|
||||
# Run test design workflow
|
||||
/bmm:workflows:test-design
|
||||
```
|
||||
|
||||
### Expected Step 3 Guidance:
|
||||
|
||||
Should include the same 3-question decision tree as ATDD:
|
||||
1. User-facing?
|
||||
2. Needs FE+BE integration?
|
||||
3. Critical acceptance gate?
|
||||
|
||||
Should include Gherkin format enforcement:
|
||||
```markdown
|
||||
**CRITICAL: ALL PLAYWRIGHT TESTS USE GHERKIN/BDD FORMAT**
|
||||
- Both root `tests/` (E2E) and `frontend/tests/` use Gherkin syntax
|
||||
- The ONLY difference: E2E uses real backend, frontend uses mocked backend
|
||||
- Same BDD principles apply to both: ONE When, declarative, specific entities
|
||||
```
|
||||
|
||||
Should include test distribution targets:
|
||||
- E2E BDD: 5-10 tests per epic
|
||||
- Frontend BDD: 15-25% of total tests (Playwright+Gherkin)
|
||||
- Unit Tests: 70-80% of total tests
|
||||
|
||||
### Validation:
|
||||
- Step 3 section 2 has pyramid/trophy strategy
|
||||
- Gherkin format emphasized for ALL Playwright tests
|
||||
- Decision tree matches ATDD workflow
|
||||
- Test distribution percentages provided
|
||||
- Real-world example included
|
||||
|
||||
---
|
||||
|
||||
## Integration Test: Complete Workflow End-to-End
|
||||
|
||||
**Objective:** Test the complete flow from epic planning through implementation.
|
||||
|
||||
### Complete Flow:
|
||||
|
||||
```bash
|
||||
# 1. Epic Planning - Identify test strategy
|
||||
/bmm:workflows:create-epics-and-stories
|
||||
# → Verify Step 4.5 categorizes ACs
|
||||
# → Note which ACs become E2E vs unit tests
|
||||
|
||||
# 2. Sprint Planning - SM offers ATDD
|
||||
/bmm:agents:sm
|
||||
# → Verify *atdd appears in menu
|
||||
|
||||
# 3. Define BDD Tests FIRST
|
||||
# Run *atdd from SM agent
|
||||
# → Verify pyramid/trophy decision tree applied
|
||||
# → Verify Gherkin format enforced for ALL Playwright tests
|
||||
# → Verify E2E tests created in root tests/ directory
|
||||
# → Verify unit test stubs created
|
||||
|
||||
# 4. Implement Story
|
||||
/bmm:workflows:dev-story
|
||||
# → Step 4.5 should find E2E BDD tests
|
||||
# → Step 6 should guide proper test placement
|
||||
# → Step 6 should enforce Gherkin for ALL Playwright tests
|
||||
# → Should NOT duplicate E2E tests at unit level
|
||||
|
||||
# 5. Verify all tests pass
|
||||
# Run test suite
|
||||
# → E2E BDD tests pass = ready to ship
|
||||
```
|
||||
|
||||
### Success Criteria:
|
||||
|
||||
- Test strategy planned at epic level
|
||||
- BDD tests defined BEFORE implementation
|
||||
- ALL Playwright tests use Gherkin format
|
||||
- Pyramid/trophy strategy followed (few E2E, many unit)
|
||||
- No duplicate coverage across test levels
|
||||
- Developer guided at every step
|
||||
|
||||
---
|
||||
|
||||
## Automated Validation Script
|
||||
|
||||
You can also run this automated check:
|
||||
|
||||
```bash
|
||||
# Check all critical files have been updated
|
||||
echo "Checking BDD implementation files..."
|
||||
|
||||
# 1. SM agent has ATDD in menu
|
||||
grep -q "atdd" src/modules/bmm/agents/sm.agent.yaml && echo "✅ SM agent menu" || echo "❌ SM agent menu"
|
||||
|
||||
# 2. Epic creation has Step 4.5
|
||||
grep -q "4.5" src/modules/bmm/workflows/3-solutioning/create-epics-and-stories/steps/step-03-create-stories.md && echo "✅ Epic Step 4.5" || echo "❌ Epic Step 4.5"
|
||||
|
||||
# 3. ATDD has pyramid/trophy in Step 2
|
||||
grep -q "PYRAMID/TROPHY" src/modules/bmm/workflows/testarch/atdd/instructions.md && echo "✅ ATDD pyramid/trophy" || echo "❌ ATDD pyramid/trophy"
|
||||
|
||||
# 4. ATDD enforces Gherkin for ALL Playwright
|
||||
grep -q "ALL PLAYWRIGHT TESTS USE GHERKIN" src/modules/bmm/workflows/testarch/atdd/instructions.md && echo "✅ ATDD Gherkin enforcement" || echo "❌ ATDD Gherkin enforcement"
|
||||
|
||||
# 5. Dev story has Step 4.5
|
||||
grep -q "4.5" src/modules/bmm/workflows/4-implementation/dev-story/instructions.xml && echo "✅ Dev story Step 4.5" || echo "❌ Dev story Step 4.5"
|
||||
|
||||
# 6. Dev story enforces Gherkin for ALL Playwright
|
||||
grep -q "ALL PLAYWRIGHT TESTS USE GHERKIN" src/modules/bmm/workflows/4-implementation/dev-story/instructions.xml && echo "✅ Dev story Gherkin enforcement" || echo "❌ Dev story Gherkin enforcement"
|
||||
|
||||
# 7. Test Design has Gherkin enforcement
|
||||
grep -q "ALL PLAYWRIGHT TESTS USE GHERKIN" src/modules/bmm/workflows/testarch/test-design/instructions.md && echo "✅ Test Design Gherkin enforcement" || echo "❌ Test Design Gherkin enforcement"
|
||||
|
||||
# 8. BDD knowledge exists
|
||||
test -f src/modules/bmm/testarch/knowledge/bdd-standards.md && echo "✅ BDD knowledge file" || echo "❌ BDD knowledge file"
|
||||
|
||||
# 9. BDD knowledge indexed
|
||||
grep -q "bdd-standards" src/modules/bmm/testarch/tea-index.csv && echo "✅ BDD knowledge indexed" || echo "❌ BDD knowledge indexed"
|
||||
|
||||
echo ""
|
||||
echo "Testing complete!"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Issues & Troubleshooting
|
||||
|
||||
### Issue: ATDD workflow not appearing in SM menu
|
||||
**Cause:** Menu item not added to sm.agent.yaml
|
||||
**Fix:** Verify `trigger: AT or fuzzy match on atdd` exists in menu section
|
||||
|
||||
### Issue: Step 4.5 not running in epic creation
|
||||
**Cause:** Step numbering might be off
|
||||
**Fix:** Check that Step 4.5 exists between Step 4 and Step 5
|
||||
|
||||
### Issue: BDD knowledge not loading for TEA
|
||||
**Cause:** Either CSV index or knowledge file missing
|
||||
**Fix:** Verify both files exist:
|
||||
- `testarch/tea-index.csv` (has bdd-standards entry)
|
||||
- `testarch/knowledge/bdd-standards.md` (knowledge content)
|
||||
|
||||
### Issue: Workflow still suggests "Playwright" without "Gherkin"
|
||||
**Cause:** Old terminology not corrected
|
||||
**Fix:** Search for "Playwright Tests" and update to "Frontend BDD Tests - Playwright with Gherkin"
|
||||
|
||||
---
|
||||
|
||||
## Summary: What Should Work
|
||||
|
||||
After implementing this BDD strategy:
|
||||
|
||||
1. **Epic Planning** - ACs categorized into E2E vs unit tests
|
||||
2. **SM Agent** - Offers ATDD workflow for test-first approach
|
||||
3. **ATDD Workflow** - Creates BDD tests BEFORE implementation using pyramid/trophy strategy
|
||||
4. **ALL Playwright tests** - Use Gherkin/BDD format (Given/When/Then)
|
||||
5. **Dev Story** - Validates BDD tests exist, guides proper test placement
|
||||
6. **TEA Agent** - Has BDD expertise via knowledge base
|
||||
7. **Test Strategy** - 5-10 E2E, 15-25% frontend BDD, 70-80% unit tests
|
||||
|
||||
**Key Success Metric:** If E2E BDD tests pass, you can ship to production with confidence.
|
||||
|
|
@ -12,14 +12,19 @@ agent:
|
|||
|
||||
persona:
|
||||
role: Senior Software Engineer
|
||||
identity: Executes approved stories with strict adherence to acceptance criteria, using Story Context XML and existing code to minimize rework and hallucinations.
|
||||
identity: Executes approved stories with strict adherence to acceptance criteria, using Story Context XML and existing code to minimize rework and hallucinations. Implements tests following BDD principles with declarative language describing user intent.
|
||||
communication_style: "Ultra-succinct. Speaks in file paths and AC IDs - every statement citable. No fluff, all precision."
|
||||
principles: |
|
||||
- The Story File is the single source of truth - tasks/subtasks sequence is authoritative over any model priors
|
||||
- Follow red-green-refactor cycle: write failing test, make it pass, improve code while keeping tests green
|
||||
- Test scenarios follow BDD: Given (setup) - When (ONE action) - Then (outcomes)
|
||||
- Use declarative test language describing user intent, not UI mechanics
|
||||
- Each test file focuses on one entry point OR one sub-flow, not mixed
|
||||
- Preserve declarative Gherkin language in test descriptions
|
||||
- Never implement anything not mapped to a specific task/subtask in the story file
|
||||
- All existing tests must pass 100% before story is ready for review
|
||||
- Every task/subtask must be covered by comprehensive unit tests before marking complete
|
||||
- Consult tea agent for BDD test structure guidance when unclear
|
||||
- Project context provides coding standards but never overrides story requirements
|
||||
- Find if this exists, if it does, always treat it as the bible I plan and execute against: `**/project-context.md`
|
||||
|
||||
|
|
|
|||
|
|
@ -11,14 +11,15 @@ agent:
|
|||
|
||||
persona:
|
||||
role: Technical Scrum Master + Story Preparation Specialist
|
||||
identity: Certified Scrum Master with deep technical background. Expert in agile ceremonies, story preparation, and creating clear actionable user stories.
|
||||
identity: Certified Scrum Master with deep technical background. Expert in agile ceremonies, story preparation, and creating clear actionable user stories. Champions test-first development with BDD acceptance tests defined before implementation.
|
||||
communication_style: "Crisp and checklist-driven. Every word has a purpose, every requirement crystal clear. Zero tolerance for ambiguity."
|
||||
principles: |
|
||||
- Strict boundaries between story prep and implementation
|
||||
- Stories are single source of truth
|
||||
- BDD acceptance tests defined BEFORE code - if tests pass, ship to production
|
||||
- Perfect alignment between PRD and dev execution
|
||||
- Enable efficient sprints
|
||||
- Deliver developer-ready specs with precise handoffs
|
||||
- Deliver developer-ready specs with BDD acceptance gates and precise handoffs
|
||||
|
||||
critical_actions:
|
||||
- "When running *create-story, always run as *yolo. Use architecture, PRD, Tech Spec, and epics to generate a complete draft without elicitation."
|
||||
|
|
@ -42,6 +43,10 @@ agent:
|
|||
data: "{project-root}/_bmad/_config/agent-manifest.csv"
|
||||
description: "[ER] Facilitate team retrospective after an epic is completed (Optional)"
|
||||
|
||||
- trigger: AT or fuzzy match on atdd
|
||||
workflow: "{project-root}/_bmad/bmm/workflows/testarch/atdd/workflow.yaml"
|
||||
description: "[AT] Define BDD acceptance tests BEFORE implementation (test-first approach)"
|
||||
|
||||
- trigger: CC or fuzzy match on correct-course
|
||||
workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/correct-course/workflow.yaml"
|
||||
description: "[CC] Execute correct-course task (When implementation is off-track)"
|
||||
|
|
|
|||
|
|
@ -11,15 +11,19 @@ agent:
|
|||
hasSidecar: false
|
||||
|
||||
persona:
|
||||
role: Master Test Architect
|
||||
identity: Test architect specializing in CI/CD, automated frameworks, and scalable quality gates.
|
||||
role: Master Test Architect + BDD Specialist
|
||||
identity: Test architect specializing in behavior-driven development, declarative test design, CI/CD automation, and preventing combinatorial explosion through strategic test architecture.
|
||||
communication_style: "Blends data with gut instinct. 'Strong opinions, weakly held' is their mantra. Speaks in risk calculations and impact assessments."
|
||||
principles: |
|
||||
- ONE When per scenario - each test validates exactly ONE behavior
|
||||
- Declarative style: describe WHAT users achieve, not HOW the UI implements it
|
||||
- Separate entry points from sub-flows when 3+ × 3+ variations exist to prevent combinatorial explosion
|
||||
- Use specific entity names when 2+ entities of same type exist in a scenario
|
||||
- Permission tests are always separate scenarios from happy path tests
|
||||
- Use And to extend Given/When/Then steps, never multiple When-Then-When-Then chains
|
||||
- Risk-based testing - depth scales with impact
|
||||
- Quality gates backed by data
|
||||
- Tests mirror usage patterns
|
||||
- Flakiness is critical technical debt
|
||||
- Tests first AI implements suite validates
|
||||
- Calculate risk vs value for every testing decision
|
||||
|
||||
critical_actions:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,171 @@
|
|||
# BDD Test Standards Reference
|
||||
|
||||
## Core Rules
|
||||
|
||||
### One When Per Scenario
|
||||
- Each scenario tests exactly ONE behavior
|
||||
- Use only ONE "When" statement per test
|
||||
- Use "And" to extend Given, When, or Then steps
|
||||
- Never use multiple When-Then-When-Then patterns
|
||||
- Split scenarios immediately if testing multiple behaviors
|
||||
|
||||
### Declarative Style (WHAT not HOW)
|
||||
Describe what the user wants to achieve, not how the UI implements it.
|
||||
|
||||
| Imperative (Avoid) | Declarative (Use) |
|
||||
|----------------------|---------------------|
|
||||
| taps the 'Save' button | saves the changes |
|
||||
| navigates to Settings screen | accesses settings |
|
||||
| the Login screen is displayed | user can log in |
|
||||
| sees the Submit button | user can submit the form |
|
||||
| enters "john@example.com" in email field | provides email "john@example.com" |
|
||||
|
||||
### Given-When-Then Structure
|
||||
- **Given** = Preconditions/setup (use "And" for multiple conditions)
|
||||
- **When** = The ONE action being tested (use "And" only for action details)
|
||||
- **Then** = Expected outcomes (use "And" for multiple assertions)
|
||||
|
||||
## Entity Naming
|
||||
|
||||
### Use Specific Values When:
|
||||
- Multiple entities of same type (2+ users, groups, items)
|
||||
- Complex scenarios with relationships (owner-admin-member)
|
||||
- Need to track which entity performs which action
|
||||
- Test involves entity names being displayed
|
||||
- Business rules about specific entities (last admin, single owner)
|
||||
|
||||
### Generic is Acceptable When:
|
||||
- Single entity where name doesn't matter for the test
|
||||
- Very short scenario (3-4 lines total)
|
||||
- No ambiguity about which entity acts
|
||||
- Simple CRUD operation on one entity
|
||||
|
||||
### Naming Patterns:
|
||||
- **People**: Sarah, Mike, Aisha, Chen, Maria, Raj (realistic, diverse)
|
||||
- **Groups**: "Family", "Work Team", "Book Club", "Grade 5A" (descriptive)
|
||||
- **Counts**: "2 admins", "3 members" (specific, never "multiple" or "several")
|
||||
|
||||
## Test Architecture
|
||||
|
||||
### Combinatorial Explosion Prevention
|
||||
|
||||
**Recognize the problem:**
|
||||
- 3+ entry points × 3+ sub-flow variations = separate them
|
||||
- Same sub-flow steps repeated across multiple entry point tests
|
||||
- Sub-flows can be tested independently of entry points
|
||||
- Changes to one sub-flow would require updating 5+ tests
|
||||
- About to write 50+ tests for one feature = stop and restructure
|
||||
|
||||
**Solution: Separate Concerns**
|
||||
|
||||
```gherkin
|
||||
## Entry Point Tests (focus on flow differences)
|
||||
Scenario: Add member via email invitation
|
||||
Given... (references: KYC sub-flow, Payment sub-flow)
|
||||
When user invites via email...
|
||||
Then...
|
||||
|
||||
Scenario: Add member via invitation code
|
||||
Given... (references: KYC sub-flow, Payment sub-flow)
|
||||
When user shares invitation code...
|
||||
Then...
|
||||
|
||||
## Sub-Flow Tests (reusable variations)
|
||||
Scenario: KYC - Already verified within 12 months
|
||||
Given user completed KYC 6 months ago...
|
||||
When user proceeds...
|
||||
Then existing verification confirmed...
|
||||
|
||||
Scenario: KYC - First time verification
|
||||
Given user has never completed KYC...
|
||||
When user submits identification...
|
||||
Then verification completed...
|
||||
```
|
||||
|
||||
### Priority Model
|
||||
- **P0**: Critical E2E smoke tests (5-10 tests maximum)
|
||||
- **P1**: Entry point tests
|
||||
- **P2**: Variation/sub-flow tests
|
||||
|
||||
### Permission Tests
|
||||
Always separate from happy path:
|
||||
- Happy path scenario tests the action succeeds
|
||||
- Separate scenario tests unauthorized user cannot perform action
|
||||
|
||||
## Pyramid/Trophy Test Strategy
|
||||
|
||||
### Test Level Decision Tree
|
||||
|
||||
For EVERY Acceptance Criterion:
|
||||
|
||||
```
|
||||
Is this user-facing?
|
||||
├─ NO → Unit Test (technical/internal)
|
||||
└─ YES → Does this REQUIRE frontend + backend integration?
|
||||
├─ NO → Unit Test (can test in isolation)
|
||||
└─ YES → Is this CRITICAL (ship-to-production gate)?
|
||||
├─ NO → Playwright Test (frontend integration only)
|
||||
└─ YES → E2E BDD Test (root tests/ directory)
|
||||
```
|
||||
|
||||
### Strategic Test Placement
|
||||
|
||||
**E2E BDD Tests (root `tests/` directory)**:
|
||||
- ONLY for true backend + frontend integration
|
||||
- ONLY for critical acceptance criteria (if these pass, ship to production)
|
||||
- REPLACES manual testing - these are the production gates
|
||||
- User-facing happy paths that cross the full stack
|
||||
- **Characteristics**: Highest confidence, slowest, most brittle
|
||||
- **Target**: 5-10 smoke tests per epic maximum
|
||||
|
||||
**Playwright Tests (`frontend/tests/` directory)**:
|
||||
- Frontend component integration (multiple components working together)
|
||||
- UI workflows that don't need real backend (can use mocked API)
|
||||
- Visual validation and interaction testing
|
||||
- **Characteristics**: Fast, stable, frontend-focused
|
||||
|
||||
**Backend Unit Tests (`backend/tests/` directory)**:
|
||||
- API contract validation
|
||||
- Business logic variations and ALL edge cases
|
||||
- Error handling and negative cases
|
||||
- Data transformations
|
||||
- **Characteristics**: Fastest, most stable
|
||||
|
||||
**Frontend Unit Tests (`frontend/src/components/*/tests/` directory)**:
|
||||
- Component behavior (buttons, forms, modals)
|
||||
- Interaction edge cases
|
||||
- Rendering logic
|
||||
- **Characteristics**: Fast, isolated, granular
|
||||
|
||||
### Avoid Duplicate Coverage
|
||||
|
||||
**DO NOT test the same behavior at multiple levels:**
|
||||
- CORRECT: E2E for critical happy path, unit tests for variations
|
||||
- WRONG: E2E + API + component tests for same behavior
|
||||
|
||||
### Example: Login Feature
|
||||
|
||||
**E2E BDD (1 test):**
|
||||
- User logs in with valid credentials → sees dashboard
|
||||
|
||||
**Backend Unit (5 tests):**
|
||||
- Invalid password, expired token, account locked, rate limiting, 2FA flow
|
||||
|
||||
**Frontend Unit (3 tests):**
|
||||
- Form validation, password visibility toggle, error message display
|
||||
|
||||
**Total**: 9 tests, only 1 slow E2E
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before finalizing any test suite:
|
||||
- [ ] Every scenario has exactly ONE When statement
|
||||
- [ ] All language is declarative (no UI element references)
|
||||
- [ ] Multi-entity scenarios use specific, named values
|
||||
- [ ] Entry points and sub-flows are separated (no explosion)
|
||||
- [ ] Entry point tests reference applicable sub-flow tests
|
||||
- [ ] E2E smoke suite identified (5-10 P0 tests)
|
||||
- [ ] Priorities assigned correctly (P0/P1/P2)
|
||||
- [ ] Permission scenarios separate from happy paths
|
||||
- [ ] Test distribution follows pyramid (most tests at unit level)
|
||||
- [ ] No duplicate coverage across test levels
|
||||
|
|
@ -20,6 +20,7 @@ test-priorities,Test Priorities Matrix,"P0–P3 criteria, coverage targets, exec
|
|||
test-healing-patterns,Test Healing Patterns,"Common failure patterns and automated fixes","healing,debugging,patterns",knowledge/test-healing-patterns.md
|
||||
selector-resilience,Selector Resilience,"Robust selector strategies and debugging techniques","selectors,locators,debugging",knowledge/selector-resilience.md
|
||||
timing-debugging,Timing Debugging,"Race condition identification and deterministic wait fixes","timing,async,debugging",knowledge/timing-debugging.md
|
||||
bdd-standards,BDD Standards,"Behavior-driven development principles: one-when-per-scenario, declarative style, entity naming, combinatorial explosion prevention, pyramid/trophy test strategy","bdd,test-design,gherkin,architecture",knowledge/bdd-standards.md
|
||||
overview,Playwright Utils Overview,"Installation, design principles, fixture patterns","playwright-utils,fixtures",knowledge/overview.md
|
||||
api-request,API Request,"Typed HTTP client, schema validation","api,playwright-utils",knowledge/api-request.md
|
||||
network-recorder,Network Recorder,"HAR record/playback, CRUD detection","network,playwright-utils",knowledge/network-recorder.md
|
||||
|
|
|
|||
|
|
|
@ -196,6 +196,63 @@ After all stories for an epic are complete:
|
|||
- Verify all FRs for the epic are covered
|
||||
- Get user confirmation to proceed to next epic
|
||||
|
||||
### 4.5 Identify E2E BDD Test Scenarios for Epic
|
||||
|
||||
**CRITICAL - TEST STRATEGY PLANNING:**
|
||||
|
||||
For each completed epic, analyze which acceptance criteria should become E2E BDD tests vs unit tests using the Pyramid/Trophy Strategy.
|
||||
|
||||
**Test Level Decision Tree - Apply to Each AC:**
|
||||
|
||||
```
|
||||
Is this user-facing?
|
||||
├─ NO → Unit Test (technical/internal)
|
||||
└─ YES → Does this REQUIRE frontend + backend integration?
|
||||
├─ NO → Unit Test (can test in isolation)
|
||||
└─ YES → Is this CRITICAL (ship-to-production gate)?
|
||||
├─ NO → Playwright Test (frontend integration only)
|
||||
└─ YES → E2E BDD Test (root tests/ directory)
|
||||
```
|
||||
|
||||
**For Each Story's Acceptance Criteria, Categorize:**
|
||||
|
||||
1. **User-Facing AC** → Candidate for E2E BDD test
|
||||
- Tests user behavior from frontend through backend
|
||||
- Validates complete user journey
|
||||
- Example: "Given user on login page, When enters valid credentials, Then sees dashboard"
|
||||
|
||||
2. **Technical AC** → Unit test only
|
||||
- Tests internal behavior, not user-visible
|
||||
- Can be tested in isolation
|
||||
- Example: "API returns 401 for expired tokens"
|
||||
|
||||
3. **Variation/Edge Case** → Unit test preferred
|
||||
- Tests alternative paths or boundaries
|
||||
- Faster to test at unit level
|
||||
- Example: "Password must contain special character"
|
||||
|
||||
**Output Test Strategy Summary for Epic:**
|
||||
|
||||
After stories are approved, present:
|
||||
|
||||
```
|
||||
## Epic {N} Test Strategy Summary
|
||||
|
||||
**E2E BDD Test Scenarios ({count}):**
|
||||
- BDD-E2E-{N}.{M}.{X}: {AC description} - Priority: P0/P1/P2
|
||||
|
||||
**Unit Test Coverage ({count}):**
|
||||
- Unit-{N}.{M}.{Y}: {AC description} - Test Level: Backend/Frontend
|
||||
|
||||
**Coverage Summary:**
|
||||
- E2E BDD Tests: {e2e_count} (focus on critical paths)
|
||||
- Unit Tests: {unit_count} (variations and technical validation)
|
||||
|
||||
**Target:** 5-10 E2E tests per epic maximum
|
||||
```
|
||||
|
||||
Include this summary in the epic section of {outputFile}.
|
||||
|
||||
### 5. Repeat for All Epics
|
||||
|
||||
Continue the process for each epic in the approved list, processing them in order (Epic 1, Epic 2, etc.).
|
||||
|
|
|
|||
|
|
@ -221,6 +221,66 @@
|
|||
</check>
|
||||
</step>
|
||||
|
||||
<step n="4.5" goal="Check for E2E BDD Acceptance Tests (Test-First Validation)">
|
||||
<critical>BDD tests should be defined BEFORE implementation (test-first approach)</critical>
|
||||
|
||||
<action>Check if epic has E2E BDD test scenarios defined:
|
||||
- Look in epic file for "E2E BDD Test Scenarios" or "Test Strategy" section
|
||||
- Look for test files in root `tests/` directory matching this story
|
||||
- Check if ATDD workflow was run (look for ATDD checklist file)
|
||||
</action>
|
||||
|
||||
<check if="E2E BDD tests are NOT defined AND story has user-facing acceptance criteria">
|
||||
<output>⚠️ **BDD TESTS NOT DEFINED**
|
||||
|
||||
This story has user-facing acceptance criteria but no E2E BDD tests defined.
|
||||
|
||||
**RECOMMENDED APPROACH (Test-First):**
|
||||
Run `*atdd` workflow to define BDD acceptance tests BEFORE implementation.
|
||||
- E2E BDD tests act as acceptance gates (if they pass, ship to production)
|
||||
- Tests replace manual testing
|
||||
- Following pyramid/trophy strategy prevents test explosion
|
||||
|
||||
**OPTIONS:**
|
||||
1. **HALT** - Run ATDD workflow first (recommended for test-first approach)
|
||||
2. **CONTINUE** - Implement now, add tests during implementation (less ideal)
|
||||
3. **SKIP** - Story has no user-facing behavior requiring E2E tests
|
||||
|
||||
Choose approach:</output>
|
||||
|
||||
<ask>How would you like to proceed? (1=HALT for ATDD, 2=Continue with implementation, 3=Skip test check)</ask>
|
||||
|
||||
<action>Based on user choice:
|
||||
- Choice 1: HALT with message "Run *atdd workflow from SM or TEA agent to define BDD tests first"
|
||||
- Choice 2: Continue to step 5 (add warning to completion notes about missing BDD tests)
|
||||
- Choice 3: Continue to step 5 (note in completion that story doesn't require E2E tests)
|
||||
</action>
|
||||
</check>
|
||||
|
||||
<check if="E2E BDD tests ARE defined">
|
||||
<output>✅ **E2E BDD TESTS FOUND**
|
||||
|
||||
Test-first approach detected! E2E BDD acceptance tests are defined.
|
||||
|
||||
**Your implementation will be validated against:**
|
||||
- E2E BDD tests in root `tests/` directory (acceptance gates)
|
||||
- Unit tests you create during implementation (edge cases, variations)
|
||||
|
||||
Proceeding with implementation...
|
||||
</output>
|
||||
</check>
|
||||
|
||||
<check if="story has NO user-facing acceptance criteria">
|
||||
<output>ℹ️ **TECHNICAL STORY - NO E2E TESTS REQUIRED**
|
||||
|
||||
This story has only technical acceptance criteria (not user-facing).
|
||||
Unit tests will be sufficient for validation.
|
||||
|
||||
Proceeding with implementation...
|
||||
</output>
|
||||
</check>
|
||||
</step>
|
||||
|
||||
<step n="5" goal="Implement task following red-green-refactor cycle">
|
||||
<critical>FOLLOW THE STORY FILE TASKS/SUBTASKS SEQUENCE EXACTLY AS WRITTEN - NO DEVIATION</critical>
|
||||
|
||||
|
|
@ -252,11 +312,53 @@
|
|||
<critical>Do NOT propose to pause for review until Step 9 completion gates are satisfied</critical>
|
||||
</step>
|
||||
|
||||
<step n="6" goal="Author comprehensive tests">
|
||||
<step n="6" goal="Author comprehensive tests (Pyramid/Trophy Strategy)">
|
||||
<critical>Follow pyramid/trophy strategy - NOT every test at every level!</critical>
|
||||
|
||||
<action>**Test Strategy (from BDD principles):**
|
||||
|
||||
**CRITICAL: ALL PLAYWRIGHT TESTS USE GHERKIN/BDD FORMAT**
|
||||
- Both root `tests/` and `frontend/tests/` use Playwright with Gherkin
|
||||
- Same BDD principles apply to both (ONE When, declarative, specific entities)
|
||||
- The ONLY difference: E2E uses real backend, frontend uses mocked backend
|
||||
|
||||
**E2E BDD Tests (root `tests/` directory) - Playwright with Gherkin**:
|
||||
- **Format**: Playwright with Gherkin/BDD (Given/When/Then)
|
||||
- ONLY if defined in ATDD workflow (should already exist if needed)
|
||||
- ONLY for user-facing acceptance criteria requiring full stack integration
|
||||
- If you need to add E2E tests, they MUST follow BDD principles:
|
||||
- ONE When per scenario
|
||||
- Declarative language (WHAT user achieves, not HOW UI implements)
|
||||
- Specific entity names when 2+ entities exist
|
||||
|
||||
**Frontend BDD Tests (`frontend/tests/` directory) - Playwright with Gherkin**:
|
||||
- **Format**: Playwright with Gherkin/BDD (Given/When/Then)
|
||||
- Frontend component integration tests
|
||||
- UI workflows that don't need real backend (mocked API)
|
||||
- Visual validation
|
||||
|
||||
**Backend Unit Tests (`backend/tests/` directory)**:
|
||||
- API contract validation (primary focus for backend changes)
|
||||
- Business logic variations and ALL edge cases
|
||||
- Error handling and negative cases
|
||||
- Data transformations
|
||||
- MOST tests should be here for backend work
|
||||
|
||||
**Frontend Unit Tests (`frontend/src/components/*/tests/` directory)**:
|
||||
- Component behavior in isolation
|
||||
- Interaction edge cases
|
||||
- Rendering logic
|
||||
- MOST tests should be here for frontend component work
|
||||
|
||||
**DO NOT test the same behavior at multiple levels!**
|
||||
- ✅ CORRECT: E2E for critical happy path, unit tests for variations
|
||||
- ❌ WRONG: E2E + API + component tests for same behavior
|
||||
</action>
|
||||
|
||||
<action>Create unit tests for business logic and core functionality introduced/changed by the task</action>
|
||||
<action>Add integration tests for component interactions specified in story requirements</action>
|
||||
<action>Include end-to-end tests for critical user flows when story requirements demand them</action>
|
||||
<action>Cover edge cases and error handling scenarios identified in story Dev Notes</action>
|
||||
<action>ONLY add E2E tests if they were defined in ATDD workflow or story explicitly requires them for critical user flows</action>
|
||||
<action>Cover ALL edge cases and error handling scenarios at unit test level (NOT E2E)</action>
|
||||
</step>
|
||||
|
||||
<step n="7" goal="Run validations and tests">
|
||||
|
|
|
|||
|
|
@ -191,62 +191,136 @@ Generates failing acceptance tests BEFORE implementation following TDD's red-gre
|
|||
|
||||
---
|
||||
|
||||
## Step 2: Select Test Levels and Strategy
|
||||
## Step 2: Select Test Levels and Strategy (PYRAMID/TROPHY APPROACH)
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Analyze Acceptance Criteria**
|
||||
1. **CRITICAL - Load Epic Test Strategy (if exists)**
|
||||
|
||||
For each acceptance criterion, determine:
|
||||
- Does it require full user journey? → E2E test
|
||||
- Does it test business logic/API contract? → API test
|
||||
- Does it validate UI component behavior? → Component test
|
||||
- Can it be unit tested? → Unit test
|
||||
**Check if epic has pre-defined test strategy:**
|
||||
- Look for "Test Strategy" or "E2E BDD Test Scenarios" section in epic file
|
||||
- If found: Use those as PRIMARY guidance for which ACs become E2E tests
|
||||
- If not found: Continue with manual analysis below
|
||||
|
||||
2. **Apply Test Level Selection Framework**
|
||||
2. **Analyze Acceptance Criteria Using Pyramid/Trophy Strategy**
|
||||
|
||||
**Knowledge Base Reference**: `test-levels-framework.md`
|
||||
**CORE PRINCIPLE**: E2E tests are EXPENSIVE - use sparingly only for true integration!
|
||||
|
||||
**E2E (End-to-End)**:
|
||||
- Critical user journeys (login, checkout, core workflow)
|
||||
- Multi-system integration
|
||||
- User-facing acceptance criteria
|
||||
- **Characteristics**: High confidence, slow execution, brittle
|
||||
**Knowledge Base References**:
|
||||
- `test-levels-framework.md`
|
||||
- `bdd-standards.md` (for BDD test structure)
|
||||
|
||||
**API (Integration)**:
|
||||
- Business logic validation
|
||||
- Service contracts
|
||||
For each acceptance criterion, apply this decision tree:
|
||||
|
||||
**QUESTION 1: Is this user-facing (tests behavior from user perspective)?**
|
||||
- NO → Unit test (technical AC, internal logic)
|
||||
- YES → Continue to Question 2
|
||||
|
||||
**QUESTION 2: Does this REQUIRE frontend + backend integration?**
|
||||
- NO → Unit test in frontend OR backend (can be tested in isolation)
|
||||
- YES → Continue to Question 3
|
||||
|
||||
**QUESTION 3: Is this a CRITICAL acceptance test (ship-to-production gate)?**
|
||||
- NO → Consider Playwright test in `frontend/tests/` (frontend integration only)
|
||||
- YES → E2E BDD test in root `tests/` directory
|
||||
|
||||
**STRATEGIC TEST PLACEMENT:**
|
||||
|
||||
**E2E BDD Tests (root `tests/` directory)**:
|
||||
- **ONLY** for true backend + frontend integration
|
||||
- **ONLY** for critical acceptance criteria (if these pass, ship to production)
|
||||
- **REPLACES manual testing** - these are the production gates
|
||||
- User-facing happy paths that cross the full stack
|
||||
- **Characteristics**: Highest confidence, slowest, most brittle
|
||||
- **Target**: 5-10 smoke tests per epic maximum
|
||||
|
||||
**Playwright Tests (`frontend/tests/` directory)**:
|
||||
- Frontend component integration (multiple components working together)
|
||||
- UI workflows that don't need real backend (can use mocked API)
|
||||
- Visual validation and interaction testing
|
||||
- **Characteristics**: Fast, stable, frontend-focused
|
||||
- **Use when**: Testing UI behavior without backend dependency
|
||||
|
||||
**Backend Unit Tests (`backend/tests/` directory)**:
|
||||
- API contract validation
|
||||
- Business logic variations and edge cases
|
||||
- Error handling and negative cases
|
||||
- Data transformations
|
||||
- **Characteristics**: Fast feedback, good balance, stable
|
||||
- **Characteristics**: Fastest, most stable
|
||||
- **Use when**: Testing backend logic in isolation
|
||||
|
||||
**Component**:
|
||||
- UI component behavior (buttons, forms, modals)
|
||||
- Interaction testing
|
||||
- Visual regression
|
||||
**Frontend Unit Tests (`frontend/src/components/*/tests/` directory)**:
|
||||
- Component behavior (buttons, forms, modals)
|
||||
- Interaction edge cases
|
||||
- Rendering logic
|
||||
- **Characteristics**: Fast, isolated, granular
|
||||
- **Use when**: Testing component logic in isolation
|
||||
|
||||
**Unit**:
|
||||
- Pure business logic
|
||||
- Edge cases
|
||||
- Error handling
|
||||
- **Characteristics**: Fastest, most granular
|
||||
3. **Apply BDD Principles to ALL Playwright Tests (E2E AND Frontend)**
|
||||
|
||||
3. **Avoid Duplicate Coverage**
|
||||
**CRITICAL**: ALL Playwright tests use Gherkin/BDD format, not just E2E!
|
||||
|
||||
Don't test same behavior at multiple levels unless necessary:
|
||||
- Use E2E for critical happy path only
|
||||
- Use API tests for complex business logic variations
|
||||
- Use component tests for UI interaction edge cases
|
||||
- Use unit tests for pure logic edge cases
|
||||
**Knowledge Base Reference**: `bdd-standards.md`
|
||||
|
||||
4. **Prioritize Tests**
|
||||
For ALL Playwright tests (both root `tests/` and `frontend/tests/`):
|
||||
- **MUST use Gherkin format**: Given/When/Then syntax
|
||||
- Follow ONE When per scenario rule
|
||||
- Use declarative language (WHAT user achieves, not HOW UI implements)
|
||||
- Use specific entity names when 2+ entities exist
|
||||
- Separate entry points from sub-flows to prevent combinatorial explosion
|
||||
- **Priority Model**: P0 (smoke) / P1 (entry point) / P2 (sub-flow)
|
||||
|
||||
If test-design document exists, align with priority levels:
|
||||
- P0 scenarios → Must cover in failing tests
|
||||
- P1 scenarios → Should cover if time permits
|
||||
- P2/P3 scenarios → Optional for this iteration
|
||||
**The ONLY difference between root tests/ and frontend/tests/**:
|
||||
- Root `tests/`: Tests against real backend (full stack integration)
|
||||
- `frontend/tests/`: Tests with mocked backend (frontend integration only)
|
||||
- **Both use identical Gherkin/BDD syntax!**
|
||||
|
||||
**Decision Point:** Set `primary_level` variable to main test level for this story (typically E2E or API)
|
||||
4. **Avoid Duplicate Coverage (CRITICAL)**
|
||||
|
||||
**DO NOT test the same behavior at multiple levels:**
|
||||
- ✅ **CORRECT**: E2E test for critical happy path, unit tests for variations
|
||||
- ❌ **WRONG**: E2E test + API test + component test for same behavior
|
||||
|
||||
**Example - Login Feature:**
|
||||
- **E2E BDD (1 test)**: User logs in with valid credentials → sees dashboard
|
||||
- **Backend Unit (5 tests)**: Invalid password, expired token, account locked, rate limiting, 2FA flow
|
||||
- **Frontend Unit (3 tests)**: Form validation, password visibility toggle, error message display
|
||||
|
||||
5. **Categorize and Count**
|
||||
|
||||
**Output test strategy for this story:**
|
||||
|
||||
**E2E BDD Tests (root `tests/`) - Playwright with Gherkin**: {{e2e_count}} scenarios
|
||||
{{for each E2E test identified}}
|
||||
- BDD-E2E-{{X}}: {{description}}
|
||||
- Priority: P0/P1/P2
|
||||
- Format: Gherkin/BDD (Given/When/Then)
|
||||
- Justification: {{why needs full stack integration}}
|
||||
{{end}}
|
||||
|
||||
**Frontend BDD Tests (`frontend/tests/`) - Playwright with Gherkin**: {{playwright_count}} scenarios
|
||||
{{for each frontend integration test}}
|
||||
- FE-BDD-{{Y}}: {{description}}
|
||||
{{end}}
|
||||
|
||||
**Backend Unit Tests**: {{backend_count}} scenarios
|
||||
{{for each backend test}}
|
||||
- BE-Unit-{{Z}}: {{description}}
|
||||
{{end}}
|
||||
|
||||
**Frontend Unit Tests**: {{frontend_count}} scenarios
|
||||
{{for each frontend component test}}
|
||||
- FE-Unit-{{W}}: {{description}}
|
||||
{{end}}
|
||||
|
||||
**VALIDATION CHECK:**
|
||||
- [ ] E2E count is reasonable (not testing every variation at E2E level)
|
||||
- [ ] Variations and edge cases are unit tests
|
||||
- [ ] Each test is at the LOWEST appropriate level (pyramid principle)
|
||||
- [ ] ALL Playwright tests use Gherkin/BDD format (Given/When/Then)
|
||||
- [ ] E2E and Frontend BDD tests follow BDD principles (ONE When, declarative, specific entities)
|
||||
|
||||
**Decision Point:** Set `primary_level` variable and test distribution for this story
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -451,37 +451,80 @@ The workflow auto-detects which mode to use based on project phase.
|
|||
- Scenarios are repeatable
|
||||
- Scenarios tie back to risk mitigations
|
||||
|
||||
2. **Select Appropriate Test Levels**
|
||||
2. **Select Appropriate Test Levels (PYRAMID/TROPHY STRATEGY)**
|
||||
|
||||
**Knowledge Base Reference**: `test-levels-framework.md`
|
||||
**Knowledge Base References**:
|
||||
- `test-levels-framework.md`
|
||||
- `bdd-standards.md`
|
||||
|
||||
Map requirements to optimal test levels (avoid duplication):
|
||||
**CRITICAL PRINCIPLE**: E2E tests are EXPENSIVE - use sparingly only for true integration!
|
||||
|
||||
**E2E (End-to-End)**:
|
||||
- Critical user journeys
|
||||
- Multi-system integration
|
||||
- Production-like environment
|
||||
- Highest confidence, slowest execution
|
||||
**Apply this decision tree for EACH test scenario:**
|
||||
|
||||
**API (Integration)**:
|
||||
- Service contracts
|
||||
- Business logic validation
|
||||
- Fast feedback
|
||||
- Good for complex scenarios
|
||||
**QUESTION 1: Is this user-facing (tests behavior from user perspective)?**
|
||||
- NO → Unit test (technical behavior, internal logic)
|
||||
- YES → Continue to Question 2
|
||||
|
||||
**Component**:
|
||||
- UI component behavior
|
||||
- Interaction testing
|
||||
- Visual regression
|
||||
- Fast, isolated
|
||||
**QUESTION 2: Does this REQUIRE frontend + backend integration?**
|
||||
- NO → Unit test in frontend OR backend (can be tested in isolation)
|
||||
- YES → Continue to Question 3
|
||||
|
||||
**Unit**:
|
||||
- Business logic
|
||||
- Edge cases
|
||||
- Error handling
|
||||
- Fastest, most granular
|
||||
**QUESTION 3: Is this a CRITICAL acceptance test (ship-to-production gate)?**
|
||||
- NO → Playwright test in `frontend/tests/` (frontend integration only)
|
||||
- YES → E2E BDD test in root `tests/` directory
|
||||
|
||||
**Avoid duplicate coverage**: Don't test same behavior at multiple levels unless necessary.
|
||||
**STRATEGIC TEST PLACEMENT:**
|
||||
|
||||
**E2E BDD Tests (root `tests/` directory)**:
|
||||
- **ONLY** for true backend + frontend integration
|
||||
- **ONLY** for critical acceptance criteria (if these pass, ship to production)
|
||||
- **REPLACES manual testing** - these are the production gates
|
||||
- User-facing happy paths that cross the full stack
|
||||
- **Characteristics**: Highest confidence, slowest, most brittle
|
||||
- **Target**: 5-10 smoke tests per epic maximum
|
||||
- **BDD Requirements**: Follow ONE When rule, declarative language, specific entities
|
||||
|
||||
**Frontend BDD Tests (`frontend/tests/` directory) - Playwright with Gherkin**:
|
||||
- Frontend component integration (multiple components working together)
|
||||
- UI workflows that don't need real backend (can use mocked API)
|
||||
- Visual validation and interaction testing
|
||||
- **Format**: Playwright with Gherkin/BDD (Given/When/Then)
|
||||
- **Characteristics**: Fast, stable, frontend-focused
|
||||
- **Use when**: Testing UI behavior without backend dependency
|
||||
|
||||
**Backend Unit Tests (`backend/tests/` directory)**:
|
||||
- API contract validation
|
||||
- Business logic variations and edge cases
|
||||
- Error handling and negative cases
|
||||
- Data transformations
|
||||
- **Characteristics**: Fastest, most stable
|
||||
- **Use when**: Testing backend logic in isolation
|
||||
|
||||
**Frontend Unit Tests (`frontend/src/components/*/tests/` directory)**:
|
||||
- Component behavior (buttons, forms, modals)
|
||||
- Interaction edge cases
|
||||
- Rendering logic
|
||||
- **Characteristics**: Fast, isolated, granular
|
||||
- **Use when**: Testing component logic in isolation
|
||||
|
||||
**CRITICAL: ALL PLAYWRIGHT TESTS USE GHERKIN/BDD FORMAT**
|
||||
- Both root `tests/` (E2E) and `frontend/tests/` use Gherkin syntax
|
||||
- The ONLY difference: E2E uses real backend, frontend uses mocked backend
|
||||
- Same BDD principles apply to both: ONE When, declarative, specific entities
|
||||
|
||||
**AVOID DUPLICATE COVERAGE (CRITICAL)**:
|
||||
- ✅ **CORRECT**: E2E for critical happy path, unit tests for variations
|
||||
- ❌ **WRONG**: E2E + API + component tests for same behavior
|
||||
|
||||
**Example - User Registration:**
|
||||
- **E2E BDD (1 test)**: User registers with valid email → receives confirmation → can log in
|
||||
- **Backend Unit (8 tests)**: Duplicate email, invalid format, weak password, rate limiting, email service down, verification token expired, database errors, concurrent registrations
|
||||
- **Frontend Unit (5 tests)**: Form validation, password strength indicator, email format checking, submit button states, error message display
|
||||
|
||||
**Test Distribution Target (Pyramid/Trophy)**:
|
||||
- **E2E BDD** (root tests/): 5-10 tests per epic (critical paths only)
|
||||
- **Frontend BDD** (frontend/tests/): 15-25% of total tests (both use Playwright+Gherkin)
|
||||
- **Unit Tests** (backend + frontend): 70-80% of total tests
|
||||
|
||||
3. **Assign Priority Levels**
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue