diff --git a/docs/explanation/features/tea-overview.md b/docs/explanation/features/tea-overview.md index 3fb95ab5..968c8de4 100644 --- a/docs/explanation/features/tea-overview.md +++ b/docs/explanation/features/tea-overview.md @@ -23,11 +23,16 @@ BMad does not mandate TEA. There are five valid ways to use it (or skip it). Pic 1. **No TEA** - Skip all TEA workflows. Use your existing team testing approach. -2. **TEA-only (Standalone)** +2. **TEA Solo (Standalone)** - Use TEA on a non-BMad project. Bring your own requirements, acceptance criteria, and environments. - Typical sequence: `*test-design` (system or epic) -> `*atdd` and/or `*automate` -> optional `*test-review` -> `*trace` for coverage and gate decisions. - Run `*framework` or `*ci` only if you want TEA to scaffold the harness or pipeline; they work best after you decide the stack/architecture. +**TEA Lite (Beginner Approach):** + - Simplest way to use TEA - just use `*automate` to test existing features. + - Perfect for learning TEA fundamentals in 30 minutes. + - See [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md). + 3. **Integrated: Greenfield - BMad Method (Simple/Standard Work)** - Phase 3: system-level `*test-design`, then `*framework` and `*ci`. - Phase 4: per-epic `*test-design`, optional `*atdd`, then `*automate` and optional `*test-review`. @@ -279,6 +284,31 @@ These cheat sheets map TEA workflows to the **BMad Method and Enterprise tracks* **Related how-to guides:** - [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - [How to Set Up a Test Framework](/docs/how-to/workflows/setup-test-framework.md) +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) +- [How to Set Up CI Pipeline](/docs/how-to/workflows/setup-ci.md) +- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) + +## Deep Dive Concepts + +Want to understand TEA principles and patterns in depth? + +**Core Principles:** +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Probability × impact scoring, P0-P3 priorities +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Definition of Done, determinism, isolation +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Context engineering with tea-index.csv + +**Technical Patterns:** +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Pure function → fixture → composition +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Eliminating flakiness with intercept-before-navigate + +**Engagement & Strategy:** +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - TEA Lite, TEA Solo, TEA Integrated (5 models explained) + +**Philosophy:** +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why TEA exists, problem statement ## Optional Integrations diff --git a/docs/explanation/tea/engagement-models.md b/docs/explanation/tea/engagement-models.md new file mode 100644 index 00000000..57c03c48 --- /dev/null +++ b/docs/explanation/tea/engagement-models.md @@ -0,0 +1,710 @@ +--- +title: "TEA Engagement Models Explained" +description: Understanding the five ways to use TEA - from standalone to full BMad Method integration +--- + +# TEA Engagement Models Explained + +TEA is optional and flexible. There are five valid ways to engage with TEA - choose intentionally based on your project needs and methodology. + +## Overview + +**TEA is not mandatory.** Pick the engagement model that fits your context: + +1. **No TEA** - Skip all TEA workflows, use existing testing approach +2. **TEA Solo** - Use TEA standalone without BMad Method +3. **TEA Lite** - Beginner approach using just `*automate` +4. **TEA Integrated (Greenfield)** - Full BMad Method integration from scratch +5. **TEA Integrated (Brownfield)** - Full BMad Method integration with existing code + +## The Problem + +### One-Size-Fits-All Doesn't Work + +**Traditional testing tools force one approach:** +- Must use entire framework +- All-or-nothing adoption +- No flexibility for different project types +- Teams abandon tool if it doesn't fit + +**TEA recognizes:** +- Different projects have different needs +- Different teams have different maturity levels +- Different contexts require different approaches +- Flexibility increases adoption + +## The Five Engagement Models + +### Model 1: No TEA + +**What:** Skip all TEA workflows, use your existing testing approach. + +**When to Use:** +- Team has established testing practices +- Quality is already high +- Testing tools already in place +- TEA doesn't add value + +**What You Miss:** +- Risk-based test planning +- Systematic quality review +- Gate decisions with evidence +- Knowledge base patterns + +**What You Keep:** +- Full control +- Existing tools +- Team expertise +- No learning curve + +**Example:** +``` +Your team: +- 10-year veteran QA team +- Established testing practices +- High-quality test suite +- No problems to solve + +Decision: Skip TEA, keep what works +``` + +**Verdict:** Valid choice if existing approach works. + +--- + +### Model 2: TEA Solo + +**What:** Use TEA workflows standalone without full BMad Method integration. + +**When to Use:** +- Non-BMad projects +- Want TEA's quality operating model only +- Don't need full planning workflow +- Bring your own requirements + +**Typical Sequence:** +``` +1. *test-design (system or epic) +2. *atdd or *automate +3. *test-review (optional) +4. *trace (coverage + gate decision) +``` + +**You Bring:** +- Requirements (user stories, acceptance criteria) +- Development environment +- Project context + +**TEA Provides:** +- Risk-based test planning (`*test-design`) +- Test generation (`*atdd`, `*automate`) +- Quality review (`*test-review`) +- Coverage traceability (`*trace`) + +**Optional:** +- Framework setup (`*framework`) if needed +- CI configuration (`*ci`) if needed + +**Example:** +``` +Your project: +- Using Scrum (not BMad Method) +- Jira for story management +- Need better test strategy + +Workflow: +1. Export stories from Jira +2. Run *test-design on epic +3. Run *atdd for each story +4. Implement features +5. Run *trace for coverage +``` + +**Verdict:** Best for teams wanting TEA benefits without BMad Method commitment. + +--- + +### Model 3: TEA Lite + +**What:** Beginner approach using just `*automate` to test existing features. + +**When to Use:** +- Learning TEA fundamentals +- Want quick results +- Testing existing application +- No time for full methodology + +**Workflow:** +``` +1. *framework (setup test infrastructure) +2. *test-design (optional, risk assessment) +3. *automate (generate tests for existing features) +4. Run tests (they pass immediately) +``` + +**Example:** +``` +Beginner developer: +- Never used TEA before +- Want to add tests to existing app +- 30 minutes available + +Steps: +1. Run *framework +2. Run *automate on TodoMVC demo +3. Tests generated and passing +4. Learn TEA basics +``` + +**What You Get:** +- Working test framework +- Passing tests for existing features +- Learning experience +- Foundation to expand + +**What You Miss:** +- TDD workflow (ATDD) +- Risk-based planning (test-design depth) +- Quality gates (trace Phase 2) +- Full TEA capabilities + +**Verdict:** Perfect entry point for beginners. + +--- + +### Model 4: TEA Integrated (Greenfield) + +**What:** Full BMad Method integration with TEA workflows across all phases. + +**When to Use:** +- New projects starting from scratch +- Using BMad Method or Enterprise track +- Want complete quality operating model +- Testing is critical to success + +**Lifecycle:** + +**Phase 2: Planning** +- PM creates PRD with NFRs +- (Optional) TEA runs `*nfr-assess` (Enterprise only) + +**Phase 3: Solutioning** +- Architect creates architecture +- TEA runs `*test-design` (system-level) → testability review +- TEA runs `*framework` → test infrastructure +- TEA runs `*ci` → CI/CD pipeline +- Architect runs `*implementation-readiness` (fed by test design) + +**Phase 4: Implementation (Per Epic)** +- SM runs `*sprint-planning` +- TEA runs `*test-design` (epic-level) → risk assessment for THIS epic +- SM creates stories +- (Optional) TEA runs `*atdd` → failing tests before dev +- DEV implements story +- TEA runs `*automate` → expand coverage +- (Optional) TEA runs `*test-review` → quality audit +- TEA runs `*trace` Phase 1 → refresh coverage + +**Release Gate:** +- (Optional) TEA runs `*test-review` → final audit +- (Optional) TEA runs `*nfr-assess` → validate NFRs +- TEA runs `*trace` Phase 2 → gate decision (PASS/CONCERNS/FAIL/WAIVED) + +**What You Get:** +- Complete quality operating model +- Systematic test planning +- Risk-based prioritization +- Evidence-based gate decisions +- Consistent patterns across epics + +**Example:** +``` +New SaaS product: +- 50 stories across 8 epics +- Security critical +- Need quality gates + +Workflow: +- Phase 2: Define NFRs in PRD +- Phase 3: Architecture → test design → framework → CI +- Phase 4: Per epic: test design → ATDD → dev → automate → review → trace +- Gate: NFR assess → trace Phase 2 → decision +``` + +**Verdict:** Most comprehensive TEA usage, best for structured teams. + +--- + +### Model 5: TEA Integrated (Brownfield) + +**What:** Full BMad Method integration with TEA for existing codebases. + +**When to Use:** +- Existing codebase with legacy tests +- Want to improve test quality incrementally +- Adding features to existing application +- Need to establish coverage baseline + +**Differences from Greenfield:** + +**Phase 0: Documentation (if needed)** +``` +- Run *document-project +- Create baseline documentation +``` + +**Phase 2: Planning** +``` +- TEA runs *trace Phase 1 → establish coverage baseline +- PM creates PRD (with existing system context) +``` + +**Phase 3: Solutioning** +``` +- Architect creates architecture (with brownfield constraints) +- TEA runs *test-design (system-level) → testability review +- TEA runs *framework (only if modernizing test infra) +- TEA runs *ci (update existing CI or create new) +``` + +**Phase 4: Implementation** +``` +- TEA runs *test-design (epic-level) → focus on REGRESSION HOTSPOTS +- Per story: ATDD → dev → automate +- TEA runs *test-review → improve legacy test quality +- TEA runs *trace Phase 1 → track coverage improvement +``` + +**Brownfield-Specific:** +- Baseline coverage BEFORE planning +- Focus on regression hotspots (bug-prone areas) +- Incremental quality improvement +- Compare coverage to baseline (trending up?) + +**Example:** +``` +Legacy e-commerce platform: +- 200 existing tests (30% passing, 70% flaky) +- Adding new checkout flow +- Want to improve quality + +Workflow: +1. Phase 2: *trace baseline → 30% coverage +2. Phase 3: *test-design → identify regression risks +3. Phase 4: Fix top 20 flaky tests + add tests for new checkout +4. Gate: *trace → 60% coverage (2x improvement) +``` + +**Verdict:** Best for incrementally improving legacy systems. + +--- + +## Decision Guide: Which Model? + +### Quick Decision Tree + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +flowchart TD + Start([Choose TEA Model]) --> BMad{Using
BMad Method?} + + BMad -->|No| NonBMad{Project Type?} + NonBMad -->|Learning| Lite[TEA Lite
Just *automate
30 min tutorial] + NonBMad -->|Serious Project| Solo[TEA Solo
Standalone workflows
Full capabilities] + + BMad -->|Yes| WantTEA{Want TEA?} + WantTEA -->|No| None[No TEA
Use existing approach
Valid choice] + WantTEA -->|Yes| ProjectType{New or
Existing?} + + ProjectType -->|New Project| Green[TEA Integrated
Greenfield
Full lifecycle] + ProjectType -->|Existing Code| Brown[TEA Integrated
Brownfield
Baseline + improve] + + Green --> Compliance{Compliance
Needs?} + Compliance -->|Yes| Enterprise[Enterprise Track
NFR + audit trails] + Compliance -->|No| Method[BMad Method Track
Standard quality] + + style Lite fill:#bbdefb,stroke:#1565c0,stroke-width:2px + style Solo fill:#c5cae9,stroke:#283593,stroke-width:2px + style None fill:#e0e0e0,stroke:#616161,stroke-width:1px + style Green fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style Brown fill:#fff9c4,stroke:#f57f17,stroke-width:2px + style Enterprise fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px + style Method fill:#e1f5fe,stroke:#01579b,stroke-width:2px +``` + +**Decision Path Examples:** +- Learning TEA → TEA Lite (blue) +- Non-BMad project → TEA Solo (purple) +- BMad + new project + compliance → Enterprise (purple) +- BMad + existing code → Brownfield (yellow) +- Don't want TEA → No TEA (gray) + +### By Project Type + +| Project Type | Recommended Model | Why | +|--------------|------------------|-----| +| **New SaaS product** | TEA Integrated (Greenfield) | Full quality operating model from day one | +| **Existing app + new feature** | TEA Integrated (Brownfield) | Improve incrementally while adding features | +| **Bug fix** | TEA Lite or No TEA | Quick flow, minimal overhead | +| **Learning project** | TEA Lite | Learn basics with immediate results | +| **Non-BMad enterprise** | TEA Solo | Quality model without full methodology | +| **High-quality existing tests** | No TEA | Keep what works | + +### By Team Maturity + +| Team Maturity | Recommended Model | Why | +|---------------|------------------|-----| +| **Beginners** | TEA Lite → TEA Solo | Learn basics, then expand | +| **Intermediate** | TEA Solo or Integrated | Depends on methodology | +| **Advanced** | TEA Integrated or No TEA | Full model or existing expertise | + +### By Compliance Needs + +| Compliance | Recommended Model | Why | +|------------|------------------|-----| +| **None** | Any model | Choose based on project needs | +| **Light** (internal audit) | TEA Solo or Integrated | Gate decisions helpful | +| **Heavy** (SOC 2, HIPAA) | TEA Integrated (Enterprise) | NFR assessment mandatory | + +## Switching Between Models + +### Can Change Models Mid-Project + +**Scenario:** Start with TEA Lite, expand to TEA Solo + +``` +Week 1: TEA Lite +- Run *framework +- Run *automate +- Learn basics + +Week 2: Expand to TEA Solo +- Add *test-design +- Use *atdd for new features +- Add *test-review + +Week 3: Continue expanding +- Add *trace for coverage +- Setup *ci +- Full TEA Solo workflow +``` + +**Benefit:** Start small, expand as comfortable. + +### Can Mix Models + +**Scenario:** TEA Integrated for main features, No TEA for bug fixes + +``` +Main features (epics): +- Use full TEA workflow +- Risk assessment, ATDD, quality gates + +Bug fixes: +- Skip TEA +- Quick Flow + manual testing +- Move fast + +Result: TEA where it adds value, skip where it doesn't +``` + +**Benefit:** Flexible, pragmatic, not dogmatic. + +## Comparison Table + +| Aspect | No TEA | TEA Lite | TEA Solo | Integrated (Green) | Integrated (Brown) | +|--------|--------|----------|----------|-------------------|-------------------| +| **BMad Required** | No | No | No | Yes | Yes | +| **Learning Curve** | None | Low | Medium | High | High | +| **Setup Time** | 0 | 30 min | 2 hours | 1 day | 2 days | +| **Workflows Used** | 0 | 2-3 | 4-6 | 8 | 8 | +| **Test Planning** | Manual | Optional | Yes | Systematic | + Regression focus | +| **Quality Gates** | No | No | Optional | Yes | Yes + baseline | +| **NFR Assessment** | No | No | No | Optional | Recommended | +| **Coverage Tracking** | Manual | No | Optional | Yes | Yes + trending | +| **Best For** | Experts | Beginners | Standalone | New projects | Legacy code | + +## Real-World Examples + +### Example 1: Startup (TEA Lite → TEA Integrated) + +**Month 1:** TEA Lite +``` +Team: 3 developers, no QA +Testing: Manual only +Decision: Start with TEA Lite + +Result: +- Run *framework (Playwright setup) +- Run *automate (20 tests generated) +- Learning TEA basics +``` + +**Month 3:** TEA Solo +``` +Team: Growing to 5 developers +Testing: Automated tests exist +Decision: Expand to TEA Solo + +Result: +- Add *test-design (risk assessment) +- Add *atdd (TDD workflow) +- Add *test-review (quality audits) +``` + +**Month 6:** TEA Integrated +``` +Team: 8 developers, 1 QA +Testing: Critical to business +Decision: Full BMad Method + TEA Integrated + +Result: +- Full lifecycle integration +- Quality gates before releases +- NFR assessment for enterprise customers +``` + +### Example 2: Enterprise (TEA Integrated - Brownfield) + +**Project:** Legacy banking application + +**Challenge:** +- 500 existing tests (50% flaky) +- Adding new features +- SOC 2 compliance required + +**Model:** TEA Integrated (Brownfield) + +**Phase 2:** +``` +- *trace baseline → 45% coverage (lots of gaps) +- Document current state +``` + +**Phase 3:** +``` +- *test-design (system) → identify regression hotspots +- *framework → modernize test infrastructure +- *ci → add selective testing +``` + +**Phase 4:** +``` +Per epic: +- *test-design → focus on regression + new features +- Fix top 10 flaky tests +- *atdd for new features +- *automate for coverage expansion +- *test-review → track quality improvement +- *trace → compare to baseline +``` + +**Result after 6 months:** +- Coverage: 45% → 85% +- Quality score: 52 → 82 +- Flakiness: 50% → 2% +- SOC 2 compliant (traceability + NFR evidence) + +### Example 3: Consultancy (TEA Solo) + +**Context:** Testing consultancy working with multiple clients + +**Challenge:** +- Different clients use different methodologies +- Need consistent testing approach +- Not always using BMad Method + +**Model:** TEA Solo (bring to any client project) + +**Workflow:** +``` +Client project 1 (Scrum): +- Import Jira stories +- Run *test-design +- Generate tests with *atdd/*automate +- Deliver quality report with *test-review + +Client project 2 (Kanban): +- Import requirements from Notion +- Same TEA workflow +- Consistent quality across clients + +Client project 3 (Ad-hoc): +- Document requirements manually +- Same TEA workflow +- Same patterns, different context +``` + +**Benefit:** Consistent testing approach regardless of client methodology. + +## Choosing Your Model + +### Start Here Questions + +**Question 1:** Are you using BMad Method? +- **No** → TEA Solo or TEA Lite or No TEA +- **Yes** → TEA Integrated or No TEA + +**Question 2:** Is this a new project? +- **Yes** → TEA Integrated (Greenfield) or TEA Lite +- **No** → TEA Integrated (Brownfield) or TEA Solo + +**Question 3:** What's your testing maturity? +- **Beginner** → TEA Lite +- **Intermediate** → TEA Solo or Integrated +- **Advanced** → TEA Integrated or No TEA (already expert) + +**Question 4:** Do you need compliance/quality gates? +- **Yes** → TEA Integrated (Enterprise) +- **No** → Any model + +**Question 5:** How much time can you invest? +- **30 minutes** → TEA Lite +- **Few hours** → TEA Solo +- **Multiple days** → TEA Integrated + +### Recommendation Matrix + +| Your Context | Recommended Model | Alternative | +|--------------|------------------|-------------| +| BMad Method + new project | TEA Integrated (Greenfield) | TEA Lite (learning) | +| BMad Method + existing code | TEA Integrated (Brownfield) | TEA Solo | +| Non-BMad + need quality | TEA Solo | TEA Lite | +| Just learning testing | TEA Lite | No TEA (learn basics first) | +| Enterprise + compliance | TEA Integrated (Enterprise) | TEA Solo | +| Established QA team | No TEA | TEA Solo (supplement) | + +## Transitioning Between Models + +### TEA Lite → TEA Solo + +**When:** Outgrow beginner approach, need more workflows. + +**Steps:** +1. Continue using `*framework` and `*automate` +2. Add `*test-design` for planning +3. Add `*atdd` for TDD workflow +4. Add `*test-review` for quality audits +5. Add `*trace` for coverage tracking + +**Timeline:** 2-4 weeks of gradual expansion + +### TEA Solo → TEA Integrated + +**When:** Adopt BMad Method, want full integration. + +**Steps:** +1. Install BMad Method (`npx bmad-method@alpha install`) +2. Run planning workflows (PRD, architecture) +3. Integrate TEA into Phase 3 (system-level test design) +4. Follow integrated lifecycle (per epic workflows) +5. Add release gates (trace Phase 2) + +**Timeline:** 1-2 sprints of transition + +### TEA Integrated → TEA Solo + +**When:** Moving away from BMad Method, keep TEA. + +**Steps:** +1. Export BMad artifacts (PRD, architecture, stories) +2. Continue using TEA workflows standalone +3. Skip BMad-specific integration +4. Bring your own requirements to TEA + +**Timeline:** Immediate (just skip BMad workflows) + +## Common Patterns + +### Pattern 1: TEA Lite for Learning, Then Choose + +``` +Phase 1 (Week 1-2): TEA Lite +- Learn with *automate on demo app +- Understand TEA fundamentals +- Low commitment + +Phase 2 (Week 3-4): Evaluate +- Try *test-design (planning) +- Try *atdd (TDD) +- See if value justifies investment + +Phase 3 (Month 2+): Decide +- Valuable → Expand to TEA Solo or Integrated +- Not valuable → Stay with TEA Lite or No TEA +``` + +### Pattern 2: TEA Solo for Quality, Skip Full Method + +``` +Team decision: +- Don't want full BMad Method (too heavyweight) +- Want systematic testing (TEA benefits) + +Approach: TEA Solo only +- Use existing project management (Jira, Linear) +- Use TEA for testing only +- Get quality without methodology commitment +``` + +### Pattern 3: Integrated for Critical, Lite for Non-Critical + +``` +Critical features (payment, auth): +- Full TEA Integrated workflow +- Risk assessment, ATDD, quality gates +- High confidence required + +Non-critical features (UI tweaks): +- TEA Lite or No TEA +- Quick tests, minimal overhead +- Move fast +``` + +## Technical Implementation + +Each model uses different TEA workflows. See: +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Model details +- [TEA Command Reference](/docs/reference/tea/commands.md) - Workflow reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - Setup options + +## Related Concepts + +**Core TEA Concepts:** +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Risk assessment in different models +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality across all models +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Consistent patterns across models + +**Technical Patterns:** +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Infrastructure in different models +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Reliability in all models + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - 5 engagement models with cheat sheets +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Design philosophy + +## Practical Guides + +**Getting Started:** +- [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md) - Model 3: TEA Lite + +**Use-Case Guides:** +- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Model 5: Brownfield +- [Running TEA for Enterprise](/docs/how-to/workflows/run-tea-for-enterprise.md) - Enterprise integration + +**All Workflow Guides:** +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Used in TEA Solo and Integrated +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - All workflows explained +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config per model +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA Lite, TEA Solo, TEA Integrated terms + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/explanation/tea/fixture-architecture.md b/docs/explanation/tea/fixture-architecture.md new file mode 100644 index 00000000..89dc17b5 --- /dev/null +++ b/docs/explanation/tea/fixture-architecture.md @@ -0,0 +1,455 @@ +--- +title: "Fixture Architecture Explained" +description: Understanding TEA's pure function → fixture → composition pattern for reusable test utilities +--- + +# Fixture Architecture Explained + +Fixture architecture is TEA's pattern for building reusable, testable, and composable test utilities. The core principle: build pure functions first, wrap in framework fixtures second. + +## Overview + +**The Pattern:** +1. Write utility as pure function (unit-testable) +2. Wrap in framework fixture (Playwright, Cypress) +3. Compose fixtures with mergeTests (combine capabilities) +4. Package for reuse across projects + +**Why this order?** +- Pure functions are easier to test +- Fixtures depend on framework (less portable) +- Composition happens at fixture level +- Reusability maximized + +#### Fixture Architecture Flow + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +flowchart TD + Start([Testing Need]) --> Pure[Step 1: Pure Function
helpers/api-request.ts] + Pure -->|Unit testable
Framework agnostic| Fixture[Step 2: Fixture Wrapper
fixtures/api-request.ts] + Fixture -->|Injects framework
dependencies| Compose[Step 3: Composition
fixtures/index.ts] + Compose -->|mergeTests| Use[Step 4: Use in Tests
tests/**.spec.ts] + + Pure -.->|Can test in isolation| UnitTest[Unit Tests
No framework needed] + Fixture -.->|Reusable pattern| Other[Other Projects
Package export] + Compose -.->|Combine utilities| Multi[Multiple Fixtures
One test] + + style Pure fill:#e3f2fd,stroke:#1565c0,stroke-width:2px + style Fixture fill:#fff3e0,stroke:#e65100,stroke-width:2px + style Compose fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px + style Use fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px + style UnitTest fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px + style Other fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px + style Multi fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px +``` + +**Benefits at Each Step:** +1. **Pure Function:** Testable, portable, reusable +2. **Fixture:** Framework integration, clean API +3. **Composition:** Combine capabilities, flexible +4. **Usage:** Simple imports, type-safe + +## The Problem + +### Framework-First Approach (Common Anti-Pattern) + +```typescript +// ❌ Bad: Built as fixture from the start +export const test = base.extend({ + apiRequest: async ({ request }, use) => { + await use(async (options) => { + const response = await request.fetch(options.url, { + method: options.method, + data: options.data + }); + + if (!response.ok()) { + throw new Error(`API request failed: ${response.status()}`); + } + + return response.json(); + }); + } +}); +``` + +**Problems:** +- Cannot unit test (requires Playwright context) +- Tied to framework (not reusable in other tools) +- Hard to compose with other fixtures +- Difficult to mock for testing the utility itself + +### Copy-Paste Utilities + +```typescript +// test-1.spec.ts +test('test 1', async ({ request }) => { + const response = await request.post('/api/users', { data: {...} }); + const body = await response.json(); + if (!response.ok()) throw new Error('Failed'); + // ... repeated in every test +}); + +// test-2.spec.ts +test('test 2', async ({ request }) => { + const response = await request.post('/api/users', { data: {...} }); + const body = await response.json(); + if (!response.ok()) throw new Error('Failed'); + // ... same code repeated +}); +``` + +**Problems:** +- Code duplication (violates DRY) +- Inconsistent error handling +- Hard to update (change 50 tests) +- No shared behavior + +## The Solution: Three-Step Pattern + +### Step 1: Pure Function + +```typescript +// helpers/api-request.ts + +/** + * Make API request with automatic error handling + * Pure function - no framework dependencies + */ +export async function apiRequest({ + request, // Passed in (dependency injection) + method, + url, + data, + headers = {} +}: ApiRequestParams): Promise { + const response = await request.fetch(url, { + method, + data, + headers + }); + + if (!response.ok()) { + throw new Error(`API request failed: ${response.status()}`); + } + + return { + status: response.status(), + body: await response.json() + }; +} + +// ✅ Can unit test this function! +describe('apiRequest', () => { + it('should throw on non-OK response', async () => { + const mockRequest = { + fetch: vi.fn().mockResolvedValue({ ok: () => false, status: () => 500 }) + }; + + await expect(apiRequest({ + request: mockRequest, + method: 'GET', + url: '/api/test' + })).rejects.toThrow('API request failed: 500'); + }); +}); +``` + +**Benefits:** +- Unit testable (mock dependencies) +- Framework-agnostic (works with any HTTP client) +- Easy to reason about (pure function) +- Portable (can use in Node scripts, CLI tools) + +### Step 2: Fixture Wrapper + +```typescript +// fixtures/api-request.ts +import { test as base } from '@playwright/test'; +import { apiRequest as apiRequestFn } from '../helpers/api-request'; + +/** + * Playwright fixture wrapping the pure function + */ +export const test = base.extend<{ apiRequest: typeof apiRequestFn }>({ + apiRequest: async ({ request }, use) => { + // Inject framework dependency (request) + await use((params) => apiRequestFn({ request, ...params })); + } +}); + +export { expect } from '@playwright/test'; +``` + +**Benefits:** +- Fixture provides framework context (request) +- Pure function handles logic +- Clean separation of concerns +- Can swap frameworks (Cypress, etc.) by changing wrapper only + +### Step 3: Composition with mergeTests + +```typescript +// fixtures/index.ts +import { mergeTests } from '@playwright/test'; +import { test as apiRequestTest } from './api-request'; +import { test as authSessionTest } from './auth-session'; +import { test as logTest } from './log'; + +/** + * Compose all fixtures into one test + */ +export const test = mergeTests( + apiRequestTest, + authSessionTest, + logTest +); + +export { expect } from '@playwright/test'; +``` + +**Usage:** +```typescript +// tests/profile.spec.ts +import { test, expect } from '../support/fixtures'; + +test('should update profile', async ({ apiRequest, authToken, log }) => { + log.info('Starting profile update test'); + + // Use API request fixture with auth token + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { name: 'New Name' }, // 'body' not 'data' + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(status).toBe(200); + expect(body.name).toBe('New Name'); + + log.info('Profile updated successfully'); +}); +``` + +**Note:** `authToken` requires auth-session fixture setup with provider configuration. See [auth-session documentation](https://seontechnologies.github.io/playwright-utils/auth-session.html). + +**Benefits:** +- Use multiple fixtures in one test +- No manual composition needed +- Type-safe (TypeScript knows all fixture types) +- Clean imports + +## How It Works in TEA + +### TEA Generates This Pattern + +When you run `*framework` with `tea_use_playwright_utils: true`: + +**TEA scaffolds:** +``` +tests/ +├── support/ +│ ├── helpers/ # Pure functions +│ │ ├── api-request.ts +│ │ └── auth-session.ts +│ └── fixtures/ # Framework wrappers +│ ├── api-request.ts +│ ├── auth-session.ts +│ └── index.ts # Composition +└── e2e/ + └── example.spec.ts # Uses composed fixtures +``` + +### TEA Reviews Against This Pattern + +When you run `*test-review`: + +**TEA checks:** +- Are utilities pure functions? ✓ +- Are fixtures minimal wrappers? ✓ +- Is composition used? ✓ +- Can utilities be unit tested? ✓ + +## Package Export Pattern + +### Make Fixtures Reusable Across Projects + +**Option 1: Build Your Own (Vanilla)** +```json +// package.json +{ + "name": "@company/test-utils", + "exports": { + "./api-request": "./fixtures/api-request.ts", + "./auth-session": "./fixtures/auth-session.ts", + "./log": "./fixtures/log.ts" + } +} +``` + +**Usage:** +```typescript +import { test as apiTest } from '@company/test-utils/api-request'; +import { test as authTest } from '@company/test-utils/auth-session'; +import { mergeTests } from '@playwright/test'; + +export const test = mergeTests(apiTest, authTest); +``` + +**Option 2: Use Playwright Utils (Recommended)** +```bash +npm install -D @seontechnologies/playwright-utils +``` + +**Usage:** +```typescript +import { test as base } from '@playwright/test'; +import { mergeTests } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; + +const authFixtureTest = base.extend(createAuthFixtures()); +export const test = mergeTests(apiRequestFixture, authFixtureTest); +// Production-ready utilities, battle-tested! +``` + +**Note:** Auth-session requires provider configuration. See [auth-session setup guide](https://seontechnologies.github.io/playwright-utils/auth-session.html). + +**Why Playwright Utils:** +- Already built, tested, and maintained +- Consistent patterns across projects +- 11 utilities available (API, auth, network, logging, files) +- Community support and documentation +- Regular updates and improvements + +**When to Build Your Own:** +- Company-specific patterns +- Custom authentication systems +- Unique requirements not covered by utilities + +## Comparison: Good vs Bad Patterns + +### Anti-Pattern: God Fixture + +```typescript +// ❌ Bad: Everything in one fixture +export const test = base.extend({ + testUtils: async ({ page, request, context }, use) => { + await use({ + // 50 different methods crammed into one fixture + apiRequest: async (...) => { }, + login: async (...) => { }, + createUser: async (...) => { }, + deleteUser: async (...) => { }, + uploadFile: async (...) => { }, + // ... 45 more methods + }); + } +}); +``` + +**Problems:** +- Cannot test individual utilities +- Cannot compose (all-or-nothing) +- Cannot reuse specific utilities +- Hard to maintain (1000+ line file) + +### Good Pattern: Single-Concern Fixtures + +```typescript +// ✅ Good: One concern per fixture + +// api-request.ts +export const test = base.extend({ apiRequest }); + +// auth-session.ts +export const test = base.extend({ authSession }); + +// log.ts +export const test = base.extend({ log }); + +// Compose as needed +import { mergeTests } from '@playwright/test'; +export const test = mergeTests(apiRequestTest, authSessionTest, logTest); +``` + +**Benefits:** +- Each fixture is unit-testable +- Compose only what you need +- Reuse individual fixtures +- Easy to maintain (small files) + +## Technical Implementation + +For detailed fixture architecture patterns, see the knowledge base: +- [Knowledge Base Index - Architecture & Fixtures](/docs/reference/tea/knowledge-base.md) +- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md) + +## When to Use This Pattern + +### Always Use For: + +**Reusable utilities:** +- API request helpers +- Authentication handlers +- File operations +- Network mocking + +**Test infrastructure:** +- Shared fixtures across teams +- Packaged utilities (playwright-utils) +- Company-wide test standards + +### Consider Skipping For: + +**One-off test setup:** +```typescript +// Simple one-time setup - inline is fine +test.beforeEach(async ({ page }) => { + await page.goto('/'); + await page.click('#accept-cookies'); +}); +``` + +**Test-specific helpers:** +```typescript +// Used in one test file only - keep local +function createTestUser(name: string) { + return { name, email: `${name}@test.com` }; +} +``` + +## Related Concepts + +**Core TEA Concepts:** +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality standards fixtures enforce +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Fixture patterns in knowledge base + +**Technical Patterns:** +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Network fixtures explained +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Fixture complexity matches risk + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Fixture architecture in workflows +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why fixtures matter + +## Practical Guides + +**Setup Guides:** +- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - TEA scaffolds fixtures +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Production-ready fixtures + +**Workflow Guides:** +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Using fixtures in tests +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Fixture composition examples + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - *framework command +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Fixture architecture fragments +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Fixture architecture term + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/explanation/tea/knowledge-base-system.md b/docs/explanation/tea/knowledge-base-system.md new file mode 100644 index 00000000..38d73290 --- /dev/null +++ b/docs/explanation/tea/knowledge-base-system.md @@ -0,0 +1,543 @@ +--- +title: "Knowledge Base System Explained" +description: Understanding how TEA uses tea-index.csv for context engineering and consistent test quality +--- + +# Knowledge Base System Explained + +TEA's knowledge base system is how context engineering works - automatically loading domain-specific standards into AI context so tests are consistently high-quality regardless of prompt variation. + +## Overview + +**The Problem:** AI without context produces inconsistent results. + +**Traditional approach:** +``` +User: "Write tests for login" +AI: [Generates tests with random quality] +- Sometimes uses hard waits +- Sometimes uses good patterns +- Inconsistent across sessions +- Quality depends on prompt +``` + +**TEA with knowledge base:** +``` +User: "Write tests for login" +TEA: [Loads test-quality.md, network-first.md, auth-session.md] +TEA: [Generates tests following established patterns] +- Always uses network-first patterns +- Always uses proper fixtures +- Consistent across all sessions +- Quality independent of prompt +``` + +**Result:** Systematic quality, not random chance. + +## The Problem + +### Prompt-Driven Testing = Inconsistency + +**Session 1:** +``` +User: "Write tests for profile editing" + +AI: [No context loaded] +// Generates test with hard waits +await page.waitForTimeout(3000); +``` + +**Session 2:** +``` +User: "Write comprehensive tests for profile editing with best practices" + +AI: [Still no systematic context] +// Generates test with some improvements, but still issues +await page.waitForSelector('.success', { timeout: 10000 }); +``` + +**Session 3:** +``` +User: "Write tests using network-first patterns and proper fixtures" + +AI: [Better prompt, but still reinventing patterns] +// Generates test with network-first, but inconsistent with other tests +``` + +**Problem:** Quality depends on prompt engineering skill, no consistency. + +### Knowledge Drift + +Without a knowledge base: +- Team A uses pattern X +- Team B uses pattern Y +- Both work, but inconsistent +- No single source of truth +- Patterns drift over time + +## The Solution: tea-index.csv Manifest + +### How It Works + +**1. Manifest Defines Fragments** + +`src/modules/bmm/testarch/tea-index.csv`: +```csv +id,name,description,tags,fragment_file +test-quality,Test Quality,Execution limits and isolation rules,quality;standards,knowledge/test-quality.md +network-first,Network-First Safeguards,Intercept-before-navigate workflow,network;stability,knowledge/network-first.md +fixture-architecture,Fixture Architecture,Composable fixture patterns,fixtures;architecture,knowledge/fixture-architecture.md +``` + +**2. Workflow Loads Relevant Fragments** + +When user runs `*atdd`: +``` +TEA reads tea-index.csv +Identifies fragments needed for ATDD: +- test-quality.md (quality standards) +- network-first.md (avoid flakiness) +- component-tdd.md (TDD patterns) +- fixture-architecture.md (reusable fixtures) +- data-factories.md (test data) + +Loads only these 5 fragments (not all 33) +Generates tests following these patterns +``` + +**3. Consistent Output** + +Every time `*atdd` runs: +- Same fragments loaded +- Same patterns applied +- Same quality standards +- Consistent test structure + +**Result:** Tests look like they were written by the same expert, every time. + +### Knowledge Base Loading Diagram + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +flowchart TD + User([User: *atdd]) --> Workflow[TEA Workflow
Triggered] + Workflow --> Read[Read Manifest
tea-index.csv] + + Read --> Identify{Identify Relevant
Fragments for ATDD} + + Identify -->|Needed| L1[✓ test-quality.md] + Identify -->|Needed| L2[✓ network-first.md] + Identify -->|Needed| L3[✓ component-tdd.md] + Identify -->|Needed| L4[✓ data-factories.md] + Identify -->|Needed| L5[✓ fixture-architecture.md] + + Identify -.->|Skip| S1[✗ contract-testing.md] + Identify -.->|Skip| S2[✗ burn-in.md] + Identify -.->|Skip| S3[+ 26 other fragments] + + L1 --> Context[AI Context
5 fragments loaded] + L2 --> Context + L3 --> Context + L4 --> Context + L5 --> Context + + Context --> Gen[Generate Tests
Following patterns] + Gen --> Out([Consistent Output
Same quality every time]) + + style User fill:#e3f2fd,stroke:#1565c0,stroke-width:2px + style Read fill:#fff3e0,stroke:#e65100,stroke-width:2px + style L1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style L2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style L3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style L4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style L5 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px + style S1 fill:#e0e0e0,stroke:#616161,stroke-width:1px + style S2 fill:#e0e0e0,stroke:#616161,stroke-width:1px + style S3 fill:#e0e0e0,stroke:#616161,stroke-width:1px + style Context fill:#f3e5f5,stroke:#6a1b9a,stroke-width:3px + style Out fill:#4caf50,stroke:#1b5e20,stroke-width:3px,color:#fff +``` + +## Fragment Structure + +### Anatomy of a Fragment + +Each fragment follows this structure: + +```markdown +# Fragment Name + +## Principle +[One sentence - what is this pattern?] + +## Rationale +[Why use this instead of alternatives?] +Why this pattern exists +Problems it solves +Benefits it provides + +## Pattern Examples + +### Example 1: Basic Usage +```code +[Runnable code example] +``` +[Explanation of example] + +### Example 2: Advanced Pattern +```code +[More complex example] +``` +[Explanation] + +## Anti-Patterns + +### Don't Do This +```code +[Bad code example] +``` +[Why it's bad] +[What breaks] + +## Related Patterns +- [Link to related fragment] +``` + +### Example: test-quality.md Fragment + +```markdown +# Test Quality + +## Principle +Tests must be deterministic, isolated, explicit, focused, and fast. + +## Rationale +Tests that fail randomly, depend on each other, or take too long lose team trust. +[... detailed explanation ...] + +## Pattern Examples + +### Example 1: Deterministic Test +```typescript +// ✅ Wait for actual response, not timeout +const promise = page.waitForResponse(matcher); +await page.click('button'); +await promise; +``` + +### Example 2: Isolated Test +```typescript +// ✅ Self-cleaning test +test('test', async ({ page }) => { + const userId = await createTestUser(); + // ... test logic ... + await deleteTestUser(userId); // Cleanup +}); +``` + +## Anti-Patterns + +### Hard Waits +```typescript +// ❌ Non-deterministic +await page.waitForTimeout(3000); +``` +[Why this causes flakiness] +``` + +**Total:** 24.5 KB, 12 code examples + +## How TEA Uses the Knowledge Base + +### Workflow-Specific Loading + +**Different workflows load different fragments:** + +| Workflow | Fragments Loaded | Purpose | +|----------|-----------------|---------| +| `*framework` | fixture-architecture, playwright-config, fixtures-composition | Infrastructure patterns | +| `*test-design` | test-quality, test-priorities-matrix, risk-governance | Planning standards | +| `*atdd` | test-quality, component-tdd, network-first, data-factories | TDD patterns | +| `*automate` | test-quality, test-levels-framework, selector-resilience | Comprehensive generation | +| `*test-review` | All quality/resilience/debugging fragments | Full audit patterns | +| `*ci` | ci-burn-in, burn-in, selective-testing | CI/CD optimization | + +**Benefit:** Only load what's needed (focused context, no bloat). + +### Dynamic Fragment Selection + +TEA doesn't load all 33 fragments at once: + +``` +User runs: *atdd for authentication feature + +TEA analyzes context: +- Feature type: Authentication +- Relevant fragments: + - test-quality.md (always loaded) + - auth-session.md (auth patterns) + - network-first.md (avoid flakiness) + - email-auth.md (if email-based auth) + - data-factories.md (test users) + +Skips: +- contract-testing.md (not relevant) +- feature-flags.md (not relevant) +- file-utils.md (not relevant) + +Result: 5 relevant fragments loaded, 28 skipped +``` + +**Benefit:** Focused context = better results, lower token usage. + +## Context Engineering in Practice + +### Example: Consistent Test Generation + +**Without Knowledge Base (Vanilla Playwright, Random Quality):** +``` +Session 1: User runs *atdd +AI: [Guesses patterns from general knowledge] + +Generated: +test('api test', async ({ request }) => { + const response = await request.get('/api/users'); + await page.waitForTimeout(2000); // Hard wait + const users = await response.json(); + // Random quality +}); + +Session 2: User runs *atdd (different day) +AI: [Different random patterns] + +Generated: +test('api test', async ({ request }) => { + const response = await request.get('/api/users'); + const users = await response.json(); + // Better but inconsistent +}); + +Result: Inconsistent quality, random patterns +``` + +**With Knowledge Base (TEA + Playwright Utils):** +``` +Session 1: User runs *atdd +TEA: [Loads test-quality.md, network-first.md, api-request.md from tea-index.csv] + +Generated: +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +test('should fetch users', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users' + }).validateSchema(UsersSchema); // Chained validation + + expect(status).toBe(200); + expect(body).toBeInstanceOf(Array); +}); + +Session 2: User runs *atdd (different day) +TEA: [Loads same fragments from tea-index.csv] + +Generated: Identical pattern, same quality + +Result: Systematic quality, established patterns (ALWAYS uses apiRequest utility when playwright-utils enabled) +``` + +**Key Difference:** +- **Without KB:** Random patterns, inconsistent APIs +- **With KB:** Always uses `apiRequest` utility, always validates schemas, always returns `{ status, body }` + +### Example: Test Review Consistency + +**Without Knowledge Base:** +``` +*test-review session 1: +"This test looks okay" [50 issues missed] + +*test-review session 2: +"This test has some issues" [Different issues flagged] + +Result: Inconsistent feedback +``` + +**With Knowledge Base:** +``` +*test-review session 1: +[Loads all quality fragments] +Flags: 12 hard waits, 5 conditionals (based on test-quality.md) + +*test-review session 2: +[Loads same fragments] +Flags: Same issues with same explanations + +Result: Consistent, reliable feedback +``` + +## Maintaining the Knowledge Base + +### When to Add a Fragment + +**Good reasons:** +- Pattern is used across multiple workflows +- Standard is non-obvious (needs documentation) +- Team asks "how should we handle X?" repeatedly +- New tool integration (e.g., new testing library) + +**Bad reasons:** +- One-off pattern (document in test file instead) +- Obvious pattern (everyone knows this) +- Experimental (not proven yet) + +### Fragment Quality Standards + +**Good fragment:** +- Principle stated in one sentence +- Rationale explains why clearly +- 3+ pattern examples with code +- Anti-patterns shown (what not to do) +- Self-contained (minimal dependencies) + +**Example size:** 10-30 KB optimal + +### Updating Existing Fragments + +**When to update:** +- Pattern evolved (better approach discovered) +- Tool updated (new Playwright API) +- Team feedback (pattern unclear) +- Bug in example code + +**How to update:** +1. Edit fragment markdown file +2. Update examples +3. Test with affected workflows +4. Ensure no breaking changes + +**No need to update tea-index.csv** unless description/tags change. + +## Benefits of Knowledge Base System + +### 1. Consistency + +**Before:** Test quality varies by who wrote it +**After:** All tests follow same patterns (TEA-generated or reviewed) + +### 2. Onboarding + +**Before:** New team member reads 20 documents, asks 50 questions +**After:** New team member runs `*atdd`, sees patterns in generated code, learns by example + +### 3. Quality Gates + +**Before:** "Is this test good?" → subjective opinion +**After:** "*test-review" → objective score against knowledge base + +### 4. Pattern Evolution + +**Before:** Update tests manually across 100 files +**After:** Update fragment once, all new tests use new pattern + +### 5. Cross-Project Reuse + +**Before:** Reinvent patterns for each project +**After:** Same fragments across all BMad projects (consistency at scale) + +## Comparison: With vs Without Knowledge Base + +### Scenario: Testing Async Background Job + +**Without Knowledge Base:** + +Developer 1: +```typescript +// Uses hard wait +await page.click('button'); +await page.waitForTimeout(10000); // Hope job finishes +``` + +Developer 2: +```typescript +// Uses polling +await page.click('button'); +for (let i = 0; i < 10; i++) { + const status = await page.locator('.status').textContent(); + if (status === 'complete') break; + await page.waitForTimeout(1000); +} +``` + +Developer 3: +```typescript +// Uses waitForSelector +await page.click('button'); +await page.waitForSelector('.success', { timeout: 30000 }); +``` + +**Result:** 3 different patterns, all suboptimal. + +**With Knowledge Base (recurse.md fragment):** + +All developers: +```typescript +import { test } from '@seontechnologies/playwright-utils/recurse/fixtures'; + +test('job completion', async ({ page, recurse }) => { + await page.click('button'); + + const result = await recurse({ + fn: () => apiRequest({ method: 'GET', path: '/api/job/123' }), + predicate: (job) => job.status === 'complete', + timeout: 30000 + }); + + expect(result.status).toBe('complete'); +}); +``` + +**Result:** Consistent pattern, established best practice. + +## Technical Implementation + +For details on the knowledge base index, see: +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) +- [TEA Configuration](/docs/reference/tea/configuration.md) + +## Related Concepts + +**Core TEA Concepts:** +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Standards in knowledge base +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Risk patterns in knowledge base +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Knowledge base across all models + +**Technical Patterns:** +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Fixture patterns in knowledge base +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Network patterns in knowledge base + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Knowledge base in workflows +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Context engineering philosophy + +## Practical Guides + +**All Workflow Guides Use Knowledge Base:** +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) + +**Integration:** +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - PW-Utils in knowledge base + +## Reference + +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Complete fragment index +- [TEA Command Reference](/docs/reference/tea/commands.md) - Which workflows load which fragments +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config affects fragment loading +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Context engineering, knowledge fragment terms + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/explanation/tea/network-first-patterns.md b/docs/explanation/tea/network-first-patterns.md new file mode 100644 index 00000000..93b45a10 --- /dev/null +++ b/docs/explanation/tea/network-first-patterns.md @@ -0,0 +1,603 @@ +--- +title: "Network-First Patterns Explained" +description: Understanding how TEA eliminates test flakiness by waiting for actual network responses +--- + +# Network-First Patterns Explained + +Network-first patterns are TEA's solution to test flakiness. Instead of guessing how long to wait with fixed timeouts, wait for the actual network event that causes UI changes. + +## Overview + +**The Core Principle:** +UI changes because APIs respond. Wait for the API response, not an arbitrary timeout. + +**Traditional approach:** +```typescript +await page.click('button'); +await page.waitForTimeout(3000); // Hope 3 seconds is enough +await expect(page.locator('.success')).toBeVisible(); +``` + +**Network-first approach:** +```typescript +const responsePromise = page.waitForResponse( + resp => resp.url().includes('/api/submit') && resp.ok() +); +await page.click('button'); +await responsePromise; // Wait for actual response +await expect(page.locator('.success')).toBeVisible(); +``` + +**Result:** Deterministic tests that wait exactly as long as needed. + +## The Problem + +### Hard Waits Create Flakiness + +```typescript +// ❌ The flaky test pattern +test('should submit form', async ({ page }) => { + await page.fill('#name', 'Test User'); + await page.click('button[type="submit"]'); + + await page.waitForTimeout(2000); // Wait 2 seconds + + await expect(page.locator('.success')).toBeVisible(); +}); +``` + +**Why this fails:** +- **Fast network:** Wastes 1.5 seconds waiting +- **Slow network:** Not enough time, test fails +- **CI environment:** Slower than local, fails randomly +- **Under load:** API takes 3 seconds, test fails + +**Result:** "Works on my machine" syndrome, flaky CI. + +### The Timeout Escalation Trap + +```typescript +// Developer sees flaky test +await page.waitForTimeout(2000); // Failed in CI + +// Increases timeout +await page.waitForTimeout(5000); // Still fails sometimes + +// Increases again +await page.waitForTimeout(10000); // Now it passes... slowly + +// Problem: Now EVERY test waits 10 seconds +// Suite that took 5 minutes now takes 30 minutes +``` + +**Result:** Slow, still-flaky tests. + +### Race Conditions + +```typescript +// ❌ Navigate-then-wait race condition +test('should load dashboard data', async ({ page }) => { + await page.goto('/dashboard'); // Navigation starts + + // Race condition! API might not have responded yet + await expect(page.locator('.data-table')).toBeVisible(); +}); +``` + +**What happens:** +1. `goto()` starts navigation +2. Page loads HTML +3. JavaScript requests `/api/dashboard` +4. Test checks for `.data-table` BEFORE API responds +5. Test fails intermittently + +**Result:** "Sometimes it works, sometimes it doesn't." + +## The Solution: Intercept-Before-Navigate + +### Wait for Response Before Asserting + +```typescript +// ✅ Good: Network-first pattern +test('should load dashboard data', async ({ page }) => { + // Set up promise BEFORE navigation + const dashboardPromise = page.waitForResponse( + resp => resp.url().includes('/api/dashboard') && resp.ok() + ); + + // Navigate + await page.goto('/dashboard'); + + // Wait for API response + const response = await dashboardPromise; + const data = await response.json(); + + // Now assert UI + await expect(page.locator('.data-table')).toBeVisible(); + await expect(page.locator('.data-table tr')).toHaveCount(data.items.length); +}); +``` + +**Why this works:** +- Wait set up BEFORE navigation (no race) +- Wait for actual API response (deterministic) +- No fixed timeout (fast when API is fast) +- Validates API response (catch backend errors) + +### Intercept-Before-Navigate Pattern + +**Key insight:** Set up wait BEFORE triggering the action. + +```typescript +// ✅ Pattern: Intercept → Action → Await + +// 1. Intercept (set up wait) +const promise = page.waitForResponse(matcher); + +// 2. Action (trigger request) +await page.click('button'); + +// 3. Await (wait for actual response) +await promise; +``` + +**Why this order:** +- `waitForResponse()` starts listening immediately +- Then trigger the action that makes the request +- Then wait for the promise to resolve +- No race condition possible + +#### Intercept-Before-Navigate Flow + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +sequenceDiagram + participant Test + participant Playwright + participant Browser + participant API + + rect rgb(200, 230, 201) + Note over Test,Playwright: ✅ CORRECT: Intercept First + Test->>Playwright: 1. waitForResponse(matcher) + Note over Playwright: Starts listening for response + Test->>Browser: 2. click('button') + Browser->>API: 3. POST /api/submit + API-->>Browser: 4. 200 OK {success: true} + Browser-->>Playwright: 5. Response captured + Test->>Playwright: 6. await promise + Playwright-->>Test: 7. Returns response + Note over Test: No race condition! + end + + rect rgb(255, 205, 210) + Note over Test,API: ❌ WRONG: Action First + Test->>Browser: 1. click('button') + Browser->>API: 2. POST /api/submit + API-->>Browser: 3. 200 OK (already happened!) + Test->>Playwright: 4. waitForResponse(matcher) + Note over Test,Playwright: Too late - response already occurred + Note over Test: Race condition! Test hangs or fails + end +``` + +**Correct Order (Green):** +1. Set up listener (`waitForResponse`) +2. Trigger action (`click`) +3. Wait for response (`await promise`) + +**Wrong Order (Red):** +1. Trigger action first +2. Set up listener too late +3. Response already happened - missed! + +## How It Works in TEA + +### TEA Generates Network-First Tests + +```typescript +// When you run *atdd or *automate, TEA generates: + +test('should create user', async ({ page }) => { + // TEA automatically includes network wait + const createUserPromise = page.waitForResponse( + resp => resp.url().includes('/api/users') && + resp.request().method() === 'POST' && + resp.ok() + ); + + await page.fill('#name', 'Test User'); + await page.click('button[type="submit"]'); + + const response = await createUserPromise; + const user = await response.json(); + + // Validate both API and UI + expect(user.id).toBeDefined(); + await expect(page.locator('.success')).toContainText(user.name); +}); +``` + +### TEA Reviews for Hard Waits + +When you run `*test-review`: + +```markdown +## Critical Issue: Hard Wait Detected + +**File:** tests/e2e/submit.spec.ts:45 +**Issue:** Using `page.waitForTimeout(3000)` +**Severity:** Critical (causes flakiness) + +**Current Code:** +```typescript +await page.click('button'); +await page.waitForTimeout(3000); // ❌ +``` + +**Fix:** +```typescript +const responsePromise = page.waitForResponse( + resp => resp.url().includes('/api/submit') && resp.ok() +); +await page.click('button'); +await responsePromise; // ✅ +``` + +**Why:** Hard waits are non-deterministic. Use network-first patterns. +``` + +## Pattern Variations + +### Basic Response Wait + +```typescript +// Wait for any successful response +const promise = page.waitForResponse(resp => resp.ok()); +await page.click('button'); +await promise; +``` + +### Specific URL Match + +```typescript +// Wait for specific endpoint +const promise = page.waitForResponse( + resp => resp.url().includes('/api/users/123') +); +await page.goto('/user/123'); +await promise; +``` + +### Method + Status Match + +```typescript +// Wait for POST that returns 201 +const promise = page.waitForResponse( + resp => + resp.url().includes('/api/users') && + resp.request().method() === 'POST' && + resp.status() === 201 +); +await page.click('button[type="submit"]'); +await promise; +``` + +### Multiple Responses + +```typescript +// Wait for multiple API calls +const [usersResp, postsResp] = await Promise.all([ + page.waitForResponse(resp => resp.url().includes('/api/users')), + page.waitForResponse(resp => resp.url().includes('/api/posts')), + page.goto('/dashboard') // Triggers both requests +]); + +const users = await usersResp.json(); +const posts = await postsResp.json(); +``` + +### Validate Response Data + +```typescript +// Verify API response before asserting UI +const promise = page.waitForResponse( + resp => resp.url().includes('/api/checkout') && resp.ok() +); + +await page.click('button:has-text("Complete Order")'); + +const response = await promise; +const order = await response.json(); + +// Response validation +expect(order.status).toBe('confirmed'); +expect(order.total).toBeGreaterThan(0); + +// UI validation +await expect(page.locator('.order-confirmation')).toContainText(order.id); +``` + +## Advanced Patterns + +### HAR Recording for Offline Testing + +**Vanilla Playwright (Manual HAR Handling):** +```typescript +test('offline testing', async ({ page, context }) => { + // Record mode: Save HAR manually + await context.routeFromHAR('./hars/dashboard.har', { + url: '**/api/**', + update: true // Update HAR file + }); + + await page.goto('/dashboard'); + + // Playback: Load HAR manually + await context.routeFromHAR('./hars/dashboard.har', { + url: '**/api/**', + update: false // Use existing HAR + }); +}); +``` + +**With Playwright Utils (Automatic HAR Management):** +```typescript +import { test } from '@seontechnologies/playwright-utils/network-recorder/fixtures'; + +// Record mode: Set environment variable +process.env.PW_NET_MODE = 'record'; + +test('should work offline', async ({ page, context, networkRecorder }) => { + await networkRecorder.setup(context); // Handles HAR automatically + + await page.goto('/dashboard'); + await page.click('#add-item'); + // All network traffic recorded, CRUD operations detected +}); +``` + +**Switch to playback:** +```bash +# Playback mode (offline) +PW_NET_MODE=playback npx playwright test +# Uses HAR file, no backend needed! +``` + +**Playwright Utils Benefits:** +- Automatic HAR file management (naming, paths) +- CRUD operation detection (stateful mocking) +- Environment variable control (easy switching) +- Works for complex interactions (create, update, delete) +- No manual route configuration + +### Network Request Interception + +**Vanilla Playwright:** +```typescript +test('should handle API error', async ({ page }) => { + // Manual route setup + await page.route('**/api/users', (route) => { + route.fulfill({ + status: 500, + body: JSON.stringify({ error: 'Internal server error' }) + }); + }); + + await page.goto('/users'); + + const response = await page.waitForResponse('**/api/users'); + const error = await response.json(); + + expect(error.error).toContain('Internal server'); + await expect(page.locator('.error-message')).toContainText('Server error'); +}); +``` + +**With Playwright Utils:** +```typescript +import { test } from '@seontechnologies/playwright-utils/fixtures'; + +test('should handle API error', async ({ page, interceptNetworkCall }) => { + // Stub API to return error (set up BEFORE navigation) + const usersCall = interceptNetworkCall({ + method: 'GET', + url: '**/api/users', + fulfillResponse: { + status: 500, + body: { error: 'Internal server error' } + } + }); + + await page.goto('/users'); + + // Wait for mocked response and access parsed data + const { status, responseJson } = await usersCall; + + expect(status).toBe(500); + expect(responseJson.error).toContain('Internal server'); + await expect(page.locator('.error-message')).toContainText('Server error'); +}); +``` + +**Playwright Utils Benefits:** +- Automatic JSON parsing (`responseJson` ready to use) +- Returns promise with `{ status, responseJson, requestJson }` +- No need to pass `page` (auto-injected by fixture) +- Glob pattern matching (simpler than regex) +- Single declarative call (setup + wait in one) + +## Comparison: Traditional vs Network-First + +### Loading Dashboard Data + +**Traditional (Flaky):** +```typescript +test('dashboard loads data', async ({ page }) => { + await page.goto('/dashboard'); + await page.waitForTimeout(2000); // ❌ Magic number + await expect(page.locator('table tr')).toHaveCount(5); +}); +``` + +**Failure modes:** +- API takes 2.5s → test fails +- API returns 3 items not 5 → hard to debug (which issue?) +- CI slower than local → fails in CI only + +**Network-First (Deterministic):** +```typescript +test('dashboard loads data', async ({ page }) => { + const apiPromise = page.waitForResponse( + resp => resp.url().includes('/api/dashboard') && resp.ok() + ); + + await page.goto('/dashboard'); + + const response = await apiPromise; + const { items } = await response.json(); + + // Validate API response + expect(items).toHaveLength(5); + + // Validate UI matches API + await expect(page.locator('table tr')).toHaveCount(items.length); +}); +``` + +**Benefits:** +- Waits exactly as long as needed (100ms or 5s, doesn't matter) +- Validates API response (catch backend errors) +- Validates UI matches API (catch frontend bugs) +- Works in any environment (local, CI, staging) + +### Form Submission + +**Traditional (Flaky):** +```typescript +test('form submission', async ({ page }) => { + await page.fill('#email', 'test@example.com'); + await page.click('button[type="submit"]'); + await page.waitForTimeout(3000); // ❌ Hope it's enough + await expect(page.locator('.success')).toBeVisible(); +}); +``` + +**Network-First (Deterministic):** +```typescript +test('form submission', async ({ page }) => { + const submitPromise = page.waitForResponse( + resp => resp.url().includes('/api/submit') && + resp.request().method() === 'POST' && + resp.ok() + ); + + await page.fill('#email', 'test@example.com'); + await page.click('button[type="submit"]'); + + const response = await submitPromise; + const result = await response.json(); + + expect(result.success).toBe(true); + await expect(page.locator('.success')).toBeVisible(); +}); +``` + +## Common Misconceptions + +### "I Already Use waitForSelector" + +```typescript +// This is still a hard wait in disguise +await page.click('button'); +await page.waitForSelector('.success', { timeout: 5000 }); +``` + +**Problem:** Waiting for DOM, not for the API that caused DOM change. + +**Better:** +```typescript +await page.waitForResponse(matcher); // Wait for root cause +await page.waitForSelector('.success'); // Then validate UI +``` + +### "My Tests Are Fast, Why Add Complexity?" + +**Short-term:** Tests are fast locally + +**Long-term problems:** +- Different environments (CI slower) +- Under load (API slower) +- Network variability (random) +- Scaling test suite (100 → 1000 tests) + +**Network-first prevents these issues before they appear.** + +### "Too Much Boilerplate" + +**Solution:** Extract to fixtures (see Fixture Architecture) + +```typescript +// Create reusable fixture +export const test = base.extend({ + waitForApi: async ({ page }, use) => { + await use((urlPattern: string) => { + // Returns promise immediately (doesn't await) + return page.waitForResponse( + resp => resp.url().includes(urlPattern) && resp.ok() + ); + }); + } +}); + +// Use in tests +test('test', async ({ page, waitForApi }) => { + const promise = waitForApi('/api/submit'); // Get promise + await page.click('button'); // Trigger action + await promise; // Wait for response +}); +``` + +## Technical Implementation + +For detailed network-first patterns, see the knowledge base: +- [Knowledge Base Index - Network & Reliability](/docs/reference/tea/knowledge-base.md) +- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md) + +## Related Concepts + +**Core TEA Concepts:** +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Determinism requires network-first +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - High-risk features need reliable tests + +**Technical Patterns:** +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Network utilities as fixtures +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - Network patterns in knowledge base + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Network-first in workflows +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why flakiness matters + +## Practical Guides + +**Workflow Guides:** +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Review for hard waits +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate network-first tests +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand with network patterns + +**Use-Case Guides:** +- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Fix flaky legacy tests + +**Customization:** +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Network utilities (recorder, interceptor, error monitor) + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - All workflows use network-first +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Network-first fragment +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Network-first pattern term + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/explanation/tea/risk-based-testing.md b/docs/explanation/tea/risk-based-testing.md new file mode 100644 index 00000000..14eb2502 --- /dev/null +++ b/docs/explanation/tea/risk-based-testing.md @@ -0,0 +1,586 @@ +--- +title: "Risk-Based Testing Explained" +description: Understanding how TEA uses probability × impact scoring to prioritize testing effort +--- + +# Risk-Based Testing Explained + +Risk-based testing is TEA's core principle: testing depth scales with business impact. Instead of testing everything equally, focus effort where failures hurt most. + +## Overview + +Traditional testing approaches treat all features equally: +- Every feature gets same test coverage +- Same level of scrutiny regardless of impact +- No systematic prioritization +- Testing becomes checkbox exercise + +**Risk-based testing asks:** +- What's the probability this will fail? +- What's the impact if it does fail? +- How much testing is appropriate for this risk level? + +**Result:** Testing effort matches business criticality. + +## The Problem + +### Equal Testing for Unequal Risk + +```markdown +Feature A: User login (critical path, millions of users) +Feature B: Export to PDF (nice-to-have, rarely used) + +Traditional approach: +- Both get 10 tests +- Both get same review scrutiny +- Both take same development time + +Problem: Wasting effort on low-impact features while under-testing critical paths. +``` + +### No Objective Prioritization + +```markdown +PM: "We need more tests for checkout" +QA: "How many tests?" +PM: "I don't know... a lot?" +QA: "How do we know when we have enough?" +PM: "When it feels safe?" + +Problem: Subjective decisions, no data, political debates. +``` + +## The Solution: Probability × Impact Scoring + +### Risk Score = Probability × Impact + +**Probability** (How likely to fail?) +- **1 (Low):** Stable, well-tested, simple logic +- **2 (Medium):** Moderate complexity, some unknowns +- **3 (High):** Complex, untested, many edge cases + +**Impact** (How bad if it fails?) +- **1 (Low):** Minor inconvenience, few users affected +- **2 (Medium):** Degraded experience, workarounds exist +- **3 (High):** Critical path broken, business impact + +**Score Range:** 1-9 + +#### Risk Scoring Matrix + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +graph TD + subgraph Matrix[" "] + direction TB + subgraph Impact3["Impact: HIGH (3)"] + P1I3["Score: 3
Low Risk"] + P2I3["Score: 6
HIGH RISK
Mitigation Required"] + P3I3["Score: 9
CRITICAL
Blocks Release"] + end + subgraph Impact2["Impact: MEDIUM (2)"] + P1I2["Score: 2
Low Risk"] + P2I2["Score: 4
Medium Risk"] + P3I2["Score: 6
HIGH RISK
Mitigation Required"] + end + subgraph Impact1["Impact: LOW (1)"] + P1I1["Score: 1
Low Risk"] + P2I1["Score: 2
Low Risk"] + P3I1["Score: 3
Low Risk"] + end + end + + Prob1["Probability: LOW (1)"] -.-> P1I1 + Prob1 -.-> P1I2 + Prob1 -.-> P1I3 + + Prob2["Probability: MEDIUM (2)"] -.-> P2I1 + Prob2 -.-> P2I2 + Prob2 -.-> P2I3 + + Prob3["Probability: HIGH (3)"] -.-> P3I1 + Prob3 -.-> P3I2 + Prob3 -.-> P3I3 + + style P3I3 fill:#f44336,stroke:#b71c1c,stroke-width:3px,color:#fff + style P2I3 fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000 + style P3I2 fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000 + style P2I2 fill:#fff9c4,stroke:#f57f17,stroke-width:1px,color:#000 + style P1I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 + style P2I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 + style P3I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 + style P1I2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 + style P1I3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 +``` + +**Legend:** +- 🔴 Red (Score 9): CRITICAL - Blocks release +- 🟠 Orange (Score 6-8): HIGH RISK - Mitigation required +- 🟡 Yellow (Score 4-5): MEDIUM - Mitigation recommended +- 🟢 Green (Score 1-3): LOW - Optional mitigation + +### Scoring Examples + +**Score 9 (Critical):** +``` +Feature: Payment processing +Probability: 3 (complex third-party integration) +Impact: 3 (broken payments = lost revenue) +Score: 3 × 3 = 9 + +Action: Extensive testing required +- E2E tests for all payment flows +- API tests for all payment scenarios +- Error handling for all failure modes +- Security testing for payment data +- Load testing for high traffic +- Monitoring and alerts +``` + +**Score 1 (Low):** +``` +Feature: Change profile theme color +Probability: 1 (simple UI toggle) +Impact: 1 (cosmetic only) +Score: 1 × 1 = 1 + +Action: Minimal testing +- One E2E smoke test +- Skip edge cases +- No API tests needed +``` + +**Score 6 (Medium-High):** +``` +Feature: User profile editing +Probability: 2 (moderate complexity) +Impact: 3 (users can't update info) +Score: 2 × 3 = 6 + +Action: Focused testing +- E2E test for happy path +- API tests for CRUD operations +- Validation testing +- Skip low-value edge cases +``` + +## How It Works in TEA + +### 1. Risk Categories + +TEA assesses risk across 6 categories: + +**TECH** - Technical debt, architecture fragility +``` +Example: Migrating from REST to GraphQL +Probability: 3 (major architectural change) +Impact: 3 (affects all API consumers) +Score: 9 - Extensive integration testing required +``` + +**SEC** - Security vulnerabilities +``` +Example: Adding OAuth integration +Probability: 2 (third-party dependency) +Impact: 3 (auth breach = data exposure) +Score: 6 - Security testing mandatory +``` + +**PERF** - Performance degradation +``` +Example: Adding real-time notifications +Probability: 2 (WebSocket complexity) +Impact: 2 (slower experience) +Score: 4 - Load testing recommended +``` + +**DATA** - Data integrity, corruption +``` +Example: Database migration +Probability: 2 (schema changes) +Impact: 3 (data loss unacceptable) +Score: 6 - Data validation tests required +``` + +**BUS** - Business logic errors +``` +Example: Discount calculation +Probability: 2 (business rules complex) +Impact: 3 (wrong prices = revenue loss) +Score: 6 - Business logic tests mandatory +``` + +**OPS** - Operational issues +``` +Example: Logging system update +Probability: 1 (straightforward) +Impact: 2 (debugging harder without logs) +Score: 2 - Basic smoke test sufficient +``` + +### 2. Test Priorities (P0-P3) + +Risk scores inform test priorities (but aren't the only factor): + +**P0 - Critical Path** +- **Risk Scores:** Typically 6-9 (high risk) +- **Other Factors:** Revenue impact, security-critical, regulatory compliance, frequent usage +- **Coverage Target:** 100% +- **Test Levels:** E2E + API +- **Example:** Login, checkout, payment processing + +**P1 - High Value** +- **Risk Scores:** Typically 4-6 (medium-high risk) +- **Other Factors:** Core user journeys, complex logic, integration points +- **Coverage Target:** 90% +- **Test Levels:** API + selective E2E +- **Example:** Profile editing, search, filters + +**P2 - Medium Value** +- **Risk Scores:** Typically 2-4 (medium risk) +- **Other Factors:** Secondary features, admin functionality, reporting +- **Coverage Target:** 50% +- **Test Levels:** API happy path only +- **Example:** Export features, advanced settings + +**P3 - Low Value** +- **Risk Scores:** Typically 1-2 (low risk) +- **Other Factors:** Rarely used, nice-to-have, cosmetic +- **Coverage Target:** 20% (smoke test) +- **Test Levels:** E2E smoke test only +- **Example:** Theme customization, experimental features + +**Note:** Priorities consider risk scores plus business context (usage frequency, user impact, etc.). See [Test Priorities Matrix](/docs/reference/tea/knowledge-base.md#test-priorities-matrix) for complete criteria. + +### 3. Mitigation Plans + +**Scores ≥6 require documented mitigation:** + +```markdown +## Risk Mitigation + +**Risk:** Payment integration failure (Score: 9) + +**Mitigation Plan:** +- Create comprehensive test suite (20+ tests) +- Add payment sandbox environment +- Implement retry logic with idempotency +- Add monitoring and alerts +- Document rollback procedure + +**Owner:** Backend team lead +**Deadline:** Before production deployment +**Status:** In progress +``` + +**Gate Rules:** +- **Score = 9** (Critical): Mandatory FAIL - blocks release without mitigation +- **Score 6-8** (High): Requires mitigation plan, becomes CONCERNS if incomplete +- **Score 4-5** (Medium): Mitigation recommended but not required +- **Score 1-3** (Low): No mitigation needed + +## Comparison: Traditional vs Risk-Based + +### Traditional Approach + +```typescript +// Test everything equally +describe('User profile', () => { + test('should display name'); + test('should display email'); + test('should display phone'); + test('should display address'); + test('should display bio'); + test('should display avatar'); + test('should display join date'); + test('should display last login'); + test('should display theme preference'); + test('should display language preference'); + // 10 tests for profile display (all equal priority) +}); +``` + +**Problems:** +- Same effort for critical (name) vs trivial (theme) +- No guidance on what matters +- Wastes time on low-value tests + +### Risk-Based Approach + +```typescript +// Test based on risk + +describe('User profile - Critical (P0)', () => { + test('should display name and email'); // Score: 9 (identity critical) + test('should allow editing name and email'); + test('should validate email format'); + test('should prevent unauthorized edits'); + // 4 focused tests on high-risk areas +}); + +describe('User profile - High Value (P1)', () => { + test('should upload avatar'); // Score: 6 (users care about this) + test('should update bio'); + // 2 tests for high-value features +}); + +// P2: Theme preference - single smoke test +// P3: Last login display - skip (read-only, low value) +``` + +**Benefits:** +- 6 focused tests vs 10 unfocused tests +- Effort matches business impact +- Clear priorities guide development +- No wasted effort on trivial features + +## When to Use Risk-Based Testing + +### Always Use For: + +**Enterprise projects:** +- High stakes (revenue, compliance, security) +- Many features competing for test effort +- Need objective prioritization + +**Large codebases:** +- Can't test everything exhaustively +- Need to focus limited QA resources +- Want data-driven decisions + +**Regulated industries:** +- Must justify testing decisions +- Auditors want risk assessments +- Compliance requires evidence + +### Consider Skipping For: + +**Tiny projects:** +- 5 features total +- Can test everything thoroughly +- Risk scoring is overhead + +**Prototypes:** +- Throw-away code +- Speed over quality +- Learning experiments + +## Real-World Example + +### Scenario: E-Commerce Checkout Redesign + +**Feature:** Redesigning checkout flow from 5 steps to 3 steps + +**Risk Assessment:** + +| Component | Probability | Impact | Score | Priority | Testing | +|-----------|-------------|--------|-------|----------|---------| +| **Payment processing** | 3 | 3 | 9 | P0 | 15 E2E + 20 API tests | +| **Order validation** | 2 | 3 | 6 | P1 | 5 E2E + 10 API tests | +| **Shipping calculation** | 2 | 2 | 4 | P1 | 3 E2E + 8 API tests | +| **Promo code validation** | 2 | 2 | 4 | P1 | 2 E2E + 5 API tests | +| **Gift message** | 1 | 1 | 1 | P3 | 1 E2E smoke test | + +**Test Budget:** 40 hours + +**Allocation:** +- Payment (Score 9): 20 hours (50%) +- Order validation (Score 6): 8 hours (20%) +- Shipping (Score 4): 6 hours (15%) +- Promo codes (Score 4): 4 hours (10%) +- Gift message (Score 1): 2 hours (5%) + +**Result:** 50% of effort on highest-risk feature (payment), proportional allocation for others. + +### Without Risk-Based Testing: + +**Equal allocation:** 8 hours per component = wasted effort on gift message, under-testing payment. + +**Result:** Payment bugs slip through (critical), perfect testing of gift message (trivial). + +## Mitigation Strategies by Risk Level + +### Score 9: Mandatory Mitigation (Blocks Release) + +```markdown +**Gate Impact:** FAIL - Cannot deploy without mitigation + +**Actions:** +- Comprehensive test suite (E2E, API, security) +- Multiple test environments (dev, staging, prod-mirror) +- Load testing and performance validation +- Security audit and penetration testing +- Monitoring and alerting +- Rollback plan documented +- On-call rotation assigned + +**Cannot deploy until score is mitigated below 9.** +``` + +### Score 6-8: Required Mitigation (Gate: CONCERNS) + +```markdown +**Gate Impact:** CONCERNS - Can deploy with documented mitigation plan + +**Actions:** +- Targeted test suite (happy path + critical errors) +- Test environment setup +- Monitoring plan +- Document mitigation and owners + +**Can deploy with approved mitigation plan.** +``` + +### Score 4-5: Recommended Mitigation + +```markdown +**Gate Impact:** Advisory - Does not affect gate decision + +**Actions:** +- Basic test coverage +- Standard monitoring +- Document known limitations + +**Can deploy, mitigation recommended but not required.** +``` + +### Score 1-3: Optional Mitigation + +```markdown +**Gate Impact:** None + +**Actions:** +- Smoke test if desired +- Feature flag for easy disable (optional) + +**Can deploy without mitigation.** +``` + +## Technical Implementation + +For detailed risk governance patterns, see the knowledge base: +- [Knowledge Base Index - Risk & Gates](/docs/reference/tea/knowledge-base.md) +- [TEA Command Reference - *test-design](/docs/reference/tea/commands.md#test-design) + +### Risk Scoring Matrix + +TEA uses this framework in `*test-design`: + +``` + Impact + 1 2 3 + ┌────┬────┬────┐ + 1 │ 1 │ 2 │ 3 │ Low risk +P 2 │ 2 │ 4 │ 6 │ Medium risk +r 3 │ 3 │ 6 │ 9 │ High risk +o └────┴────┴────┘ +b Low Med High +``` + +### Gate Decision Rules + +| Score | Mitigation Required | Gate Impact | +|-------|-------------------|-------------| +| **9** | Mandatory, blocks release | FAIL if no mitigation | +| **6-8** | Required, documented plan | CONCERNS if incomplete | +| **4-5** | Recommended | Advisory only | +| **1-3** | Optional | No impact | + +#### Gate Decision Flow + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +flowchart TD + Start([Risk Assessment]) --> Score{Risk Score?} + + Score -->|Score = 9| Critical[CRITICAL RISK
Score: 9] + Score -->|Score 6-8| High[HIGH RISK
Score: 6-8] + Score -->|Score 4-5| Medium[MEDIUM RISK
Score: 4-5] + Score -->|Score 1-3| Low[LOW RISK
Score: 1-3] + + Critical --> HasMit9{Mitigation
Plan?} + HasMit9 -->|Yes| Concerns9[CONCERNS ⚠️
Can deploy with plan] + HasMit9 -->|No| Fail[FAIL ❌
Blocks release] + + High --> HasMit6{Mitigation
Plan?} + HasMit6 -->|Yes| Pass6[PASS ✅
or CONCERNS ⚠️] + HasMit6 -->|No| Concerns6[CONCERNS ⚠️
Document plan needed] + + Medium --> Advisory[Advisory Only
No gate impact] + Low --> NoAction[No Action
Proceed] + + style Critical fill:#f44336,stroke:#b71c1c,stroke-width:3px,color:#fff + style Fail fill:#d32f2f,stroke:#b71c1c,stroke-width:3px,color:#fff + style High fill:#ff9800,stroke:#e65100,stroke-width:2px,color:#000 + style Concerns9 fill:#ffc107,stroke:#f57f17,stroke-width:2px,color:#000 + style Concerns6 fill:#ffc107,stroke:#f57f17,stroke-width:2px,color:#000 + style Pass6 fill:#4caf50,stroke:#1b5e20,stroke-width:2px,color:#fff + style Medium fill:#fff9c4,stroke:#f57f17,stroke-width:1px,color:#000 + style Low fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px,color:#000 + style Advisory fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px,color:#000 + style NoAction fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px,color:#000 +``` + +## Common Misconceptions + +### "Risk-based = Less Testing" + +**Wrong:** Risk-based testing often means MORE testing where it matters. + +**Example:** +- Traditional: 50 tests spread equally +- Risk-based: 70 tests focused on P0/P1 (more total, better allocated) + +### "Low Priority = Skip Testing" + +**Wrong:** P3 still gets smoke tests. + +**Correct:** +- P3: Smoke test (feature works at all) +- P2: Happy path (feature works correctly) +- P1: Happy path + errors +- P0: Comprehensive (all scenarios) + +### "Risk Scores Are Permanent" + +**Wrong:** Risk changes over time. + +**Correct:** +- Initial launch: Payment is Score 9 (untested integration) +- After 6 months: Payment is Score 6 (proven in production) +- Re-assess risk quarterly + +## Related Concepts + +**Core TEA Concepts:** +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Quality complements risk assessment +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - When risk-based testing matters most +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - How risk patterns are loaded + +**Technical Patterns:** +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Building risk-appropriate test infrastructure +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Quality patterns for high-risk features + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Risk assessment in TEA lifecycle +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Design philosophy + +## Practical Guides + +**Workflow Guides:** +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Apply risk scoring +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Gate decisions based on risk +- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) - NFR risk assessment + +**Use-Case Guides:** +- [Running TEA for Enterprise](/docs/how-to/workflows/run-tea-for-enterprise.md) - Enterprise risk management + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - *test-design, *nfr-assess, *trace +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Risk governance fragments +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Risk-based testing term + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/explanation/tea/test-quality-standards.md b/docs/explanation/tea/test-quality-standards.md new file mode 100644 index 00000000..500ad0cf --- /dev/null +++ b/docs/explanation/tea/test-quality-standards.md @@ -0,0 +1,834 @@ +--- +title: "Test Quality Standards Explained" +description: Understanding TEA's Definition of Done for deterministic, isolated, and maintainable tests +--- + +# Test Quality Standards Explained + +Test quality standards define what makes a test "good" in TEA. These aren't suggestions - they're the Definition of Done that prevents tests from rotting in review. + +## Overview + +**TEA's Quality Principles:** +- **Deterministic** - Same result every run +- **Isolated** - No dependencies on other tests +- **Explicit** - Assertions visible in test body +- **Focused** - Single responsibility, appropriate size +- **Fast** - Execute in reasonable time + +**Why these matter:** Tests that violate these principles create maintenance burden, slow down development, and lose team trust. + +## The Problem + +### Tests That Rot in Review + +```typescript +// ❌ The anti-pattern: This test will rot +test('user can do stuff', async ({ page }) => { + await page.goto('/'); + await page.waitForTimeout(5000); // Non-deterministic + + if (await page.locator('.banner').isVisible()) { // Conditional + await page.click('.dismiss'); + } + + try { // Try-catch for flow control + await page.click('#load-more'); + } catch (e) { + // Silently continue + } + + // ... 300 more lines of test logic + // ... no clear assertions +}); +``` + +**What's wrong:** +- **Hard wait** - Flaky, wastes time +- **Conditional** - Non-deterministic behavior +- **Try-catch** - Hides failures +- **Too large** - Hard to maintain +- **Vague name** - Unclear purpose +- **No explicit assertions** - What's being tested? + +**Result:** PR review comments: "This test is flaky, please fix" → never merged → test deleted → coverage lost + +### The Slop Factory + +AI-generated tests without quality standards: + +```typescript +// AI generates 50 tests like this: +test('test1', async ({ page }) => { + await page.goto('/'); + await page.waitForTimeout(3000); + // ... flaky, vague, redundant +}); + +test('test2', async ({ page }) => { + await page.goto('/'); + await page.waitForTimeout(3000); + // ... duplicates test1 +}); + +// ... 48 more similar tests +``` + +**Result:** 50 tests, 80% redundant, 90% flaky, 0% trusted by team. + +## The Solution: TEA's Quality Standards + +### 1. Determinism (No Flakiness) + +**Rule:** Test produces same result every run. + +**Requirements:** +- ❌ No hard waits (`waitForTimeout`) +- ❌ No conditionals for flow control (`if/else`) +- ❌ No try-catch for flow control +- ✅ Use network-first patterns (wait for responses) +- ✅ Use explicit waits (waitForSelector, waitForResponse) + +**Bad Example:** +```typescript +test('flaky test', async ({ page }) => { + await page.click('button'); + await page.waitForTimeout(2000); // ❌ Might be too short + + if (await page.locator('.modal').isVisible()) { // ❌ Non-deterministic + await page.click('.dismiss'); + } + + try { // ❌ Silently handles errors + await expect(page.locator('.success')).toBeVisible(); + } catch (e) { + // Test passes even if assertion fails! + } +}); +``` + +**Good Example:** +```typescript +test('deterministic test', async ({ page }) => { + const responsePromise = page.waitForResponse( + resp => resp.url().includes('/api/submit') && resp.ok() + ); + + await page.click('button'); + await responsePromise; // ✅ Wait for actual response + + // Modal should ALWAYS show (make it deterministic) + await expect(page.locator('.modal')).toBeVisible(); + await page.click('.dismiss'); + + // Explicit assertion (fails if not visible) + await expect(page.locator('.success')).toBeVisible(); +}); +``` + +**Why it works:** +- Waits for actual event (network response) +- No conditionals (behavior is deterministic) +- Assertions fail loudly (no silent failures) +- Same result every run (deterministic) + +### 2. Isolation (No Dependencies) + +**Rule:** Test runs independently, no shared state. + +**Requirements:** +- ✅ Self-cleaning (cleanup after test) +- ✅ No global state dependencies +- ✅ Can run in parallel +- ✅ Can run in any order +- ✅ Use unique test data + +**Bad Example:** +```typescript +// ❌ Tests depend on execution order +let userId: string; // Shared global state + +test('create user', async ({ apiRequest }) => { + const { body } = await apiRequest({ + method: 'POST', + path: '/api/users', + body: { email: 'test@example.com' } // 'body' not 'data' (hard-coded) + }); + userId = body.id; // Store in global +}); + +test('update user', async ({ apiRequest }) => { + // Depends on previous test setting userId + await apiRequest({ + method: 'PATCH', + path: `/api/users/${userId}`, + body: { name: 'Updated' } // 'body' not 'data' + }); + // No cleanup - leaves user in database +}); +``` + +**Problems:** +- Tests must run in order (can't parallelize) +- Second test fails if first skipped (`.only`) +- Hard-coded data causes conflicts +- No cleanup (database fills with test data) + +**Good Example (Vanilla Playwright):** +```typescript +test('should update user profile', async ({ request }) => { + // Create unique test data + const testEmail = `test-${Date.now()}@example.com`; + + // Setup: Create user + const createResp = await request.post('/api/users', { + data: { email: testEmail, name: 'Original' } + }); + const user = await createResp.json(); + + // Test: Update user + const updateResp = await request.patch(`/api/users/${user.id}`, { + data: { name: 'Updated' } + }); + const updated = await updateResp.json(); + + expect(updated.name).toBe('Updated'); + + // Cleanup: Delete user + await request.delete(`/api/users/${user.id}`); +}); +``` + +**Even Better (With Playwright Utils):** +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { expect } from '@playwright/test'; +import { faker } from '@faker-js/faker'; + +test('should update user profile', async ({ apiRequest }) => { + // Dynamic unique test data + const testEmail = faker.internet.email(); + + // Setup: Create user + const { status: createStatus, body: user } = await apiRequest({ + method: 'POST', + path: '/api/users', + body: { email: testEmail, name: faker.person.fullName() } // 'body' not 'data' + }); + + expect(createStatus).toBe(201); + + // Test: Update user + const { status, body: updated } = await apiRequest({ + method: 'PATCH', + path: `/api/users/${user.id}`, + body: { name: 'Updated Name' } // 'body' not 'data' + }); + + expect(status).toBe(200); + expect(updated.name).toBe('Updated Name'); + + // Cleanup: Delete user + await apiRequest({ + method: 'DELETE', + path: `/api/users/${user.id}` + }); +}); +``` + +**Playwright Utils Benefits:** +- `{ status, body }` destructuring (cleaner than `response.status()` + `await response.json()`) +- No manual `await response.json()` +- Automatic retry for 5xx errors +- Optional schema validation with `.validateSchema()` + +**Why it works:** +- No global state +- Unique test data (no conflicts) +- Self-cleaning (deletes user) +- Can run in parallel +- Can run in any order + +### 3. Explicit Assertions (No Hidden Validation) + +**Rule:** Assertions visible in test body, not abstracted. + +**Requirements:** +- ✅ Assertions in test code (not helper functions) +- ✅ Specific assertions (not generic `toBeTruthy`) +- ✅ Meaningful expectations (test actual behavior) + +**Bad Example:** +```typescript +// ❌ Assertions hidden in helper +async function verifyProfilePage(page: Page) { + // Assertions buried in helper (not visible in test) + await expect(page.locator('h1')).toBeVisible(); + await expect(page.locator('.email')).toContainText('@'); + await expect(page.locator('.name')).not.toBeEmpty(); +} + +test('profile page', async ({ page }) => { + await page.goto('/profile'); + await verifyProfilePage(page); // What's being verified? +}); +``` + +**Problems:** +- Can't see what's tested (need to read helper) +- Hard to debug failures (which assertion failed?) +- Reduces test readability +- Hides important validation + +**Good Example:** +```typescript +// ✅ Assertions explicit in test +test('should display profile with correct data', async ({ page }) => { + await page.goto('/profile'); + + // Explicit assertions - clear what's tested + await expect(page.locator('h1')).toContainText('Test User'); + await expect(page.locator('.email')).toContainText('test@example.com'); + await expect(page.locator('.bio')).toContainText('Software Engineer'); + await expect(page.locator('img[alt="Avatar"]')).toBeVisible(); +}); +``` + +**Why it works:** +- See what's tested at a glance +- Debug failures easily (know which assertion failed) +- Test is self-documenting +- No hidden behavior + +**Exception:** Use helper for setup/cleanup, not assertions. + +### 4. Focused Tests (Appropriate Size) + +**Rule:** Test has single responsibility, reasonable size. + +**Requirements:** +- ✅ Test size < 300 lines +- ✅ Single responsibility (test one thing well) +- ✅ Clear describe/test names +- ✅ Appropriate scope (not too granular, not too broad) + +**Bad Example:** +```typescript +// ❌ 500-line test testing everything +test('complete user flow', async ({ page }) => { + // Registration (50 lines) + await page.goto('/register'); + await page.fill('#email', 'test@example.com'); + // ... 48 more lines + + // Profile setup (100 lines) + await page.goto('/profile'); + // ... 98 more lines + + // Settings configuration (150 lines) + await page.goto('/settings'); + // ... 148 more lines + + // Data export (200 lines) + await page.goto('/export'); + // ... 198 more lines + + // Total: 500 lines, testing 4 different features +}); +``` + +**Problems:** +- Failure in line 50 prevents testing lines 51-500 +- Hard to understand (what's being tested?) +- Slow to execute (testing too much) +- Hard to debug (which feature failed?) + +**Good Example:** +```typescript +// ✅ Focused tests - one responsibility each + +test('should register new user', async ({ page }) => { + await page.goto('/register'); + await page.fill('#email', 'test@example.com'); + await page.fill('#password', 'password123'); + await page.click('button[type="submit"]'); + + await expect(page).toHaveURL('/welcome'); + await expect(page.locator('h1')).toContainText('Welcome'); +}); + +test('should configure user profile', async ({ page, authSession }) => { + await authSession.login({ email: 'test@example.com', password: 'pass' }); + await page.goto('/profile'); + + await page.fill('#name', 'Test User'); + await page.fill('#bio', 'Software Engineer'); + await page.click('button:has-text("Save")'); + + await expect(page.locator('.success')).toBeVisible(); +}); + +// ... separate tests for settings, export (each < 50 lines) +``` + +**Why it works:** +- Each test has one responsibility +- Failure is easy to diagnose +- Can run tests independently +- Test names describe exactly what's tested + +### 5. Fast Execution (Performance Budget) + +**Rule:** Individual test executes in < 1.5 minutes. + +**Requirements:** +- ✅ Test execution < 90 seconds +- ✅ Efficient selectors (getByRole > XPath) +- ✅ Minimal redundant actions +- ✅ Parallel execution enabled + +**Bad Example:** +```typescript +// ❌ Slow test (3+ minutes) +test('slow test', async ({ page }) => { + await page.goto('/'); + await page.waitForTimeout(10000); // 10s wasted + + // Navigate through 10 pages (2 minutes) + for (let i = 1; i <= 10; i++) { + await page.click(`a[href="/page-${i}"]`); + await page.waitForTimeout(5000); // 5s per page = 50s wasted + } + + // Complex XPath selector (slow) + await page.locator('//div[@class="container"]/section[3]/div[2]/p').click(); + + // More waiting + await page.waitForTimeout(30000); // 30s wasted + + await expect(page.locator('.result')).toBeVisible(); +}); +``` + +**Total time:** 3+ minutes (95 seconds wasted on hard waits) + +**Good Example:** +```typescript +// ✅ Fast test (< 10 seconds) +test('fast test', async ({ page }) => { + // Set up response wait + const apiPromise = page.waitForResponse( + resp => resp.url().includes('/api/result') && resp.ok() + ); + + await page.goto('/'); + + // Direct navigation (skip intermediate pages) + await page.goto('/page-10'); + + // Efficient selector + await page.getByRole('button', { name: 'Submit' }).click(); + + // Wait for actual response (fast when API is fast) + await apiPromise; + + await expect(page.locator('.result')).toBeVisible(); +}); +``` + +**Total time:** < 10 seconds (no wasted waits) + +## TEA's Quality Scoring + +TEA reviews tests against these standards in `*test-review`: + +### Scoring Categories (100 points total) + +**Determinism (35 points):** +- No hard waits: 10 points +- No conditionals: 10 points +- No try-catch flow: 10 points +- Network-first patterns: 5 points + +**Isolation (25 points):** +- Self-cleaning: 15 points +- No global state: 5 points +- Parallel-safe: 5 points + +**Assertions (20 points):** +- Explicit in test body: 10 points +- Specific and meaningful: 10 points + +**Structure (10 points):** +- Test size < 300 lines: 5 points +- Clear naming: 5 points + +**Performance (10 points):** +- Execution time < 1.5 min: 10 points + +#### Quality Scoring Breakdown + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'14px'}}}%% +pie title Test Quality Score (100 points) + "Determinism" : 35 + "Isolation" : 25 + "Assertions" : 20 + "Structure" : 10 + "Performance" : 10 +``` + +```mermaid +%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'13px'}}}%% +flowchart LR + subgraph Det[Determinism - 35 pts] + D1[No hard waits
10 pts] + D2[No conditionals
10 pts] + D3[No try-catch flow
10 pts] + D4[Network-first
5 pts] + end + + subgraph Iso[Isolation - 25 pts] + I1[Self-cleaning
15 pts] + I2[No global state
5 pts] + I3[Parallel-safe
5 pts] + end + + subgraph Assrt[Assertions - 20 pts] + A1[Explicit in body
10 pts] + A2[Specific/meaningful
10 pts] + end + + subgraph Struct[Structure - 10 pts] + S1[Size < 300 lines
5 pts] + S2[Clear naming
5 pts] + end + + subgraph Perf[Performance - 10 pts] + P1[Time < 1.5 min
10 pts] + end + + Det --> Total([Total: 100 points]) + Iso --> Total + Assrt --> Total + Struct --> Total + Perf --> Total + + style Det fill:#ffebee,stroke:#c62828,stroke-width:2px + style Iso fill:#e3f2fd,stroke:#1565c0,stroke-width:2px + style Assrt fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px + style Struct fill:#fff9c4,stroke:#f57f17,stroke-width:2px + style Perf fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px + style Total fill:#fff,stroke:#000,stroke-width:3px +``` + +### Score Interpretation + +| Score | Interpretation | Action | +|-------|----------------|--------| +| **90-100** | Excellent | Production-ready, minimal changes | +| **80-89** | Good | Minor improvements recommended | +| **70-79** | Acceptable | Address recommendations before release | +| **60-69** | Needs Work | Fix critical issues | +| **< 60** | Critical | Significant refactoring needed | + +## Comparison: Good vs Bad Tests + +### Example: User Login + +**Bad Test (Score: 45/100):** +```typescript +test('login test', async ({ page }) => { // Vague name + await page.goto('/login'); + await page.waitForTimeout(3000); // -10 (hard wait) + + await page.fill('[name="email"]', 'test@example.com'); + await page.fill('[name="password"]', 'password'); + + if (await page.locator('.remember-me').isVisible()) { // -10 (conditional) + await page.click('.remember-me'); + } + + await page.click('button'); + + try { // -10 (try-catch flow) + await page.waitForURL('/dashboard', { timeout: 5000 }); + } catch (e) { + // Ignore navigation failure + } + + // No assertions! -10 + // No cleanup! -10 +}); +``` + +**Issues:** +- Determinism: 5/35 (hard wait, conditional, try-catch) +- Isolation: 10/25 (no cleanup) +- Assertions: 0/20 (no assertions!) +- Structure: 15/10 (okay) +- Performance: 5/10 (slow) +- **Total: 45/100** + +**Good Test (Score: 95/100):** +```typescript +test('should login with valid credentials and redirect to dashboard', async ({ page, authSession }) => { + // Use fixture for deterministic auth + const loginPromise = page.waitForResponse( + resp => resp.url().includes('/api/auth/login') && resp.ok() + ); + + await page.goto('/login'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password').fill('password123'); + await page.getByRole('button', { name: 'Sign in' }).click(); + + // Wait for actual API response + const response = await loginPromise; + const { token } = await response.json(); + + // Explicit assertions + expect(token).toBeDefined(); + await expect(page).toHaveURL('/dashboard'); + await expect(page.getByText('Welcome back')).toBeVisible(); + + // Cleanup handled by authSession fixture +}); +``` + +**Quality:** +- Determinism: 35/35 (network-first, no conditionals) +- Isolation: 25/25 (fixture handles cleanup) +- Assertions: 20/20 (explicit and specific) +- Structure: 10/10 (clear name, focused) +- Performance: 5/10 (< 1 min) +- **Total: 95/100** + +### Example: API Testing + +**Bad Test (Score: 50/100):** +```typescript +test('api test', async ({ request }) => { + const response = await request.post('/api/users', { + data: { email: 'test@example.com' } // Hard-coded (conflicts) + }); + + if (response.ok()) { // Conditional + const user = await response.json(); + // Weak assertion + expect(user).toBeTruthy(); + } + + // No cleanup - user left in database +}); +``` + +**Good Test (Score: 92/100):** +```typescript +test('should create user with valid data', async ({ apiRequest }) => { + // Unique test data + const testEmail = `test-${Date.now()}@example.com`; + + // Create user + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/users', + data: { email: testEmail, name: 'Test User' } + }); + + // Explicit assertions + expect(status).toBe(201); + expect(body.id).toBeDefined(); + expect(body.email).toBe(testEmail); + expect(body.name).toBe('Test User'); + + // Cleanup + await apiRequest({ + method: 'DELETE', + path: `/api/users/${body.id}` + }); +}); +``` + +## How TEA Enforces Standards + +### During Test Generation (*atdd, *automate) + +TEA generates tests following standards by default: + +```typescript +// TEA-generated test (automatically follows standards) +test('should submit contact form', async ({ page }) => { + // Network-first pattern (no hard waits) + const submitPromise = page.waitForResponse( + resp => resp.url().includes('/api/contact') && resp.ok() + ); + + // Accessible selectors (resilient) + await page.getByLabel('Name').fill('Test User'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Message').fill('Test message'); + await page.getByRole('button', { name: 'Send' }).click(); + + const response = await submitPromise; + const result = await response.json(); + + // Explicit assertions + expect(result.success).toBe(true); + await expect(page.getByText('Message sent')).toBeVisible(); + + // Size: 15 lines (< 300 ✓) + // Execution: ~2 seconds (< 90s ✓) +}); +``` + +### During Test Review (*test-review) + +TEA audits tests and flags violations: + +```markdown +## Critical Issues + +### Hard Wait Detected (tests/login.spec.ts:23) +**Issue:** `await page.waitForTimeout(3000)` +**Score Impact:** -10 (Determinism) +**Fix:** Use network-first pattern + +### Conditional Flow Control (tests/profile.spec.ts:45) +**Issue:** `if (await page.locator('.banner').isVisible())` +**Score Impact:** -10 (Determinism) +**Fix:** Make banner presence deterministic + +## Recommendations + +### Extract Fixture (tests/auth.spec.ts) +**Issue:** Login code repeated 5 times +**Score Impact:** -3 (Structure) +**Fix:** Extract to authSession fixture +``` + +## Definition of Done Checklist + +When is a test "done"? + +**Test Quality DoD:** +- [ ] No hard waits (`waitForTimeout`) +- [ ] No conditionals for flow control +- [ ] No try-catch for flow control +- [ ] Network-first patterns used +- [ ] Assertions explicit in test body +- [ ] Test size < 300 lines +- [ ] Clear, descriptive test name +- [ ] Self-cleaning (cleanup in afterEach or test) +- [ ] Unique test data (no hard-coded values) +- [ ] Execution time < 1.5 minutes +- [ ] Can run in parallel +- [ ] Can run in any order + +**Code Review DoD:** +- [ ] Test quality score > 80 +- [ ] No critical issues from `*test-review` +- [ ] Follows project patterns (fixtures, selectors) +- [ ] Test reviewed by team member + +## Common Quality Issues + +### Issue: "My test needs conditionals for optional elements" + +**Wrong approach:** +```typescript +if (await page.locator('.banner').isVisible()) { + await page.click('.dismiss'); +} +``` + +**Right approach - Make it deterministic:** +```typescript +// Option 1: Always expect banner +await expect(page.locator('.banner')).toBeVisible(); +await page.click('.dismiss'); + +// Option 2: Test both scenarios separately +test('should show banner for new users', ...); +test('should not show banner for returning users', ...); +``` + +### Issue: "My test needs try-catch for error handling" + +**Wrong approach:** +```typescript +try { + await page.click('#optional-button'); +} catch (e) { + // Silently continue +} +``` + +**Right approach - Make failures explicit:** +```typescript +// Option 1: Button should exist +await page.click('#optional-button'); // Fails loudly if missing + +// Option 2: Button might not exist (test both) +test('should work with optional button', async ({ page }) => { + const hasButton = await page.locator('#optional-button').count() > 0; + if (hasButton) { + await page.click('#optional-button'); + } + // But now you're testing optional behavior explicitly +}); +``` + +### Issue: "Hard waits are easier than network patterns" + +**Short-term:** Hard waits seem simpler +**Long-term:** Flaky tests waste more time than learning network patterns + +**Investment:** +- 30 minutes to learn network-first patterns +- Prevents hundreds of hours debugging flaky tests +- Tests run faster (no wasted waits) +- Team trusts test suite + +## Technical Implementation + +For detailed test quality patterns, see: +- [Test Quality Fragment](/docs/reference/tea/knowledge-base.md#test-quality) +- [Test Levels Framework Fragment](/docs/reference/tea/knowledge-base.md#test-levels-framework) +- [Complete Knowledge Base Index](/docs/reference/tea/knowledge-base.md) + +## Related Concepts + +**Core TEA Concepts:** +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Quality scales with risk +- [Knowledge Base System](/docs/explanation/tea/knowledge-base-system.md) - How standards are enforced +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Quality in different models + +**Technical Patterns:** +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Determinism explained +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Isolation through fixtures + +**Overview:** +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Quality standards in lifecycle +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Why quality matters + +## Practical Guides + +**Workflow Guides:** +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Audit against these standards +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate quality tests +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand with quality + +**Use-Case Guides:** +- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Improve legacy quality +- [Running TEA for Enterprise](/docs/how-to/workflows/run-tea-for-enterprise.md) - Enterprise quality thresholds + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - *test-review command +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Test quality fragment +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA terminology + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/brownfield/use-tea-with-existing-tests.md b/docs/how-to/brownfield/use-tea-with-existing-tests.md new file mode 100644 index 00000000..231a0279 --- /dev/null +++ b/docs/how-to/brownfield/use-tea-with-existing-tests.md @@ -0,0 +1,597 @@ +--- +title: "Using TEA with Existing Tests (Brownfield)" +description: Apply TEA workflows to legacy codebases with existing test suites +--- + +# Using TEA with Existing Tests (Brownfield) + +Use TEA on brownfield projects (existing codebases with legacy tests) to establish coverage baselines, identify gaps, and improve test quality without starting from scratch. + +## When to Use This + +- Existing codebase with some tests already written +- Legacy test suite needs quality improvement +- Adding features to existing application +- Need to understand current test coverage +- Want to prevent regression as you add features + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Existing codebase with tests (even if incomplete or low quality) +- Tests run successfully (or at least can be executed) + +**Note:** If your codebase is completely undocumented, run `*document-project` first to create baseline documentation. + +## Brownfield Strategy + +### Phase 1: Establish Baseline + +Understand what you have before changing anything. + +#### Step 1: Baseline Coverage with *trace + +Run `*trace` Phase 1 to map existing tests to requirements: + +``` +*trace +``` + +**Select:** Phase 1 (Requirements Traceability) + +**Provide:** +- Existing requirements docs (PRD, user stories, feature specs) +- Test location (`tests/` or wherever tests live) +- Focus areas (specific features if large codebase) + +**Output:** `traceability-matrix.md` showing: +- Which requirements have tests +- Which requirements lack coverage +- Coverage classification (FULL/PARTIAL/NONE) +- Gap prioritization + +**Example Baseline:** +```markdown +# Baseline Coverage (Before Improvements) + +**Total Requirements:** 50 +**Full Coverage:** 15 (30%) +**Partial Coverage:** 20 (40%) +**No Coverage:** 15 (30%) + +**By Priority:** +- P0: 50% coverage (5/10) ❌ Critical gap +- P1: 40% coverage (8/20) ⚠️ Needs improvement +- P2: 20% coverage (2/10) ✅ Acceptable +``` + +This baseline becomes your improvement target. + +#### Step 2: Quality Audit with *test-review + +Run `*test-review` on existing tests: + +``` +*test-review tests/ +``` + +**Output:** `test-review.md` with quality score and issues. + +**Common Brownfield Issues:** +- Hard waits everywhere (`page.waitForTimeout(5000)`) +- Fragile CSS selectors (`.class > div:nth-child(3)`) +- No test isolation (tests depend on execution order) +- Try-catch for flow control +- Tests don't clean up (leave test data in DB) + +**Example Baseline Quality:** +```markdown +# Quality Score: 55/100 + +**Critical Issues:** 12 +- 8 hard waits +- 4 conditional flow control + +**Recommendations:** 25 +- Extract fixtures +- Improve selectors +- Add network assertions +``` + +This shows where to focus improvement efforts. + +### Phase 2: Prioritize Improvements + +Don't try to fix everything at once. + +#### Focus on Critical Path First + +**Priority 1: P0 Requirements** +``` +Goal: Get P0 coverage to 100% + +Actions: +1. Identify P0 requirements with no tests (from trace) +2. Run *automate to generate tests for missing P0 scenarios +3. Fix critical quality issues in P0 tests (from test-review) +``` + +**Priority 2: Fix Flaky Tests** +``` +Goal: Eliminate flakiness + +Actions: +1. Identify tests with hard waits (from test-review) +2. Replace with network-first patterns +3. Run burn-in loops to verify stability +``` + +**Example Modernization:** + +**Before (Flaky - Hard Waits):** +```typescript +test('checkout completes', async ({ page }) => { + await page.click('button[name="checkout"]'); + await page.waitForTimeout(5000); // ❌ Flaky + await expect(page.locator('.confirmation')).toBeVisible(); +}); +``` + +**After (Network-First - Vanilla):** +```typescript +test('checkout completes', async ({ page }) => { + const checkoutPromise = page.waitForResponse( + resp => resp.url().includes('/api/checkout') && resp.ok() + ); + await page.click('button[name="checkout"]'); + await checkoutPromise; // ✅ Deterministic + await expect(page.locator('.confirmation')).toBeVisible(); +}); +``` + +**After (With Playwright Utils + Auto Error Detection):** +```typescript +import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +// That's it! Just import the fixture - monitoring is automatic +test('checkout completes', async ({ page }) => { + const checkoutPromise = page.waitForResponse( + resp => resp.url().includes('/api/checkout') && resp.ok() + ); + + await page.click('button[name="checkout"]'); + const response = await checkoutPromise; + const order = await response.json(); + + expect(order.status).toBe('confirmed'); + await expect(page.locator('.confirmation')).toBeVisible(); + + // Zero setup - automatically fails if ANY 4xx/5xx occurred + // Error message: "Network errors detected: POST 500 /api/payment" +}); +``` + +**Playwright Utils Benefits:** +- Auto-enabled by fixture import (zero code changes) +- Catches silent backend errors (500, 503, 504) +- Test fails even if UI shows cached/stale success message +- Structured error report in test output +- No manual error checking needed + +**Priority 3: P1 Requirements** +``` +Goal: Get P1 coverage to 80%+ + +Actions: +1. Generate tests for highest-risk P1 gaps +2. Improve test quality incrementally +``` + +#### Create Improvement Roadmap + +```markdown +# Test Improvement Roadmap + +## Week 1: Critical Path (P0) +- [ ] Add 5 missing P0 tests (Epic 1: Auth) +- [ ] Fix 8 hard waits in auth tests +- [ ] Verify P0 coverage = 100% + +## Week 2: Flakiness +- [ ] Replace all hard waits with network-first +- [ ] Fix conditional flow control +- [ ] Run burn-in loops (target: 0 failures in 10 runs) + +## Week 3: High-Value Coverage (P1) +- [ ] Add 10 missing P1 tests +- [ ] Improve selector resilience +- [ ] P1 coverage target: 80% + +## Week 4: Quality Polish +- [ ] Extract fixtures for common patterns +- [ ] Add network assertions +- [ ] Quality score target: 75+ +``` + +### Phase 3: Incremental Improvement + +Apply TEA workflows to new work while improving legacy tests. + +#### For New Features (Greenfield Within Brownfield) + +**Use full TEA workflow:** +``` +1. *test-design (epic-level) - Plan tests for new feature +2. *atdd - Generate failing tests first (TDD) +3. Implement feature +4. *automate - Expand coverage +5. *test-review - Ensure quality +``` + +**Benefits:** +- New code has high-quality tests from day one +- Gradually raises overall quality +- Team learns good patterns + +#### For Bug Fixes (Regression Prevention) + +**Add regression tests:** +``` +1. Reproduce bug with failing test +2. Fix bug +3. Verify test passes +4. Run *test-review on regression test +5. Add to regression test suite +``` + +#### For Refactoring (Regression Safety) + +**Before refactoring:** +``` +1. Run *trace - Baseline coverage +2. Note current coverage % +3. Refactor code +4. Run *trace - Verify coverage maintained +5. No coverage should decrease +``` + +### Phase 4: Continuous Improvement + +Track improvement over time. + +#### Quarterly Quality Audits + +**Q1 Baseline:** +``` +Coverage: 30% +Quality Score: 55/100 +Flakiness: 15% fail rate +``` + +**Q2 Target:** +``` +Coverage: 50% (focus on P0) +Quality Score: 65/100 +Flakiness: 5% +``` + +**Q3 Target:** +``` +Coverage: 70% +Quality Score: 75/100 +Flakiness: 1% +``` + +**Q4 Target:** +``` +Coverage: 85% +Quality Score: 85/100 +Flakiness: <0.5% +``` + +## Brownfield-Specific Tips + +### Don't Rewrite Everything + +**Common mistake:** +``` +"Our tests are bad, let's delete them all and start over!" +``` + +**Better approach:** +``` +"Our tests are bad, let's: +1. Keep tests that work (even if not perfect) +2. Fix critical quality issues incrementally +3. Add tests for gaps +4. Gradually improve over time" +``` + +**Why:** +- Rewriting is risky (might lose coverage) +- Incremental improvement is safer +- Team learns gradually +- Business value delivered continuously + +### Use Regression Hotspots + +**Identify regression-prone areas:** +```markdown +## Regression Hotspots + +**Based on:** +- Bug reports (last 6 months) +- Customer complaints +- Code complexity (cyclomatic complexity >10) +- Frequent changes (git log analysis) + +**High-Risk Areas:** +1. Authentication flow (12 bugs in 6 months) +2. Checkout process (8 bugs) +3. Payment integration (6 bugs) + +**Test Priority:** +- Add regression tests for these areas FIRST +- Ensure P0 coverage before touching code +``` + +### Quarantine Flaky Tests + +Don't let flaky tests block improvement: + +```typescript +// Mark flaky tests with .skip temporarily +test.skip('flaky test - needs fixing', async ({ page }) => { + // TODO: Fix hard wait on line 45 + // TODO: Add network-first pattern +}); +``` + +**Track quarantined tests:** +```markdown +# Quarantined Tests + +| Test | Reason | Owner | Target Fix Date | +|------|--------|-------|----------------| +| checkout.spec.ts:45 | Hard wait causes flakiness | QA Team | 2026-01-20 | +| profile.spec.ts:28 | Conditional flow control | Dev Team | 2026-01-25 | +``` + +**Fix systematically:** +- Don't accumulate quarantined tests +- Set deadlines for fixes +- Review quarantine list weekly + +### Migrate One Directory at a Time + +**Large test suite?** Improve incrementally: + +**Week 1:** `tests/auth/` +``` +1. Run *test-review on auth tests +2. Fix critical issues +3. Re-review +4. Mark directory as "modernized" +``` + +**Week 2:** `tests/api/` +``` +Same process +``` + +**Week 3:** `tests/e2e/` +``` +Same process +``` + +**Benefits:** +- Focused improvement +- Visible progress +- Team learns patterns +- Lower risk + +### Document Migration Status + +**Track which tests are modernized:** + +```markdown +# Test Suite Status + +| Directory | Tests | Quality Score | Status | Notes | +|-----------|-------|---------------|--------|-------| +| tests/auth/ | 15 | 85/100 | ✅ Modernized | Week 1 cleanup | +| tests/api/ | 32 | 78/100 | ⚠️ In Progress | Week 2 | +| tests/e2e/ | 28 | 62/100 | ❌ Legacy | Week 3 planned | +| tests/integration/ | 12 | 45/100 | ❌ Legacy | Week 4 planned | + +**Legend:** +- ✅ Modernized: Quality >80, no critical issues +- ⚠️ In Progress: Active improvement +- ❌ Legacy: Not yet touched +``` + +## Common Brownfield Challenges + +### "We Don't Know What Tests Cover" + +**Problem:** No documentation, unclear what tests do. + +**Solution:** +``` +1. Run *trace - TEA analyzes tests and maps to requirements +2. Review traceability matrix +3. Document findings +4. Use as baseline for improvement +``` + +TEA reverse-engineers test coverage even without documentation. + +### "Tests Are Too Brittle to Touch" + +**Problem:** Afraid to modify tests (might break them). + +**Solution:** +``` +1. Run tests, capture current behavior (baseline) +2. Make small improvement (fix one hard wait) +3. Run tests again +4. If still pass, continue +5. If fail, investigate why + +Incremental changes = lower risk +``` + +### "No One Knows How to Run Tests" + +**Problem:** Test documentation is outdated or missing. + +**Solution:** +``` +1. Document manually or ask TEA to help analyze test structure +2. Create tests/README.md with: + - How to install dependencies + - How to run tests (npx playwright test, npm test, etc.) + - What each test directory contains + - Common issues and troubleshooting +3. Commit documentation for team +``` + +**Note:** `*framework` is for new test setup, not existing tests. For brownfield, document what you have. + +### "Tests Take Hours to Run" + +**Problem:** Full test suite takes 4+ hours. + +**Solution:** +``` +1. Run *ci to add selective testing +2. Run only affected tests on PR +3. Run full suite nightly +4. Parallelize with sharding + +Before: 4 hours sequential +After: 15 minutes with sharding + selective testing +``` + +### "We Have Tests But They Always Fail" + +**Problem:** Tests are so flaky they're ignored. + +**Solution:** +``` +1. Run *test-review to identify flakiness patterns +2. Fix top 5 flaky tests (biggest impact) +3. Quarantine remaining flaky tests +4. Re-enable as you fix them + +Don't let perfect be the enemy of good +``` + +## Brownfield TEA Workflow + +### Recommended Sequence + +**1. Documentation (if needed):** +``` +*document-project +``` + +**2. Baseline (Phase 2):** +``` +*trace Phase 1 - Establish coverage baseline +*test-review - Establish quality baseline +``` + +**3. Planning (Phase 2-3):** +``` +*prd - Document requirements (if missing) +*architecture - Document architecture (if missing) +*test-design (system-level) - Testability review +``` + +**4. Infrastructure (Phase 3):** +``` +*framework - Modernize test framework (if needed) +*ci - Setup or improve CI/CD +``` + +**5. Per Epic (Phase 4):** +``` +*test-design (epic-level) - Focus on regression hotspots +*automate - Add missing tests +*test-review - Ensure quality +*trace Phase 1 - Refresh coverage +``` + +**6. Release Gate:** +``` +*nfr-assess - Validate NFRs (if enterprise) +*trace Phase 2 - Gate decision +``` + +## Success Stories + +### Example: E-Commerce Platform + +**Starting Point:** +- 200 E2E tests, 30% passing, 15-minute flakiness +- No API tests +- No coverage visibility + +**After 3 Months with TEA:** +- 150 E2E tests (removed duplicates), 95% passing, <1% flakiness +- 300 API tests added (faster, more reliable) +- P0 coverage: 100%, P1 coverage: 85% +- Quality score: 82/100 + +**How:** +- Month 1: Baseline + fix top 20 flaky tests +- Month 2: Add API tests for critical path +- Month 3: Improve quality + expand P1 coverage + +### Example: SaaS Application + +**Starting Point:** +- 50 tests, quality score 48/100 +- Hard waits everywhere +- Tests take 45 minutes + +**After 6 Weeks with TEA:** +- 120 tests, quality score 78/100 +- No hard waits (network-first patterns) +- Tests take 8 minutes (parallel execution) + +**How:** +- Week 1-2: Replace hard waits with network-first +- Week 3-4: Add selective testing + CI parallelization +- Week 5-6: Generate tests for gaps with *automate + +## Related Guides + +**Workflow Guides:** +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Baseline coverage analysis +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Quality audit +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Fill coverage gaps +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Risk assessment + +**Customization:** +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Modernize tests with utilities + +## Understanding the Concepts + +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Brownfield model explained +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - What makes tests good +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Fix flakiness +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Prioritize improvements + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - All 8 workflows +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config options +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Testing patterns +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA terminology + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/customization/enable-tea-mcp-enhancements.md b/docs/how-to/customization/enable-tea-mcp-enhancements.md new file mode 100644 index 00000000..0b6bb158 --- /dev/null +++ b/docs/how-to/customization/enable-tea-mcp-enhancements.md @@ -0,0 +1,563 @@ +--- +title: "Enable TEA MCP Enhancements" +description: Configure Playwright MCP servers for live browser verification during TEA workflows +--- + +# Enable TEA MCP Enhancements + +Configure Model Context Protocol (MCP) servers to enable live browser verification, exploratory mode, and recording mode in TEA workflows. + +## What are MCP Enhancements? + +MCP (Model Context Protocol) servers enable AI agents to interact with live browsers during test generation. This allows TEA to: + +- **Explore UIs interactively** - Discover actual functionality through browser automation +- **Verify selectors** - Generate accurate locators from real DOM +- **Validate behavior** - Confirm test scenarios against live applications +- **Debug visually** - Use trace viewer and screenshots during generation + +## When to Use This + +- Want exploratory mode in `*test-design` (browser-based UI discovery) +- Want recording mode in `*atdd` (verify selectors with live browser) +- Want healing mode in `*automate` (fix tests with visual debugging) +- Debugging complex UI issues +- Need accurate selectors from actual DOM + +**Don't use if:** +- You're new to TEA (adds complexity) +- You don't have MCP servers configured +- Your tests work fine without it +- You're testing APIs only (no UI) + +## Prerequisites + +- BMad Method installed +- TEA agent available +- IDE with MCP support (Cursor, VS Code with Claude extension) +- Node.js v18 or later +- Playwright installed + +## Available MCP Servers + +**Two Playwright MCP servers** (actively maintained, continuously updated): + +### 1. Playwright MCP - Browser Automation + +**Command:** `npx @playwright/mcp@latest` + +**Capabilities:** +- Navigate to URLs +- Click elements +- Fill forms +- Take screenshots +- Extract DOM information + +**Best for:** Exploratory mode, recording mode + +### 2. Playwright Test MCP - Test Runner + +**Command:** `npx playwright run-test-mcp-server` + +**Capabilities:** +- Run test files +- Analyze failures +- Extract error messages +- Show trace files + +**Best for:** Healing mode, debugging + +### Recommended: Configure Both + +Both servers work together to provide full TEA MCP capabilities. + +## Installation + +### Step 1: Configure MCP Servers in IDE + +Add this configuration to your IDE's MCP settings. See [TEA Overview](/docs/explanation/features/tea-overview.md#playwright-mcp-enhancements) for IDE-specific configuration locations. + +**MCP Configuration:** + +```json +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": ["@playwright/mcp@latest"] + }, + "playwright-test": { + "command": "npx", + "args": ["playwright", "run-test-mcp-server"] + } + } +} +``` + +### Step 2: Install Playwright Browsers + +```bash +npx playwright install +``` + +### Step 3: Enable in TEA Config + +Edit `_bmad/bmm/config.yaml`: + +```yaml +tea_use_mcp_enhancements: true +``` + +### Step 4: Restart IDE + +Restart your IDE to load MCP server configuration. + +### Step 5: Verify MCP Servers + +Check MCP servers are running: + +**In Cursor:** +- Open command palette (Cmd/Ctrl + Shift + P) +- Search "MCP" +- Should see "Playwright" and "Playwright Test" servers listed + +**In VS Code:** +- Check Claude extension settings +- Verify MCP servers are enabled + +## How MCP Enhances TEA Workflows + +### *test-design: Exploratory Mode + +**Without MCP:** +- TEA infers UI functionality from documentation +- Relies on your description of features +- May miss actual UI behavior + +**With MCP:** +TEA can open live browser to: +``` +"Let me explore the profile page to understand the UI" + +[TEA navigates to /profile] +[Takes screenshot] +[Extracts accessible elements] + +"I see the profile has: +- Name field (editable) +- Email field (editable) +- Avatar upload button +- Save button +- Cancel button + +I'll design tests for these interactions." +``` + +**Benefits:** +- Accurate test design based on actual UI +- Discovers functionality you might not describe +- Validates test scenarios are possible + +### *atdd: Recording Mode + +**Without MCP:** +- TEA generates selectors from best practices +- May use `getByRole()` that doesn't match actual app +- Selectors might need adjustment + +**With MCP:** +TEA verifies selectors with live browser: +``` +"Let me verify the login form selectors" + +[TEA navigates to /login] +[Inspects form fields] + +"I see: +- Email input has label 'Email Address' (not 'Email') +- Password input has label 'Your Password' +- Submit button has text 'Sign In' (not 'Login') + +I'll use these exact selectors." +``` + +**Generated test:** +```typescript +await page.getByLabel('Email Address').fill('test@example.com'); +await page.getByLabel('Your Password').fill('password'); +await page.getByRole('button', { name: 'Sign In' }).click(); +// Selectors verified against actual DOM +``` + +**Benefits:** +- Accurate selectors from real DOM +- Tests work on first run +- No trial-and-error selector debugging + +### *automate: Healing Mode + +**Without MCP:** +- TEA analyzes test code only +- Suggests fixes based on static analysis +- Can't verify fixes work + +**With MCP:** +TEA uses visual debugging: +``` +"This test is failing. Let me debug with trace viewer" + +[TEA opens trace file] +[Analyzes screenshots] +[Identifies selector changed] + +"The button selector changed from 'Save' to 'Save Changes' +I'll update the test and verify it works" + +[TEA makes fix] +[Runs test with MCP] +[Confirms test passes] +``` + +**Benefits:** +- Visual debugging during healing +- Verified fixes (not guesses) +- Faster resolution + +## Usage Examples + +### Exploratory Mode Example + +``` +User: Run *test-design + +TEA: What feature are you designing tests for? + +User: The checkout flow at https://shop.example.com + +TEA: [With MCP] Let me explore the checkout flow +[Opens browser] +[Navigates to checkout] +[Takes screenshots] +[Identifies all interactive elements] + +TEA: I've explored the checkout flow. I found: +- 4-step wizard (Cart → Shipping → Payment → Confirm) +- Shipping options (Standard, Express, Overnight) +- Payment methods (Credit Card, PayPal) +- Promo code field +- Order summary with real-time totals + +Test design will cover all these scenarios. +``` + +### Recording Mode Example + +``` +User: Run *atdd for user registration feature + +TEA: [With MCP] Let me verify the registration form selectors +[Opens browser at /register] +[Inspects form] + +TEA: I see the actual form has: +- "Full Name" label (not "Name") +- "Email Address" label (not "Email") +- "Create Password" label (not "Password") +- "I agree to Terms" checkbox +- "Create Account" button (not "Register") + +Generating tests with verified selectors... +``` + +### Healing Mode Example + +``` +User: Run *automate with healing mode + +TEA: [With MCP] Running tests and checking for failures +[Runs test suite] +[Test fails: selector not found] + +TEA: Test failed on selector '.submit-btn' +[Opens trace viewer] +[Sees button class changed to '.submit-button'] + +Fixing selector and verifying... +[Updates test] +[Re-runs with MCP] +[Test passes] + +Updated test with corrected selector. +``` + +## Configuration Options + +### MCP Server Arguments + +**Playwright MCP with custom port:** +```json +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": ["@playwright/mcp@latest", "--port", "3000"] + } + } +} +``` + +**Playwright Test with specific browser:** +```json +{ + "mcpServers": { + "playwright-test": { + "command": "npx", + "args": ["playwright", "run-test-mcp-server", "--browser", "chromium"] + } + } +} +``` + +### Environment Variables + +```bash +# .env +PLAYWRIGHT_BROWSER=chromium # Browser for MCP +PLAYWRIGHT_HEADLESS=false # Show browser during MCP +PLAYWRIGHT_SLOW_MO=100 # Slow down for visibility +``` + +## Troubleshooting + +### MCP Servers Not Running + +**Problem:** TEA says MCP enhancements aren't available. + +**Causes:** +1. MCP servers not configured in IDE +2. Config syntax error in JSON +3. IDE not restarted after config + +**Solution:** +```bash +# Verify MCP config file exists +ls ~/.cursor/config.json + +# Validate JSON syntax +cat ~/.cursor/config.json | python -m json.tool + +# Restart IDE +# Cmd+Q (quit) then reopen +``` + +### Browser Doesn't Open + +**Problem:** MCP enabled but browser never opens. + +**Causes:** +1. Playwright browsers not installed +2. Headless mode enabled +3. MCP server crashed + +**Solution:** +```bash +# Install browsers +npx playwright install + +# Check MCP server logs (in IDE) +# Look for error messages + +# Try manual MCP server +npx @playwright/mcp@latest +# Should start without errors +``` + +### TEA Doesn't Use MCP + +**Problem:** `tea_use_mcp_enhancements: true` but TEA doesn't use browser. + +**Causes:** +1. Config not saved +2. Workflow run before config update +3. MCP servers not running + +**Solution:** +```bash +# Verify config +grep tea_use_mcp_enhancements _bmad/bmm/config.yaml +# Should show: tea_use_mcp_enhancements: true + +# Restart IDE (reload MCP servers) + +# Start fresh chat (TEA loads config at start) +``` + +### Selector Verification Fails + +**Problem:** MCP can't find elements TEA is looking for. + +**Causes:** +1. Page not fully loaded +2. Element behind modal/overlay +3. Element requires authentication + +**Solution:** +TEA will handle this automatically: +- Wait for page load +- Dismiss modals if present +- Handle auth if needed + +If persistent, provide TEA more context: +``` +"The element is behind a modal - dismiss the modal first" +"The page requires login - use credentials X" +``` + +### MCP Slows Down Workflows + +**Problem:** Workflows take much longer with MCP enabled. + +**Cause:** Browser automation adds overhead. + +**Solution:** +Use MCP selectively: +- **Enable for:** Complex UIs, new projects, debugging +- **Disable for:** Simple features, well-known patterns, API-only testing + +Toggle quickly: +```yaml +# For this feature (complex UI) +tea_use_mcp_enhancements: true + +# For next feature (simple API) +tea_use_mcp_enhancements: false +``` + +## Best Practices + +### Use MCP for Complex UIs + +**Simple UI (skip MCP):** +``` +Standard login form with email/password +TEA can infer selectors without MCP +``` + +**Complex UI (use MCP):** +``` +Multi-step wizard with dynamic fields +Conditional UI elements +Third-party components +Custom form widgets +``` + +### Start Without MCP, Enable When Needed + +**Learning path:** +1. Week 1-2: TEA without MCP (learn basics) +2. Week 3: Enable MCP (explore advanced features) +3. Week 4+: Use MCP selectively (when it adds value) + +### Combine with Playwright Utils + +**Powerful combination:** +```yaml +tea_use_playwright_utils: true +tea_use_mcp_enhancements: true +``` + +**Benefits:** +- Playwright Utils provides production-ready utilities +- MCP verifies utilities work with actual app +- Best of both worlds + +### Use for Test Healing + +**Scenario:** Test suite has 50 failing tests after UI update. + +**With MCP:** +``` +*automate (healing mode) + +TEA: +1. Opens trace viewer for each failure +2. Identifies changed selectors +3. Updates tests with corrected selectors +4. Verifies fixes with browser +5. Provides updated tests + +Result: 45/50 tests auto-healed +``` + +### Use for New Team Members + +**Onboarding:** +``` +New developer: "I don't know this codebase's UI" + +Senior: "Run *test-design with MCP exploratory mode" + +TEA explores UI and generates documentation: +- UI structure discovered +- Interactive elements mapped +- Test design created automatically +``` + +## Security Considerations + +### MCP Servers Have Browser Access + +**What MCP can do:** +- Navigate to any URL +- Click any element +- Fill any form +- Access browser storage +- Read page content + +**Best practices:** +- Only configure MCP in trusted environments +- Don't use MCP on production sites (use staging/dev) +- Review generated tests before running on production +- Keep MCP config in local files (not committed) + +### Protect Credentials + +**Don't:** +``` +"TEA, login with mypassword123" +# Password visible in chat history +``` + +**Do:** +``` +"TEA, login using credentials from .env" +# Password loaded from environment, not in chat +``` + +## Related Guides + +**Getting Started:** +- [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md) - Learn TEA basics first + +**Workflow Guides (MCP-Enhanced):** +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Exploratory mode with browser +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Recording mode for accurate selectors +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Healing mode for debugging + +**Other Customization:** +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Production-ready utilities + +## Understanding the Concepts + +- [TEA Overview](/docs/explanation/features/tea-overview.md) - MCP enhancements in lifecycle +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - When to use MCP enhancements + +## Reference + +- [TEA Configuration](/docs/reference/tea/configuration.md) - tea_use_mcp_enhancements option +- [TEA Command Reference](/docs/reference/tea/commands.md) - MCP-enhanced workflows +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - MCP Enhancements term + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/customization/integrate-playwright-utils.md b/docs/how-to/customization/integrate-playwright-utils.md new file mode 100644 index 00000000..b2c996fb --- /dev/null +++ b/docs/how-to/customization/integrate-playwright-utils.md @@ -0,0 +1,771 @@ +--- +title: "Integrate Playwright Utils with TEA" +description: Add production-ready fixtures and utilities to your TEA-generated tests +--- + +# Integrate Playwright Utils with TEA + +Integrate `@seontechnologies/playwright-utils` with TEA to get production-ready fixtures, utilities, and patterns in your test suite. + +## What is Playwright Utils? + +A production-ready utility library that provides: +- Typed API request helper +- Authentication session management +- Network recording and replay (HAR) +- Network request interception +- Async polling (recurse) +- Structured logging +- File validation (CSV, PDF, XLSX, ZIP) +- Burn-in testing utilities +- Network error monitoring + +**Repository:** https://github.com/seontechnologies/playwright-utils + +**npm Package:** `@seontechnologies/playwright-utils` + +## When to Use This + +- You want production-ready fixtures (not DIY) +- Your team benefits from standardized patterns +- You need utilities like API testing, auth handling, network mocking +- You want TEA to generate tests using these utilities +- You're building reusable test infrastructure + +**Don't use if:** +- You're just learning testing (keep it simple first) +- You have your own fixture library +- You don't need the utilities + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Test framework setup complete (Playwright) +- Node.js v18 or later + +**Note:** Playwright Utils is for Playwright only (not Cypress). + +## Installation + +### Step 1: Install Package + +```bash +npm install -D @seontechnologies/playwright-utils +``` + +### Step 2: Enable in TEA Config + +Edit `_bmad/bmm/config.yaml`: + +```yaml +tea_use_playwright_utils: true +``` + +**Note:** If you enabled this during installation (`npx bmad-method@alpha install`), it's already set. + +### Step 3: Verify Installation + +```bash +# Check package installed +npm list @seontechnologies/playwright-utils + +# Check TEA config +grep tea_use_playwright_utils _bmad/bmm/config.yaml +``` + +Should show: +``` +@seontechnologies/playwright-utils@2.x.x +tea_use_playwright_utils: true +``` + +## What Changes When Enabled + +### *framework Workflow + +**Vanilla Playwright:** +```typescript +// Basic Playwright fixtures only +import { test, expect } from '@playwright/test'; + +test('api test', async ({ request }) => { + const response = await request.get('/api/users'); + const users = await response.json(); + expect(response.status()).toBe(200); +}); +``` + +**With Playwright Utils (Combined Fixtures):** +```typescript +// All utilities available via single import +import { test } from '@seontechnologies/playwright-utils/fixtures'; +import { expect } from '@playwright/test'; + +test('api test', async ({ apiRequest, authToken, log }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users', + headers: { Authorization: `Bearer ${authToken}` } + }); + + log.info('Fetched users', body); + expect(status).toBe(200); +}); +``` + +**With Playwright Utils (Selective Merge):** +```typescript +import { mergeTests } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { test as logFixture } from '@seontechnologies/playwright-utils/log/fixtures'; + +export const test = mergeTests(apiRequestFixture, logFixture); +export { expect } from '@playwright/test'; + +test('api test', async ({ apiRequest, log }) => { + log.info('Fetching users'); + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users' + }); + expect(status).toBe(200); +}); +``` + +### *atdd and *automate Workflows + +**Without Playwright Utils:** +```typescript +// Manual API calls +test('should fetch profile', async ({ page, request }) => { + const response = await request.get('/api/profile'); + const profile = await response.json(); + // Manual parsing and validation +}); +``` + +**With Playwright Utils:** +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +test('should fetch profile', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/profile' // 'path' not 'url' + }).validateSchema(ProfileSchema); // Chained validation + + expect(status).toBe(200); + // body is type-safe: { id: string, name: string, email: string } +}); +``` + +### *test-review Workflow + +**Without Playwright Utils:** +Reviews against generic Playwright patterns + +**With Playwright Utils:** +Reviews against playwright-utils best practices: +- Fixture composition patterns +- Utility usage (apiRequest, authSession, etc.) +- Network-first patterns +- Structured logging + +### *ci Workflow + +**Without Playwright Utils:** +Basic CI configuration + +**With Playwright Utils:** +Enhanced CI with: +- Burn-in utility for smart test selection +- Selective testing based on git diff +- Test prioritization + +## Available Utilities + +### api-request + +Typed HTTP client with schema validation. + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { expect } from '@playwright/test'; +import { z } from 'zod'; + +const UserSchema = z.object({ + id: z.string(), + name: z.string(), + email: z.string().email() +}); + +test('should create user', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/users', // Note: 'path' not 'url' + body: { name: 'Test User', email: 'test@example.com' } // Note: 'body' not 'data' + }).validateSchema(UserSchema); // Note: chained method + + expect(status).toBe(201); + expect(body.id).toBeDefined(); + expect(body.email).toBe('test@example.com'); +}); +``` + +**Benefits:** +- Returns `{ status, body }` structure +- Schema validation with `.validateSchema()` chained method +- Automatic retry for 5xx errors +- Type-safe response body + +### auth-session + +Authentication session management with token persistence. + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/auth-session/fixtures'; +import { expect } from '@playwright/test'; + +test('should access protected route', async ({ page, authToken }) => { + // authToken automatically fetched and persisted + // No manual login needed - handled by fixture + + await page.goto('/dashboard'); + await expect(page).toHaveURL('/dashboard'); + + // Token is reused across tests (persisted to disk) +}); +``` + +**Configuration required** (see auth-session docs for provider setup): +```typescript +// global-setup.ts +import { authStorageInit, setAuthProvider, authGlobalInit } from '@seontechnologies/playwright-utils/auth-session'; + +async function globalSetup() { + authStorageInit(); + setAuthProvider(myCustomProvider); // Define your auth mechanism + await authGlobalInit(); // Fetch token once +} +``` + +**Benefits:** +- Token fetched once, reused across all tests +- Persisted to disk (faster subsequent runs) +- Multi-user support via `authOptions.userIdentifier` +- Automatic token renewal if expired + +### network-recorder + +Record and replay network traffic (HAR) for offline testing. + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/network-recorder/fixtures'; + +// Record mode: Set environment variable +process.env.PW_NET_MODE = 'record'; + +test('should work with recorded traffic', async ({ page, context, networkRecorder }) => { + // Setup recorder (records or replays based on PW_NET_MODE) + await networkRecorder.setup(context); + + // Your normal test code + await page.goto('/dashboard'); + await page.click('#add-item'); + + // First run (record): Saves traffic to HAR file + // Subsequent runs (playback): Uses HAR file, no backend needed +}); +``` + +**Switch modes:** +```bash +# Record traffic +PW_NET_MODE=record npx playwright test + +# Playback traffic (offline) +PW_NET_MODE=playback npx playwright test +``` + +**Benefits:** +- Offline testing (no backend needed) +- Deterministic responses (same every time) +- Faster execution (no network latency) +- Stateful mocking (CRUD operations work) + +### intercept-network-call + +Spy or stub network requests with automatic JSON parsing. + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/fixtures'; + +test('should handle API errors', async ({ page, interceptNetworkCall }) => { + // Stub API to return error (set up BEFORE navigation) + const profileCall = interceptNetworkCall({ + method: 'GET', + url: '**/api/profile', + fulfillResponse: { + status: 500, + body: { error: 'Server error' } + } + }); + + await page.goto('/profile'); + + // Wait for the intercepted response + const { status, responseJson } = await profileCall; + + expect(status).toBe(500); + expect(responseJson.error).toBe('Server error'); + await expect(page.getByText('Server error occurred')).toBeVisible(); +}); +``` + +**Benefits:** +- Automatic JSON parsing (`responseJson` ready to use) +- Spy mode (observe real traffic) or stub mode (mock responses) +- Glob pattern URL matching +- Returns promise with `{ status, responseJson, requestJson }` + +### recurse + +Async polling for eventual consistency (Cypress-style). + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/fixtures'; + +test('should wait for async job completion', async ({ apiRequest, recurse }) => { + // Start async job + const { body: job } = await apiRequest({ + method: 'POST', + path: '/api/jobs' + }); + + // Poll until complete (smart waiting) + const completed = await recurse( + () => apiRequest({ method: 'GET', path: `/api/jobs/${job.id}` }), + (result) => result.body.status === 'completed', + { + timeout: 30000, + interval: 2000, + log: 'Waiting for job to complete' + } + }); + + expect(completed.body.status).toBe('completed'); +}); +``` + +**Benefits:** +- Smart polling with configurable interval +- Handles async jobs, background tasks +- Optional logging for debugging +- Better than hard waits or manual polling loops + +### log + +Structured logging that integrates with Playwright reports. + +**Usage:** +```typescript +import { log } from '@seontechnologies/playwright-utils'; +import { test, expect } from '@playwright/test'; + +test('should login', async ({ page }) => { + await log.info('Starting login test'); + + await page.goto('/login'); + await log.step('Navigated to login page'); // Shows in Playwright UI + + await page.getByLabel('Email').fill('test@example.com'); + await log.debug('Filled email field'); + + await log.success('Login completed'); + // Logs appear in test output and Playwright reports +}); +``` + +**Benefits:** +- Direct import (no fixture needed for basic usage) +- Structured logs in test reports +- `.step()` shows in Playwright UI +- Supports object logging with `.debug()` +- Trace test execution + +### file-utils + +Read and validate CSV, PDF, XLSX, ZIP files. + +**Usage:** +```typescript +import { handleDownload, readCSV } from '@seontechnologies/playwright-utils/file-utils'; +import { expect } from '@playwright/test'; +import path from 'node:path'; + +const DOWNLOAD_DIR = path.join(__dirname, '../downloads'); + +test('should export valid CSV', async ({ page }) => { + // Handle download and get file path + const downloadPath = await handleDownload({ + page, + downloadDir: DOWNLOAD_DIR, + trigger: () => page.click('button:has-text("Export")') + }); + + // Read and parse CSV + const csvResult = await readCSV({ filePath: downloadPath }); + const { data, headers } = csvResult.content; + + // Validate structure + expect(headers).toEqual(['Name', 'Email', 'Status']); + expect(data.length).toBeGreaterThan(0); + expect(data[0]).toMatchObject({ + Name: expect.any(String), + Email: expect.any(String), + Status: expect.any(String) + }); +}); +``` + +**Benefits:** +- Handles downloads automatically +- Auto-parses CSV, XLSX, PDF, ZIP +- Type-safe access to parsed data +- Returns structured `{ headers, data }` + +### burn-in + +Smart test selection with git diff analysis for CI optimization. + +**Usage:** +```typescript +// scripts/burn-in-changed.ts +import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'; + +async function main() { + await runBurnIn({ + configPath: 'playwright.burn-in.config.ts', + baseBranch: 'main' + }); +} + +main().catch(console.error); +``` + +**Config:** +```typescript +// playwright.burn-in.config.ts +import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'; + +const config: BurnInConfig = { + skipBurnInPatterns: [ + '**/config/**', + '**/*.md', + '**/*types*' + ], + burnInTestPercentage: 0.3, + burnIn: { + repeatEach: 3, + retries: 1 + } +}; + +export default config; +``` + +**Package script:** +```json +{ + "scripts": { + "test:burn-in": "tsx scripts/burn-in-changed.ts" + } +} +``` + +**Benefits:** +- Smart filtering (skip config, types, docs changes) +- Volume control (run percentage of affected tests) +- Git diff-based test selection +- Faster CI feedback + +### network-error-monitor + +Automatically detect HTTP 4xx/5xx errors during tests. + +**Usage:** +```typescript +import { test } from '@seontechnologies/playwright-utils/network-error-monitor/fixtures'; + +// That's it! Network monitoring is automatically enabled +test('should not have API errors', async ({ page }) => { + await page.goto('/dashboard'); + await page.click('button'); + + // Test fails automatically if any HTTP 4xx/5xx errors occur + // Error message shows: "Network errors detected: 2 request(s) failed" + // GET 500 https://api.example.com/users + // POST 503 https://api.example.com/metrics +}); +``` + +**Opt-out for validation tests:** +```typescript +// When testing error scenarios, opt-out with annotation +test('should show error message on 404', + { annotation: [{ type: 'skipNetworkMonitoring' }] }, // Array format + async ({ page }) => { + await page.goto('/invalid-page'); // Will 404 + await expect(page.getByText('Page not found')).toBeVisible(); + // Test won't fail on 404 because of annotation + } +); + +// Or opt-out entire describe block +test.describe('error handling', + { annotation: [{ type: 'skipNetworkMonitoring' }] }, + () => { + test('handles 404', async ({ page }) => { + // Monitoring disabled for all tests in block + }); + } +); +``` + +**Benefits:** +- Auto-enabled (zero setup) +- Catches silent backend failures +- Opt-out with annotations +- Structured error reporting + +## Fixture Composition + +Combine utilities using `mergeTests`: + +**Option 1: Use Combined Fixtures (Simplest)** +```typescript +// Import all utilities at once +import { test } from '@seontechnologies/playwright-utils/fixtures'; +import { log } from '@seontechnologies/playwright-utils'; +import { expect } from '@playwright/test'; + +test('full test', async ({ apiRequest, authToken, interceptNetworkCall }) => { + await log.info('Starting test'); // log is direct import + + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/data', + headers: { Authorization: `Bearer ${authToken}` } + }); + + await log.info('Data fetched', body); + expect(status).toBe(200); +}); +``` + +**Note:** `log` is imported directly (not a fixture). `authToken` requires auth-session provider setup. + +**Option 2: Merge Individual Fixtures (Selective)** +```typescript +import { test as base } from '@playwright/test'; +import { mergeTests } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { test as recurseFixture } from '@seontechnologies/playwright-utils/recurse/fixtures'; +import { log } from '@seontechnologies/playwright-utils'; + +// Merge only the fixtures you need +export const test = mergeTests( + apiRequestFixture, + recurseFixture +); + +export { expect } from '@playwright/test'; + +// Use merged utilities in tests +test('selective test', async ({ apiRequest, recurse }) => { + await log.info('Starting test'); // log is direct import, not fixture + + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/data' + }); + + await log.info('Data fetched', body); + expect(status).toBe(200); +}); +``` + +**Note:** `log` is a direct utility (not a fixture), so import it separately. + +**Recommended:** Use Option 1 (combined fixtures) unless you need fine control over which utilities are included. + +## Configuration + +### Environment Variables + +```bash +# .env +PLAYWRIGHT_UTILS_LOG_LEVEL=debug # debug | info | warn | error +PLAYWRIGHT_UTILS_RETRY_ATTEMPTS=3 +PLAYWRIGHT_UTILS_TIMEOUT=30000 +``` + +### Playwright Config + +```typescript +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + use: { + // Playwright Utils works with standard Playwright config + baseURL: process.env.BASE_URL || 'http://localhost:3000', + extraHTTPHeaders: { + // Add headers used by utilities + } + } +}); +``` + +## Troubleshooting + +### Import Errors + +**Problem:** Cannot find module '@seontechnologies/playwright-utils/api-request' + +**Solution:** +```bash +# Verify package installed +npm list @seontechnologies/playwright-utils + +# Check package.json has correct version +"@seontechnologies/playwright-utils": "^2.0.0" + +# Reinstall if needed +npm install -D @seontechnologies/playwright-utils +``` + +### TEA Not Using Utilities + +**Problem:** TEA generates tests without playwright-utils. + +**Causes:** +1. Config not set: `tea_use_playwright_utils: false` +2. Workflow run before config change +3. Package not installed + +**Solution:** +```bash +# Check config +grep tea_use_playwright_utils _bmad/bmm/config.yaml + +# Should show: tea_use_playwright_utils: true + +# Start fresh chat (TEA loads config at start) +``` + +### Type Errors with apiRequest + +**Problem:** TypeScript errors on apiRequest response. + +**Cause:** No schema validation. + +**Solution:** +```typescript +// Add Zod schema for type safety +import { z } from 'zod'; + +const ProfileSchema = z.object({ + id: z.string(), + name: z.string(), + email: z.string().email() +}); + +const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/profile' // 'path' not 'url' +}).validateSchema(ProfileSchema); // Chained method + +expect(status).toBe(200); +// body is typed as { id: string, name: string, email: string } +``` + +## Migration Guide + +### Migrating Existing Tests + +**Before (Vanilla Playwright):** +```typescript +test('should access protected route', async ({ page, request }) => { + // Manual auth token fetch + const response = await request.post('/api/auth/login', { + data: { email: 'test@example.com', password: 'pass' } + }); + const { token } = await response.json(); + + // Manual token storage + await page.goto('/dashboard'); + await page.evaluate((token) => { + localStorage.setItem('authToken', token); + }, token); + + await expect(page).toHaveURL('/dashboard'); +}); +``` + +**After (With Playwright Utils):** +```typescript +import { test } from '@seontechnologies/playwright-utils/auth-session/fixtures'; + +test('should access protected route', async ({ page, authToken }) => { + // authToken automatically fetched and persisted by fixture + await page.goto('/dashboard'); + + // Token is already in place (no manual storage needed) + await expect(page).toHaveURL('/dashboard'); +}); +``` + +**Benefits:** +- Token fetched once, reused across all tests (persisted to disk) +- No manual token storage or management +- Automatic token renewal if expired +- Multi-user support via `authOptions.userIdentifier` +- 10 lines → 5 lines (less code) + +## Related Guides + +**Getting Started:** +- [TEA Lite Quickstart Tutorial](/docs/tutorials/getting-started/tea-lite-quickstart.md) - Learn TEA basics +- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - Initial framework setup + +**Workflow Guides:** +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate tests with utilities +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand coverage with utilities +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Review against PW-Utils patterns + +**Other Customization:** +- [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) - Live browser verification + +## Understanding the Concepts + +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Pure function → fixture pattern +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Network utilities explained +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Patterns PW-Utils enforces + +## Reference + +- [TEA Configuration](/docs/reference/tea/configuration.md) - tea_use_playwright_utils option +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Playwright Utils fragments +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - Playwright Utils term +- [Official PW-Utils Docs](https://seontechnologies.github.io/playwright-utils/) - Complete API reference + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-atdd.md b/docs/how-to/workflows/run-atdd.md new file mode 100644 index 00000000..9806078d --- /dev/null +++ b/docs/how-to/workflows/run-atdd.md @@ -0,0 +1,499 @@ +--- +title: "How to Run ATDD with TEA" +description: Generate failing acceptance tests before implementation using TEA's ATDD workflow +--- + +# How to Run ATDD with TEA + +Use TEA's `*atdd` workflow to generate failing acceptance tests BEFORE implementation. This is the TDD (Test-Driven Development) red phase - tests fail first, guide development, then pass. + +## When to Use This + +- You're about to implement a NEW feature (feature doesn't exist yet) +- You want to follow TDD workflow (red → green → refactor) +- You want tests to guide your implementation +- You're practicing acceptance test-driven development + +**Don't use this if:** +- Feature already exists (use `*automate` instead) +- You want tests that pass immediately + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Test framework setup complete (run `*framework` if needed) +- Story or feature defined with acceptance criteria + +**Note:** This guide uses Playwright examples. If using Cypress, commands and syntax will differ (e.g., `cy.get()` instead of `page.locator()`). + +## Steps + +### 1. Load TEA Agent + +Start a fresh chat and load TEA: + +``` +*tea +``` + +### 2. Run the ATDD Workflow + +``` +*atdd +``` + +### 3. Provide Context + +TEA will ask for: + +**Story/Feature Details:** +``` +We're adding a user profile page where users can: +- View their profile information +- Edit their name and email +- Upload a profile picture +- Save changes with validation +``` + +**Acceptance Criteria:** +``` +Given I'm logged in +When I navigate to /profile +Then I see my current name and email + +Given I'm on the profile page +When I click "Edit Profile" +Then I can modify my name and email + +Given I've edited my profile +When I click "Save" +Then my changes are persisted +And I see a success message + +Given I upload an invalid file type +When I try to save +Then I see an error message +And changes are not saved +``` + +**Reference Documents** (optional): +- Point to your story file +- Reference PRD or tech spec +- Link to test design (if you ran `*test-design` first) + +### 4. Specify Test Levels + +TEA will ask what test levels to generate: + +**Options:** +- E2E tests (browser-based, full user journey) +- API tests (backend only, faster) +- Component tests (UI components in isolation) +- Mix of levels (recommended) + +### Component Testing by Framework + +TEA generates component tests using framework-appropriate tools: + +| Your Framework | Component Testing Tool | +|----------------|----------------------| +| **Cypress** | Cypress Component Testing (*.cy.tsx) | +| **Playwright** | Vitest + React Testing Library (*.test.tsx) | + +**Example response:** +``` +Generate: +- API tests for profile CRUD operations +- E2E tests for the complete profile editing flow +- Component tests for ProfileForm validation (if using Cypress or Vitest) +- Focus on P0 and P1 scenarios +``` + +### 5. Review Generated Tests + +TEA generates **failing tests** in appropriate directories: + +#### API Tests (`tests/api/profile.spec.ts`): + +**Vanilla Playwright:** +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('Profile API', () => { + test('should fetch user profile', async ({ request }) => { + const response = await request.get('/api/profile'); + + expect(response.status()).toBe(200); + const profile = await response.json(); + expect(profile).toHaveProperty('name'); + expect(profile).toHaveProperty('email'); + expect(profile).toHaveProperty('avatarUrl'); + }); + + test('should update user profile', async ({ request }) => { + const response = await request.patch('/api/profile', { + data: { + name: 'Updated Name', + email: 'updated@example.com' + } + }); + + expect(response.status()).toBe(200); + const updated = await response.json(); + expect(updated.name).toBe('Updated Name'); + expect(updated.email).toBe('updated@example.com'); + }); + + test('should validate email format', async ({ request }) => { + const response = await request.patch('/api/profile', { + data: { + email: 'invalid-email' + } + }); + + expect(response.status()).toBe(400); + const error = await response.json(); + expect(error.message).toContain('Invalid email format'); + }); +}); +``` + +**With Playwright Utils:** +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { expect } from '@playwright/test'; +import { z } from 'zod'; + +const ProfileSchema = z.object({ + name: z.string(), + email: z.string().email(), + avatarUrl: z.string().url() +}); + +test.describe('Profile API', () => { + test('should fetch user profile', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/profile' + }).validateSchema(ProfileSchema); // Chained validation + + expect(status).toBe(200); + // Schema already validated, type-safe access + expect(body.name).toBeDefined(); + expect(body.email).toContain('@'); + }); + + test('should update user profile', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { // 'body' not 'data' + name: 'Updated Name', + email: 'updated@example.com' + } + }).validateSchema(ProfileSchema); // Chained validation + + expect(status).toBe(200); + expect(body.name).toBe('Updated Name'); + expect(body.email).toBe('updated@example.com'); + }); + + test('should validate email format', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { email: 'invalid-email' } // 'body' not 'data' + }); + + expect(status).toBe(400); + expect(body.message).toContain('Invalid email format'); + }); +}); +``` + +**Key Benefits:** +- Returns `{ status, body }` (cleaner than `response.status()` + `await response.json()`) +- Automatic schema validation with Zod +- Type-safe response bodies +- Automatic retry for 5xx errors +- Less boilerplate + +#### E2E Tests (`tests/e2e/profile.spec.ts`): + +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('Profile Page', () => { + test.beforeEach(async ({ page }) => { + // Login first + await page.goto('/login'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password').fill('password123'); + await page.getByRole('button', { name: 'Sign in' }).click(); + }); + + test('should display current profile information', async ({ page }) => { + await page.goto('/profile'); + + await expect(page.getByText('test@example.com')).toBeVisible(); + await expect(page.getByText('Test User')).toBeVisible(); + }); + + test('should edit and save profile', async ({ page }) => { + await page.goto('/profile'); + + // Click edit + await page.getByRole('button', { name: 'Edit Profile' }).click(); + + // Modify fields + await page.getByLabel('Name').fill('Updated Name'); + await page.getByLabel('Email').fill('updated@example.com'); + + // Save + await page.getByRole('button', { name: 'Save' }).click(); + + // Verify success + await expect(page.getByText('Profile updated successfully')).toBeVisible(); + await expect(page.getByText('Updated Name')).toBeVisible(); + }); + + test('should show validation error for invalid email', async ({ page }) => { + await page.goto('/profile'); + await page.getByRole('button', { name: 'Edit Profile' }).click(); + + await page.getByLabel('Email').fill('invalid-email'); + await page.getByRole('button', { name: 'Save' }).click(); + + await expect(page.getByText('Invalid email format')).toBeVisible(); + }); +}); +``` + +#### Implementation Checklist + +TEA also provides an implementation checklist: + +```markdown +## Implementation Checklist + +### Backend +- [ ] Create `GET /api/profile` endpoint +- [ ] Create `PATCH /api/profile` endpoint +- [ ] Add email validation middleware +- [ ] Add profile picture upload handling +- [ ] Write API unit tests + +### Frontend +- [ ] Create ProfilePage component +- [ ] Implement profile form with validation +- [ ] Add file upload for avatar +- [ ] Handle API errors gracefully +- [ ] Add loading states + +### Tests +- [x] API tests generated (failing) +- [x] E2E tests generated (failing) +- [ ] Run tests after implementation (should pass) +``` + +### 6. Verify Tests Fail + +This is the TDD red phase - tests MUST fail before implementation. + +**For Playwright:** +```bash +npx playwright test +``` + +**For Cypress:** +```bash +npx cypress run +``` + +Expected output: +``` +Running 6 tests using 1 worker + + ✗ tests/api/profile.spec.ts:3:3 › should fetch user profile + Error: expect(received).toBe(expected) + Expected: 200 + Received: 404 + + ✗ tests/e2e/profile.spec.ts:10:3 › should display current profile information + Error: page.goto: net::ERR_ABORTED +``` + +**All tests should fail!** This confirms: +- Feature doesn't exist yet +- Tests will guide implementation +- You have clear success criteria + +### 7. Implement the Feature + +Now implement the feature following the test guidance: + +1. Start with API tests (backend first) +2. Make API tests pass +3. Move to E2E tests (frontend) +4. Make E2E tests pass +5. Refactor with confidence (tests protect you) + +### 8. Verify Tests Pass + +After implementation, run your test suite. + +**For Playwright:** +```bash +npx playwright test +``` + +**For Cypress:** +```bash +npx cypress run +``` + +Expected output: +``` +Running 6 tests using 1 worker + + ✓ tests/api/profile.spec.ts:3:3 › should fetch user profile (850ms) + ✓ tests/api/profile.spec.ts:15:3 › should update user profile (1.2s) + ✓ tests/api/profile.spec.ts:30:3 › should validate email format (650ms) + ✓ tests/e2e/profile.spec.ts:10:3 › should display current profile (2.1s) + ✓ tests/e2e/profile.spec.ts:18:3 › should edit and save profile (3.2s) + ✓ tests/e2e/profile.spec.ts:35:3 › should show validation error (1.8s) + + 6 passed (9.8s) +``` + +**Green!** You've completed the TDD cycle: red → green → refactor. + +## What You Get + +### Failing Tests +- API tests for backend endpoints +- E2E tests for user workflows +- Component tests (if requested) +- All tests fail initially (red phase) + +### Implementation Guidance +- Clear checklist of what to build +- Acceptance criteria translated to assertions +- Edge cases and error scenarios identified + +### TDD Workflow Support +- Tests guide implementation +- Confidence to refactor +- Living documentation of features + +## Tips + +### Start with Test Design + +Run `*test-design` before `*atdd` for better results: + +``` +*test-design # Risk assessment and priorities +*atdd # Generate tests based on design +``` + +### Use Recording Mode (Optional) + +If MCP enhancements are enabled (`tea_use_mcp_enhancements: true` in config): + +When prompted during `*atdd`, select "recording mode" to: +- Verify selectors against actual UI with live browser +- Capture network requests in real-time +- Generate accurate locators from actual DOM + +See [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) + +### Focus on P0/P1 Scenarios + +Don't generate tests for everything at once: + +``` +Generate tests for: +- P0: Critical path (happy path) +- P1: High value (validation, errors) + +Skip P2/P3 for now - add later with *automate +``` + +### API Tests First, E2E Later + +Recommended order: +1. Generate API tests with `*atdd` +2. Implement backend (make API tests pass) +3. Generate E2E tests with `*atdd` (or `*automate`) +4. Implement frontend (make E2E tests pass) + +This "outside-in" approach is faster and more reliable. + +### Keep Tests Deterministic + +TEA generates deterministic tests by default: +- No hard waits (`waitForTimeout`) +- Network-first patterns (wait for responses) +- Explicit assertions (no conditionals) + +Don't modify these patterns - they prevent flakiness! + +## Common Issues + +### Tests Don't Fail Initially + +**Problem:** Tests pass on first run but feature doesn't exist. + +**Cause:** Tests are hitting wrong endpoints or checking wrong things. + +**Solution:** Review generated tests - ensure they match your feature requirements. + +### Too Many Tests Generated + +**Problem:** TEA generated 50 tests for a simple feature. + +**Cause:** Didn't specify priorities or scope. + +**Solution:** Be specific: +``` +Generate ONLY: +- P0 scenarios (2-3 tests) +- Happy path for API +- One E2E test for full flow +``` + +### Selectors Are Fragile + +**Problem:** E2E tests use brittle selectors (CSS, XPath). + +**Solution:** Use MCP recording mode or specify accessible selectors: +``` +Use accessible locators: +- getByRole() +- getByLabel() +- getByText() +Avoid CSS selectors +``` + +## Related Guides + +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Plan before generating +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Tests for existing features +- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - Initial setup + +## Understanding the Concepts + +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Why P0 vs P3 matters +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - What makes tests good +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Avoiding flakiness + +## Reference + +- [Command: *atdd](/docs/reference/tea/commands.md#atdd) - Full command reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - MCP and Playwright Utils options + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-automate.md b/docs/how-to/workflows/run-automate.md new file mode 100644 index 00000000..444a6182 --- /dev/null +++ b/docs/how-to/workflows/run-automate.md @@ -0,0 +1,700 @@ +--- +title: "How to Run Automate with TEA" +description: Expand test automation coverage after implementation using TEA's automate workflow +--- + +# How to Run Automate with TEA + +Use TEA's `*automate` workflow to generate comprehensive tests for existing features. Unlike `*atdd`, these tests pass immediately because the feature already exists. + +## When to Use This + +- Feature already exists and works +- Want to add test coverage to existing code +- Need tests that pass immediately +- Expanding existing test suite +- Adding tests to legacy code + +**Don't use this if:** +- Feature doesn't exist yet (use `*atdd` instead) +- Want failing tests to guide development (use `*atdd` for TDD) + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Test framework setup complete (run `*framework` if needed) +- Feature implemented and working + +**Note:** This guide uses Playwright examples. If using Cypress, commands and syntax will differ. + +## Steps + +### 1. Load TEA Agent + +Start a fresh chat and load TEA: + +``` +*tea +``` + +### 2. Run the Automate Workflow + +``` +*automate +``` + +### 3. Provide Context + +TEA will ask for context about what you're testing. + +#### Option A: BMad-Integrated Mode (Recommended) + +If you have BMad artifacts (stories, test designs, PRDs): + +**What are you testing?** +``` +I'm testing the user profile feature we just implemented. +Story: story-profile-management.md +Test Design: test-design-epic-1.md +``` + +**Reference documents:** +- Story file with acceptance criteria +- Test design document (if available) +- PRD sections relevant to this feature +- Tech spec (if available) + +**Existing tests:** +``` +We have basic tests in tests/e2e/profile-view.spec.ts +Avoid duplicating that coverage +``` + +TEA will analyze your artifacts and generate comprehensive tests that: +- Cover acceptance criteria from the story +- Follow priorities from test design (P0 → P1 → P2) +- Avoid duplicating existing tests +- Include edge cases and error scenarios + +#### Option B: Standalone Mode + +If you're using TEA Solo or don't have BMad artifacts: + +**What are you testing?** +``` +TodoMVC React application at https://todomvc.com/examples/react/ +Features: Create todos, mark as complete, filter by status, delete todos +``` + +**Specific scenarios to cover:** +``` +- Creating todos (happy path) +- Marking todos as complete/incomplete +- Filtering (All, Active, Completed) +- Deleting todos +- Edge cases (empty input, long text) +``` + +TEA will analyze the application and generate tests based on your description. + +### 4. Specify Test Levels + +TEA will ask which test levels to generate: + +**Options:** +- **E2E tests** - Full browser-based user workflows +- **API tests** - Backend endpoint testing (faster, more reliable) +- **Component tests** - UI component testing in isolation (framework-dependent) +- **Mix** - Combination of levels (recommended) + +**Example response:** +``` +Generate: +- API tests for all CRUD operations +- E2E tests for critical user workflows (P0) +- Focus on P0 and P1 scenarios +- Skip P3 (low priority edge cases) +``` + +### 5. Review Generated Tests + +TEA generates a comprehensive test suite with multiple test levels. + +#### API Tests (`tests/api/profile.spec.ts`): + +**Vanilla Playwright:** +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('Profile API', () => { + let authToken: string; + + test.beforeAll(async ({ request }) => { + // Manual auth token fetch + const response = await request.post('/api/auth/login', { + data: { email: 'test@example.com', password: 'password123' } + }); + const { token } = await response.json(); + authToken = token; + }); + + test('should fetch user profile', async ({ request }) => { + const response = await request.get('/api/profile', { + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(response.ok()).toBeTruthy(); + const profile = await response.json(); + expect(profile).toMatchObject({ + id: expect.any(String), + name: expect.any(String), + email: expect.any(String) + }); + }); + + test('should update profile successfully', async ({ request }) => { + const response = await request.patch('/api/profile', { + headers: { Authorization: `Bearer ${authToken}` }, + data: { + name: 'Updated Name', + bio: 'Test bio' + } + }); + + expect(response.ok()).toBeTruthy(); + const updated = await response.json(); + expect(updated.name).toBe('Updated Name'); + expect(updated.bio).toBe('Test bio'); + }); + + test('should validate email format', async ({ request }) => { + const response = await request.patch('/api/profile', { + headers: { Authorization: `Bearer ${authToken}` }, + data: { email: 'invalid-email' } + }); + + expect(response.status()).toBe(400); + const error = await response.json(); + expect(error.message).toContain('Invalid email'); + }); + + test('should require authentication', async ({ request }) => { + const response = await request.get('/api/profile'); + expect(response.status()).toBe(401); + }); +}); +``` + +**With Playwright Utils:** +```typescript +import { test as base, expect } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; +import { mergeTests } from '@playwright/test'; +import { z } from 'zod'; + +const ProfileSchema = z.object({ + id: z.string(), + name: z.string(), + email: z.string().email() +}); + +// Merge API and auth fixtures +const authFixtureTest = base.extend(createAuthFixtures()); +export const testWithAuth = mergeTests(apiRequestFixture, authFixtureTest); + +testWithAuth.describe('Profile API', () => { + testWithAuth('should fetch user profile', async ({ apiRequest, authToken }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/profile', + headers: { Authorization: `Bearer ${authToken}` } + }).validateSchema(ProfileSchema); // Chained validation + + expect(status).toBe(200); + // Schema already validated, type-safe access + expect(body.name).toBeDefined(); + }); + + testWithAuth('should update profile successfully', async ({ apiRequest, authToken }) => { + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { name: 'Updated Name', bio: 'Test bio' }, // 'body' not 'data' + headers: { Authorization: `Bearer ${authToken}` } + }).validateSchema(ProfileSchema); // Chained validation + + expect(status).toBe(200); + expect(body.name).toBe('Updated Name'); + }); + + testWithAuth('should validate email format', async ({ apiRequest, authToken }) => { + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { email: 'invalid-email' }, // 'body' not 'data' + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(status).toBe(400); + expect(body.message).toContain('Invalid email'); + }); +}); +``` + +**Key Differences:** +- `authToken` fixture (persisted, reused across tests) +- `apiRequest` returns `{ status, body }` (cleaner) +- Schema validation with Zod (type-safe) +- Automatic retry for 5xx errors +- Less boilerplate (no manual `await response.json()` everywhere) + +#### E2E Tests (`tests/e2e/profile-workflow.spec.ts`): + +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('Profile Management Workflow', () => { + test.beforeEach(async ({ page }) => { + // Login + await page.goto('/login'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password').fill('password123'); + await page.getByRole('button', { name: 'Sign in' }).click(); + + // Wait for login to complete + await expect(page).toHaveURL(/\/dashboard/); + }); + + test('should view and edit profile', async ({ page }) => { + // Navigate to profile + await page.goto('/profile'); + + // Verify profile displays + await expect(page.getByText('test@example.com')).toBeVisible(); + + // Edit profile + await page.getByRole('button', { name: 'Edit Profile' }).click(); + await page.getByLabel('Name').fill('New Name'); + await page.getByRole('button', { name: 'Save' }).click(); + + // Verify success + await expect(page.getByText('Profile updated')).toBeVisible(); + await expect(page.getByText('New Name')).toBeVisible(); + }); + + test('should show validation errors', async ({ page }) => { + await page.goto('/profile'); + await page.getByRole('button', { name: 'Edit Profile' }).click(); + + // Enter invalid email + await page.getByLabel('Email').fill('invalid'); + await page.getByRole('button', { name: 'Save' }).click(); + + // Verify error shown + await expect(page.getByText('Invalid email format')).toBeVisible(); + + // Profile should not be updated + await page.reload(); + await expect(page.getByText('test@example.com')).toBeVisible(); + }); +}); +``` + +#### Fixtures (`tests/support/fixtures/profile.ts`): + +**Vanilla Playwright:** +```typescript +import { test as base, Page } from '@playwright/test'; + +type ProfileFixtures = { + authenticatedPage: Page; + testProfile: { + name: string; + email: string; + bio: string; + }; +}; + +export const test = base.extend({ + authenticatedPage: async ({ page }, use) => { + // Manual login flow + await page.goto('/login'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password').fill('password123'); + await page.getByRole('button', { name: 'Sign in' }).click(); + await page.waitForURL(/\/dashboard/); + + await use(page); + }, + + testProfile: async ({ request }, use) => { + // Static test data + const profile = { + name: 'Test User', + email: 'test@example.com', + bio: 'Test bio' + }; + + await use(profile); + } +}); +``` + +**With Playwright Utils:** +```typescript +import { test as base } from '@playwright/test'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; +import { mergeTests } from '@playwright/test'; +import { faker } from '@faker-js/faker'; + +type ProfileFixtures = { + testProfile: { + name: string; + email: string; + bio: string; + }; +}; + +// Merge auth fixtures with custom fixtures +const authTest = base.extend(createAuthFixtures()); +const profileTest = base.extend({ + testProfile: async ({}, use) => { + // Dynamic test data with faker + const profile = { + name: faker.person.fullName(), + email: faker.internet.email(), + bio: faker.person.bio() + }; + + await use(profile); + } +}); + +export const test = mergeTests(authTest, profileTest); +export { expect } from '@playwright/test'; +``` + +**Usage:** +```typescript +import { test, expect } from '../support/fixtures/profile'; + +test('should update profile', async ({ page, authToken, testProfile }) => { + // authToken from auth-session (automatic, persisted) + // testProfile from custom fixture (dynamic data) + + await page.goto('/profile'); + // Test with dynamic, unique data +}); +``` + +**Key Benefits:** +- `authToken` fixture (persisted token, no manual login) +- Dynamic test data with faker (no conflicts) +- Fixture composition with mergeTests +- Reusable across test files + +### 6. Review Additional Artifacts + +TEA also generates: + +#### Updated README (`tests/README.md`): + +```markdown +# Test Suite + +## Running Tests + +### All Tests +npm test + +### Specific Levels +npm run test:api # API tests only +npm run test:e2e # E2E tests only +npm run test:smoke # Smoke tests (@smoke tag) + +### Single File +npx playwright test tests/api/profile.spec.ts + +## Test Structure + +tests/ +├── api/ # API tests (fast, reliable) +├── e2e/ # E2E tests (full workflows) +├── fixtures/ # Shared test utilities +└── README.md + +## Writing Tests + +Follow the patterns in existing tests: +- Use fixtures for authentication +- Network-first patterns (no hard waits) +- Explicit assertions +- Self-cleaning tests +``` + +#### Definition of Done Summary: + +```markdown +## Test Quality Checklist + +✅ All tests pass on first run +✅ No hard waits (waitForTimeout) +✅ No conditionals for flow control +✅ Assertions are explicit +✅ Tests clean up after themselves +✅ Tests can run in parallel +✅ Execution time < 1.5 minutes per test +✅ Test files < 300 lines +``` + +### 7. Run the Tests + +All tests should pass immediately since the feature exists: + +**For Playwright:** +```bash +npx playwright test +``` + +**For Cypress:** +```bash +npx cypress run +``` + +Expected output: +``` +Running 15 tests using 4 workers + + ✓ tests/api/profile.spec.ts (4 tests) - 2.1s + ✓ tests/e2e/profile-workflow.spec.ts (2 tests) - 5.3s + + 15 passed (7.4s) +``` + +**All green!** Tests pass because feature already exists. + +### 8. Review Test Coverage + +Check which scenarios are covered: + +```bash +# View test report +npx playwright show-report + +# Check coverage (if configured) +npm run test:coverage +``` + +Compare against: +- Acceptance criteria from story +- Test priorities from test design +- Edge cases and error scenarios + +## What You Get + +### Comprehensive Test Suite +- **API tests** - Fast, reliable backend testing +- **E2E tests** - Critical user workflows +- **Component tests** - UI component testing (if requested) +- **Fixtures** - Shared utilities and setup + +### Component Testing by Framework + +TEA supports component testing using framework-appropriate tools: + +| Your Framework | Component Testing Tool | Tests Location | +|----------------|----------------------|----------------| +| **Cypress** | Cypress Component Testing | `tests/component/` | +| **Playwright** | Vitest + React Testing Library | `tests/component/` or `src/**/*.test.tsx` | + +**Note:** Component tests use separate tooling from E2E tests: +- Cypress users: TEA generates Cypress Component Tests +- Playwright users: TEA generates Vitest + React Testing Library tests + +### Quality Features +- **Network-first patterns** - Wait for actual responses, not timeouts +- **Deterministic tests** - No flakiness, no conditionals +- **Self-cleaning** - Tests don't leave test data behind +- **Parallel-safe** - Can run all tests concurrently + +### Documentation +- **Updated README** - How to run tests +- **Test structure explanation** - Where tests live +- **Definition of Done** - Quality standards + +## Tips + +### Start with Test Design + +Run `*test-design` before `*automate` for better results: + +``` +*test-design # Risk assessment, priorities +*automate # Generate tests based on priorities +``` + +TEA will focus on P0/P1 scenarios and skip low-value tests. + +### Prioritize Test Levels + +Not everything needs E2E tests: + +**Good strategy:** +``` +- P0 scenarios: API + E2E tests +- P1 scenarios: API tests only +- P2 scenarios: API tests (happy path) +- P3 scenarios: Skip or add later +``` + +**Why?** +- API tests are 10x faster than E2E +- API tests are more reliable (no browser flakiness) +- E2E tests reserved for critical user journeys + +### Avoid Duplicate Coverage + +Tell TEA about existing tests: + +``` +We already have tests in: +- tests/e2e/profile-view.spec.ts (viewing profile) +- tests/api/auth.spec.ts (authentication) + +Don't duplicate that coverage +``` + +TEA will analyze existing tests and only generate new scenarios. + +### Use Healing Mode (Optional) + +If MCP enhancements enabled (`tea_use_mcp_enhancements: true`): + +When prompted, select "healing mode" to: +- Fix broken selectors in existing tests +- Update outdated assertions +- Enhance with trace viewer insights + +See [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) + +### Use Recording Mode (Optional) + +If MCP enhancements enabled: + +When prompted, select "recording mode" to: +- Verify selectors against live browser +- Generate accurate locators from actual DOM +- Capture network requests + +### Generate Tests Incrementally + +Don't generate all tests at once: + +**Iteration 1:** +``` +Generate P0 tests only (critical path) +Run: *automate +``` + +**Iteration 2:** +``` +Generate P1 tests (high value scenarios) +Run: *automate +Tell TEA to avoid P0 coverage +``` + +**Iteration 3:** +``` +Generate P2 tests (if time permits) +Run: *automate +``` + +This iterative approach: +- Provides fast feedback +- Allows validation before proceeding +- Keeps test generation focused + +## Common Issues + +### Tests Pass But Coverage Is Incomplete + +**Problem:** Tests pass but don't cover all scenarios. + +**Cause:** TEA wasn't given complete context. + +**Solution:** Provide more details: +``` +Generate tests for: +- All acceptance criteria in story-profile.md +- Error scenarios (validation, authorization) +- Edge cases (empty fields, long inputs) +``` + +### Too Many Tests Generated + +**Problem:** TEA generated 50 tests for a simple feature. + +**Cause:** Didn't specify priorities or scope. + +**Solution:** Be specific: +``` +Generate ONLY: +- P0 and P1 scenarios +- API tests for all scenarios +- E2E tests only for critical workflows +- Skip P2/P3 for now +``` + +### Tests Duplicate Existing Coverage + +**Problem:** New tests cover the same scenarios as existing tests. + +**Cause:** Didn't tell TEA about existing tests. + +**Solution:** Specify existing coverage: +``` +We already have these tests: +- tests/api/profile.spec.ts (GET /api/profile) +- tests/e2e/profile-view.spec.ts (viewing profile) + +Generate tests for scenarios NOT covered by those files +``` + +### Selectors Are Fragile + +**Problem:** E2E tests use brittle CSS selectors. + +**Solution:** Request accessible selectors: +``` +Use accessible locators: +- getByRole() +- getByLabel() +- getByText() + +Avoid CSS selectors like .class-name or #id +``` + +Or use MCP recording mode for verified selectors. + +## Related Guides + +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Plan before generating +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Failing tests before implementation +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Audit generated quality + +## Understanding the Concepts + +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Why prioritize P0 over P3 +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - What makes tests good +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Reusable test patterns + +## Reference + +- [Command: *automate](/docs/reference/tea/commands.md#automate) - Full command reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - MCP and Playwright Utils options + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-nfr-assess.md b/docs/how-to/workflows/run-nfr-assess.md new file mode 100644 index 00000000..739650d1 --- /dev/null +++ b/docs/how-to/workflows/run-nfr-assess.md @@ -0,0 +1,679 @@ +--- +title: "How to Run NFR Assessment with TEA" +description: Validate non-functional requirements for security, performance, reliability, and maintainability using TEA +--- + +# How to Run NFR Assessment with TEA + +Use TEA's `*nfr-assess` workflow to validate non-functional requirements (NFRs) with evidence-based assessment across security, performance, reliability, and maintainability. + +## When to Use This + +- Enterprise projects with compliance requirements +- Projects with strict NFR thresholds +- Before production release +- When NFRs are critical to project success +- Security or performance is mission-critical + +**Best for:** +- Enterprise track projects +- Compliance-heavy industries (finance, healthcare, government) +- High-traffic applications +- Security-critical systems + +## Prerequisites + +- BMad Method installed +- TEA agent available +- NFRs defined in PRD or requirements doc +- Evidence preferred but not required (test results, security scans, performance metrics) + +**Note:** You can run NFR assessment without complete evidence. TEA will mark categories as CONCERNS where evidence is missing and document what's needed. + +## Steps + +### 1. Run the NFR Assessment Workflow + +Start a fresh chat and run: + +``` +*nfr-assess +``` + +This loads TEA and starts the NFR assessment workflow. + +### 2. Specify NFR Categories + +TEA will ask which NFR categories to assess. + +**Available Categories:** + +| Category | Focus Areas | +|----------|-------------| +| **Security** | Authentication, authorization, encryption, vulnerabilities, security headers, input validation | +| **Performance** | Response time, throughput, resource usage, database queries, frontend load time | +| **Reliability** | Error handling, recovery mechanisms, availability, failover, data backup | +| **Maintainability** | Code quality, test coverage, technical debt, documentation, dependency health | + +**Example Response:** +``` +Assess: +- Security (critical for user data) +- Performance (API must be fast) +- Reliability (99.9% uptime requirement) + +Skip maintainability for now +``` + +### 3. Provide NFR Thresholds + +TEA will ask for specific thresholds for each category. + +**Critical Principle: Never guess thresholds.** + +If you don't know the exact requirement, tell TEA to mark as CONCERNS and request clarification from stakeholders. + +#### Security Thresholds + +**Example:** +``` +Requirements: +- All endpoints require authentication: YES +- Data encrypted at rest: YES (PostgreSQL TDE) +- Zero critical vulnerabilities: YES (npm audit) +- Input validation on all endpoints: YES (Zod schemas) +- Security headers configured: YES (helmet.js) +``` + +#### Performance Thresholds + +**Example:** +``` +Requirements: +- API response time P99: < 200ms +- API response time P95: < 150ms +- Throughput: > 1000 requests/second +- Frontend initial load: < 2 seconds +- Database query time P99: < 50ms +``` + +#### Reliability Thresholds + +**Example:** +``` +Requirements: +- Error handling: All endpoints return structured errors +- Availability: 99.9% uptime +- Recovery time: < 5 minutes (RTO) +- Data backup: Daily automated backups +- Failover: Automatic with < 30s downtime +``` + +#### Maintainability Thresholds + +**Example:** +``` +Requirements: +- Test coverage: > 80% +- Code quality: SonarQube grade A +- Documentation: All APIs documented +- Dependency age: < 6 months outdated +- Technical debt: < 10% of codebase +``` + +### 4. Provide Evidence + +TEA will ask where to find evidence for each requirement. + +**Evidence Sources:** + +| Category | Evidence Type | Location | +|----------|---------------|----------| +| Security | Security scan reports | `/reports/security-scan.pdf` | +| Security | Vulnerability scan | `npm audit`, `snyk test` results | +| Security | Auth test results | Test reports showing auth coverage | +| Performance | Load test results | `/reports/k6-load-test.json` | +| Performance | APM data | Datadog, New Relic dashboards | +| Performance | Lighthouse scores | `/reports/lighthouse.json` | +| Reliability | Error rate metrics | Production monitoring dashboards | +| Reliability | Uptime data | StatusPage, PagerDuty logs | +| Maintainability | Coverage reports | `/reports/coverage/index.html` | +| Maintainability | Code quality | SonarQube dashboard | + +**Example Response:** +``` +Evidence: +- Security: npm audit results (clean), auth tests 15/15 passing +- Performance: k6 load test at /reports/k6-results.json +- Reliability: Error rate 0.01% in staging (logs in Datadog) + +Don't have: +- Uptime data (new system, no baseline) +- Mark as CONCERNS and request monitoring setup +``` + +### 5. Review NFR Assessment Report + +TEA generates a comprehensive assessment report. + +#### Assessment Report (`nfr-assessment.md`): + +```markdown +# Non-Functional Requirements Assessment + +**Date:** 2026-01-13 +**Epic:** User Profile Management +**Release:** v1.2.0 +**Overall Decision:** CONCERNS ⚠️ + +## Executive Summary + +| Category | Status | Critical Issues | +|----------|--------|-----------------| +| Security | PASS ✅ | 0 | +| Performance | CONCERNS ⚠️ | 2 | +| Reliability | PASS ✅ | 0 | +| Maintainability | PASS ✅ | 0 | + +**Decision Rationale:** +Performance metrics below target (P99 latency, throughput). Mitigation plan in place. Security and reliability meet all requirements. + +--- + +## Security Assessment + +**Status:** PASS ✅ + +### Requirements Met + +| Requirement | Target | Actual | Status | +|-------------|--------|--------|--------| +| Authentication required | All endpoints | 100% enforced | ✅ | +| Data encryption at rest | PostgreSQL TDE | Enabled | ✅ | +| Critical vulnerabilities | 0 | 0 | ✅ | +| Input validation | All endpoints | Zod schemas on 100% | ✅ | +| Security headers | Configured | helmet.js enabled | ✅ | + +### Evidence + +**Security Scan:** +```bash +$ npm audit +found 0 vulnerabilities +``` + +**Authentication Tests:** +- 15/15 auth tests passing +- Tested unauthorized access (401 responses) +- Token validation working + +**Penetration Testing:** +- Report: `/reports/pentest-2026-01.pdf` +- Findings: 0 critical, 2 low (addressed) + +**Conclusion:** All security requirements met. No blockers. + +--- + +## Performance Assessment + +**Status:** CONCERNS ⚠️ + +### Requirements Status + +| Metric | Target | Actual | Status | +|--------|--------|--------|--------| +| API response P99 | < 200ms | 350ms | ❌ Exceeds | +| API response P95 | < 150ms | 180ms | ⚠️ Exceeds | +| Throughput | > 1000 rps | 850 rps | ⚠️ Below | +| Frontend load | < 2s | 1.8s | ✅ Met | +| DB query P99 | < 50ms | 85ms | ❌ Exceeds | + +### Issues Identified + +#### Issue 1: P99 Latency Exceeds Target + +**Measured:** 350ms P99 (target: <200ms) +**Root Cause:** Database queries not optimized +- Missing indexes on profile queries +- N+1 query problem in profile endpoint + +**Impact:** User experience degraded for 1% of requests + +**Mitigation Plan:** +- Add composite index on `(user_id, profile_id)` - backend team, 2 days +- Refactor profile endpoint to use joins instead of multiple queries - backend team, 3 days +- Re-run load tests after optimization - QA team, 1 day + +**Owner:** Backend team lead +**Deadline:** Before release (January 20, 2026) + +#### Issue 2: Throughput Below Target + +**Measured:** 850 rps (target: >1000 rps) +**Root Cause:** Connection pool size too small +- PostgreSQL max_connections = 100 (too low) +- No connection pooling in application + +**Impact:** System cannot handle expected traffic + +**Mitigation Plan:** +- Increase PostgreSQL max_connections to 500 - DevOps, 1 day +- Implement connection pooling with pg-pool - backend team, 2 days +- Re-run load tests - QA team, 1 day + +**Owner:** DevOps + Backend team +**Deadline:** Before release (January 20, 2026) + +### Evidence + +**Load Testing:** +``` +Tool: k6 +Duration: 10 minutes +Virtual Users: 500 concurrent +Report: /reports/k6-load-test.json +``` + +**Results:** +``` +scenarios: (100.00%) 1 scenario, 500 max VUs, 10m30s max duration + ✓ http_req_duration..............: avg=250ms min=45ms med=180ms max=2.1s p(90)=280ms p(95)=350ms + http_reqs......................: 85000 (850/s) + http_req_failed................: 0.1% +``` + +**APM Data:** +- Tool: Datadog +- Dashboard: https://app.datadoghq.com/dashboard/abc123 + +**Conclusion:** Performance issues identified with mitigation plan. Re-assess after optimization. + +--- + +## Reliability Assessment + +**Status:** PASS ✅ + +### Requirements Met + +| Requirement | Target | Actual | Status | +|-------------|--------|--------|--------| +| Error handling | Structured errors | 100% endpoints | ✅ | +| Availability | 99.9% uptime | 99.95% (staging) | ✅ | +| Recovery time | < 5 min (RTO) | 3 min (tested) | ✅ | +| Data backup | Daily | Automated daily | ✅ | +| Failover | < 30s downtime | 15s (tested) | ✅ | + +### Evidence + +**Error Handling Tests:** +- All endpoints return structured JSON errors +- Error codes standardized (400, 401, 403, 404, 500) +- Error messages user-friendly (no stack traces) + +**Chaos Engineering:** +- Tested database failover: 15s downtime ✅ +- Tested service crash recovery: 3 min ✅ +- Tested network partition: Graceful degradation ✅ + +**Monitoring:** +- Staging uptime (30 days): 99.95% +- Error rate: 0.01% (target: <0.1%) +- P50 availability: 100% + +**Conclusion:** All reliability requirements exceeded. No issues. + +--- + +## Maintainability Assessment + +**Status:** PASS ✅ + +### Requirements Met + +| Requirement | Target | Actual | Status | +|-------------|--------|--------|--------| +| Test coverage | > 80% | 85% | ✅ | +| Code quality | Grade A | Grade A | ✅ | +| Documentation | All APIs | 100% documented | ✅ | +| Outdated dependencies | < 6 months | 3 months avg | ✅ | +| Technical debt | < 10% | 7% | ✅ | + +### Evidence + +**Test Coverage:** +``` +Statements : 85.2% ( 1205/1414 ) +Branches : 82.1% ( 412/502 ) +Functions : 88.5% ( 201/227 ) +Lines : 85.2% ( 1205/1414 ) +``` + +**Code Quality:** +- SonarQube: Grade A +- Maintainability rating: A +- Technical debt ratio: 7% +- Code smells: 12 (all minor) + +**Documentation:** +- API docs: 100% coverage (OpenAPI spec) +- README: Complete and up-to-date +- Architecture docs: ADRs for all major decisions + +**Conclusion:** All maintainability requirements met. Codebase is healthy. + +--- + +## Overall Gate Decision + +### Decision: CONCERNS ⚠️ + +**Rationale:** +- **Blockers:** None +- **Concerns:** Performance metrics below target (P99 latency, throughput) +- **Mitigation:** Plan in place with clear owners and deadlines (5 days total) +- **Passing:** Security, reliability, maintainability all green + +### Actions Required Before Release + +1. **Optimize database queries** (backend team, 3 days) + - Add indexes + - Fix N+1 queries + - Implement connection pooling + +2. **Re-run performance tests** (QA team, 1 day) + - Validate P99 < 200ms + - Validate throughput > 1000 rps + +3. **Update this assessment** (TEA, 1 hour) + - Re-run `*nfr-assess` with new results + - Confirm PASS status + +### Waiver Option (If Business Approves) + +If business decides to deploy with current performance: + +**Waiver Justification:** +```markdown +## Performance Waiver + +**Waived By:** VP Engineering, Product Manager +**Date:** 2026-01-15 +**Reason:** Business priority to launch by Q1 +**Conditions:** +- Set monitoring alerts for P99 > 300ms +- Plan optimization for v1.3 (February release) +- Document known performance limitations in release notes + +**Accepted Risk:** +- 1% of users experience slower response (350ms vs 200ms) +- System can handle current traffic (850 rps sufficient for launch) +- Optimization planned for next release +``` + +### Approvals + +- [ ] Product Manager - Review business impact +- [ ] Tech Lead - Review mitigation plan +- [ ] QA Lead - Validate test evidence +- [ ] DevOps - Confirm infrastructure ready + +--- + +## Monitoring Plan Post-Release + +**Performance Alerts:** +- P99 latency > 400ms (critical) +- Throughput < 700 rps (warning) +- Error rate > 1% (critical) + +**Review Cadence:** +- Daily: Check performance dashboards +- Weekly: Review alert trends +- Monthly: Re-assess NFRs +``` + +## What You Get + +### NFR Assessment Report +- Category-by-category analysis (Security, Performance, Reliability, Maintainability) +- Requirements status (target vs actual) +- Evidence for each requirement +- Issues identified with root cause analysis + +### Gate Decision +- **PASS** ✅ - All NFRs met, ready to release +- **CONCERNS** ⚠️ - Some NFRs not met, mitigation plan exists +- **FAIL** ❌ - Critical NFRs not met, blocks release +- **WAIVED** ⏭️ - Business-approved waiver with documented risk + +### Mitigation Plans +- Specific actions to address concerns +- Owners and deadlines +- Re-assessment criteria + +### Monitoring Plan +- Post-release monitoring strategy +- Alert thresholds +- Review cadence + +## Tips + +### Run NFR Assessment Early + +**Phase 2 (Enterprise):** +Run `*nfr-assess` during planning to: +- Identify NFR requirements early +- Plan for performance testing +- Budget for security audits +- Set up monitoring infrastructure + +**Phase 4 or Gate:** +Re-run before release to validate all requirements met. + +### Never Guess Thresholds + +If you don't know the NFR target: + +**Don't:** +``` +API response time should probably be under 500ms +``` + +**Do:** +``` +Mark as CONCERNS - Request threshold from stakeholders +"What is the acceptable API response time?" +``` + +### Collect Evidence Beforehand + +Before running `*nfr-assess`, gather: + +**Security:** +```bash +npm audit # Vulnerability scan +snyk test # Alternative security scan +npm run test:security # Security test suite +``` + +**Performance:** +```bash +npm run test:load # k6 or artillery load tests +npm run test:lighthouse # Frontend performance +npm run test:db-performance # Database query analysis +``` + +**Reliability:** +- Production error rate (last 30 days) +- Uptime data (StatusPage, PagerDuty) +- Incident response times + +**Maintainability:** +```bash +npm run test:coverage # Test coverage report +npm run lint # Code quality check +npm outdated # Dependency freshness +``` + +### Use Real Data, Not Assumptions + +**Don't:** +``` +System is probably fast enough +Security seems fine +``` + +**Do:** +``` +Load test results show P99 = 350ms +npm audit shows 0 vulnerabilities +Test coverage report shows 85% +``` + +Evidence-based decisions prevent surprises in production. + +### Document Waivers Thoroughly + +If business approves waiver: + +**Required:** +- Who approved (name, role, date) +- Why (business justification) +- Conditions (monitoring, future plans) +- Accepted risk (quantified impact) + +**Example:** +```markdown +Waived by: CTO, VP Product (2026-01-15) +Reason: Q1 launch critical for investor demo +Conditions: Optimize in v1.3, monitor closely +Risk: 1% of users experience 350ms latency (acceptable for launch) +``` + +### Re-Assess After Fixes + +After implementing mitigations: + +``` +1. Fix performance issues +2. Run load tests again +3. Run *nfr-assess with new evidence +4. Verify PASS status +``` + +Don't deploy with CONCERNS without mitigation or waiver. + +### Integrate with Release Checklist + +```markdown +## Release Checklist + +### Pre-Release +- [ ] All tests passing +- [ ] Test coverage > 80% +- [ ] Run *nfr-assess +- [ ] NFR status: PASS or WAIVED + +### Performance +- [ ] Load tests completed +- [ ] P99 latency meets threshold +- [ ] Throughput meets threshold + +### Security +- [ ] Security scan clean +- [ ] Auth tests passing +- [ ] Penetration test complete + +### Post-Release +- [ ] Monitoring alerts configured +- [ ] Dashboards updated +- [ ] Incident response plan ready +``` + +## Common Issues + +### No Evidence Available + +**Problem:** Don't have performance data, security scans, etc. + +**Solution:** +``` +Mark as CONCERNS for categories without evidence +Document what evidence is needed +Set up tests/scans before re-assessment +``` + +**Don't block on missing evidence** - document what's needed and proceed. + +### Thresholds Too Strict + +**Problem:** Can't meet unrealistic thresholds. + +**Symptoms:** +- P99 < 50ms (impossible for complex queries) +- 100% test coverage (impractical) +- Zero technical debt (unrealistic) + +**Solution:** +``` +Negotiate thresholds with stakeholders: +- "P99 < 50ms is unrealistic for our DB queries" +- "Propose P99 < 200ms based on industry standards" +- "Show evidence from load tests" +``` + +Use data to negotiate realistic requirements. + +### Assessment Takes Too Long + +**Problem:** Gathering evidence for all categories is time-consuming. + +**Solution:** Focus on critical categories first: + +**For most projects:** +``` +Priority 1: Security (always critical) +Priority 2: Performance (if high-traffic) +Priority 3: Reliability (if uptime critical) +Priority 4: Maintainability (nice to have) +``` + +Assess categories incrementally, not all at once. + +### CONCERNS vs FAIL - When to Block? + +**CONCERNS** ⚠️: +- Issues exist but not critical +- Mitigation plan in place +- Business accepts risk (with waiver) +- Can deploy with monitoring + +**FAIL** ❌: +- Critical security vulnerability (CVE critical) +- System unusable (error rate >10%) +- Data loss risk (no backups) +- Zero mitigation possible + +**Rule of thumb:** If you can mitigate or monitor, use CONCERNS. Reserve FAIL for absolute blockers. + +## Related Guides + +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Gate decision complements NFR +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Quality complements NFR +- [Run TEA for Enterprise](/docs/how-to/workflows/run-tea-for-enterprise.md) - Enterprise workflow + +## Understanding the Concepts + +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Risk assessment principles +- [TEA Overview](/docs/explanation/features/tea-overview.md) - NFR in release gates + +## Reference + +- [Command: *nfr-assess](/docs/reference/tea/commands.md#nfr-assess) - Full command reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - Enterprise config options + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-tea-for-enterprise.md b/docs/how-to/workflows/run-tea-for-enterprise.md new file mode 100644 index 00000000..5285153c --- /dev/null +++ b/docs/how-to/workflows/run-tea-for-enterprise.md @@ -0,0 +1,526 @@ +--- +title: "Running TEA for Enterprise Projects" +description: Use TEA with compliance, security, and regulatory requirements in enterprise environments +--- + +# Running TEA for Enterprise Projects + +Use TEA on enterprise projects with compliance, security, audit, and regulatory requirements. This guide covers NFR assessment, audit trails, and evidence collection. + +## When to Use This + +- Enterprise track projects (not Quick Flow or simple BMad Method) +- Compliance requirements (SOC 2, HIPAA, GDPR, etc.) +- Security-critical applications (finance, healthcare, government) +- Audit trail requirements +- Strict NFR thresholds (performance, security, reliability) + +## Prerequisites + +- BMad Method installed (Enterprise track selected) +- TEA agent available +- Compliance requirements documented +- Stakeholders identified (who approves gates) + +## Enterprise-Specific TEA Workflows + +### NFR Assessment (*nfr-assess) + +**Purpose:** Validate non-functional requirements with evidence. + +**When:** Phase 2 (early) and Release Gate + +**Why Enterprise Needs This:** +- Compliance mandates specific thresholds +- Audit trails required for certification +- Security requirements are non-negotiable +- Performance SLAs are contractual + +**Example:** +``` +*nfr-assess + +Categories: Security, Performance, Reliability, Maintainability + +Security thresholds: +- Zero critical vulnerabilities (required by SOC 2) +- All endpoints require authentication +- Data encrypted at rest (FIPS 140-2) +- Audit logging on all data access + +Evidence: +- Security scan: reports/nessus-scan.pdf +- Penetration test: reports/pentest-2026-01.pdf +- Compliance audit: reports/soc2-evidence.zip +``` + +**Output:** NFR assessment with PASS/CONCERNS/FAIL for each category. + +### Trace with Audit Evidence (*trace) + +**Purpose:** Requirements traceability with audit trail. + +**When:** Phase 2 (baseline), Phase 4 (refresh), Release Gate + +**Why Enterprise Needs This:** +- Auditors require requirements-to-test mapping +- Compliance certifications need traceability +- Regulatory bodies want evidence + +**Example:** +``` +*trace Phase 1 + +Requirements: PRD.md (with compliance requirements) +Test location: tests/ + +Output: traceability-matrix.md with: +- Requirement-to-test mapping +- Compliance requirement coverage +- Gap prioritization +- Recommendations +``` + +**For Release Gate:** +``` +*trace Phase 2 + +Generate gate-decision-{gate_type}-{story_id}.md with: +- Evidence references +- Approver signatures +- Compliance checklist +- Decision rationale +``` + +### Test Design with Compliance Focus (*test-design) + +**Purpose:** Risk assessment with compliance and security focus. + +**When:** Phase 3 (system-level), Phase 4 (epic-level) + +**Why Enterprise Needs This:** +- Security architecture alignment required +- Compliance requirements must be testable +- Performance requirements are contractual + +**Example:** +``` +*test-design + +Mode: System-level + +Focus areas: +- Security architecture (authentication, authorization, encryption) +- Performance requirements (SLA: P99 <200ms) +- Compliance (HIPAA PHI handling, audit logging) + +Output: test-design-system.md with: +- Security testing strategy +- Compliance requirement → test mapping +- Performance testing plan +- Audit logging validation +``` + +## Enterprise TEA Lifecycle + +### Phase 1: Discovery (Optional but Recommended) + +**Research compliance requirements:** +``` +Analyst: *research + +Topics: +- Industry compliance (SOC 2, HIPAA, GDPR) +- Security standards (OWASP Top 10) +- Performance benchmarks (industry P99) +``` + +### Phase 2: Planning (Required) + +**1. Define NFRs early:** +``` +PM: *prd + +Include in PRD: +- Security requirements (authentication, encryption) +- Performance SLAs (response time, throughput) +- Reliability targets (uptime, RTO, RPO) +- Compliance mandates (data retention, audit logs) +``` + +**2. Assess NFRs:** +``` +TEA: *nfr-assess + +Categories: All (Security, Performance, Reliability, Maintainability) + +Output: nfr-assessment.md +- NFR requirements documented +- Acceptance criteria defined +- Test strategy planned +``` + +**3. Baseline (brownfield only):** +``` +TEA: *trace Phase 1 + +Establish baseline coverage before new work +``` + +### Phase 3: Solutioning (Required) + +**1. Architecture with testability review:** +``` +Architect: *architecture + +TEA: *test-design (system-level) + +Focus: +- Security architecture testability +- Performance testing strategy +- Compliance requirement mapping +``` + +**2. Test infrastructure:** +``` +TEA: *framework + +Requirements: +- Separate test environments (dev, staging, prod-mirror) +- Secure test data handling (PHI, PII) +- Audit logging in tests +``` + +**3. CI/CD with compliance:** +``` +TEA: *ci + +Requirements: +- Secrets management (Vault, AWS Secrets Manager) +- Test isolation (no cross-contamination) +- Artifact retention (compliance audit trail) +- Access controls (who can run production tests) +``` + +### Phase 4: Implementation (Required) + +**Per epic:** +``` +1. TEA: *test-design (epic-level) + Focus: Compliance, security, performance for THIS epic + +2. TEA: *atdd (optional) + Generate tests including security/compliance scenarios + +3. DEV: Implement story + +4. TEA: *automate + Expand coverage including compliance edge cases + +5. TEA: *test-review + Audit quality (score >80 per epic, rises to >85 at release) + +6. TEA: *trace Phase 1 + Refresh coverage, verify compliance requirements tested +``` + +### Release Gate (Required) + +**1. Final NFR assessment:** +``` +TEA: *nfr-assess + +All categories (if not done earlier) +Latest evidence (performance tests, security scans) +``` + +**2. Final quality audit:** +``` +TEA: *test-review tests/ + +Full suite review +Quality target: >85 for enterprise +``` + +**3. Gate decision:** +``` +TEA: *trace Phase 2 + +Evidence required: +- traceability-matrix.md (from Phase 1) +- test-review.md (from quality audit) +- nfr-assessment.md (from NFR assessment) +- Test execution results (must have test results available) + +Decision: PASS/CONCERNS/FAIL/WAIVED + +Archive all artifacts for compliance audit +``` + +**Note:** Phase 2 requires test execution results. If results aren't available, Phase 2 will be skipped. + +**4. Archive for audit:** +``` +Archive: +- All test results +- Coverage reports +- NFR assessments +- Gate decisions +- Approver signatures + +Retention: Per compliance requirements (7 years for HIPAA) +``` + +## Enterprise-Specific Requirements + +### Evidence Collection + +**Required artifacts:** +- Requirements traceability matrix +- Test execution results (with timestamps) +- NFR assessment reports +- Security scan results +- Performance test results +- Gate decision records +- Approver signatures + +**Storage:** +``` +compliance/ +├── 2026-Q1/ +│ ├── release-1.2.0/ +│ │ ├── traceability-matrix.md +│ │ ├── test-review.md +│ │ ├── nfr-assessment.md +│ │ ├── gate-decision-release-v1.2.0.md +│ │ ├── test-results/ +│ │ ├── security-scans/ +│ │ └── approvals.pdf +``` + +**Retention:** 7 years (HIPAA), 3 years (SOC 2), per your compliance needs + +### Approver Workflows + +**Multi-level approval required:** + +```markdown +## Gate Approvals Required + +### Technical Approval +- [ ] QA Lead - Test coverage adequate +- [ ] Tech Lead - Technical quality acceptable +- [ ] Security Lead - Security requirements met + +### Business Approval +- [ ] Product Manager - Business requirements met +- [ ] Compliance Officer - Regulatory requirements met + +### Executive Approval (for major releases) +- [ ] VP Engineering - Overall quality acceptable +- [ ] CTO - Architecture approved for production +``` + +### Compliance Checklists + +**SOC 2 Example:** +```markdown +## SOC 2 Compliance Checklist + +### Access Controls +- [ ] All API endpoints require authentication +- [ ] Authorization tested for all protected resources +- [ ] Session management secure (token expiration tested) + +### Audit Logging +- [ ] All data access logged +- [ ] Logs immutable (append-only) +- [ ] Log retention policy enforced + +### Data Protection +- [ ] Data encrypted at rest (tested) +- [ ] Data encrypted in transit (HTTPS enforced) +- [ ] PII handling compliant (masking tested) + +### Testing Evidence +- [ ] Test coverage >80% (verified) +- [ ] Security tests passing (100%) +- [ ] Traceability matrix complete +``` + +**HIPAA Example:** +```markdown +## HIPAA Compliance Checklist + +### PHI Protection +- [ ] PHI encrypted at rest (AES-256) +- [ ] PHI encrypted in transit (TLS 1.3) +- [ ] PHI access logged (audit trail) + +### Access Controls +- [ ] Role-based access control (RBAC tested) +- [ ] Minimum necessary access (tested) +- [ ] Authentication strong (MFA tested) + +### Breach Notification +- [ ] Breach detection tested +- [ ] Notification workflow tested +- [ ] Incident response plan tested +``` + +## Enterprise Tips + +### Start with Security + +**Priority 1:** Security requirements +``` +1. Document all security requirements +2. Generate security tests with *atdd +3. Run security test suite +4. Pass security audit BEFORE moving forward +``` + +**Why:** Security failures block everything in enterprise. + +**Example: RBAC Testing** + +**Vanilla Playwright:** +```typescript +test('should enforce role-based access', async ({ request }) => { + // Login as regular user + const userResp = await request.post('/api/auth/login', { + data: { email: 'user@example.com', password: 'pass' } + }); + const { token: userToken } = await userResp.json(); + + // Try to access admin endpoint + const adminResp = await request.get('/api/admin/users', { + headers: { Authorization: `Bearer ${userToken}` } + }); + + expect(adminResp.status()).toBe(403); // Forbidden +}); +``` + +**With Playwright Utils (Cleaner, Reusable):** +```typescript +import { test as base, expect } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; +import { mergeTests } from '@playwright/test'; + +const authFixtureTest = base.extend(createAuthFixtures()); +export const testWithAuth = mergeTests(apiRequestFixture, authFixtureTest); + +testWithAuth('should enforce role-based access', async ({ apiRequest, authToken }) => { + // Auth token from fixture (configured for 'user' role) + const { status } = await apiRequest({ + method: 'GET', + path: '/api/admin/users', // Admin endpoint + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(status).toBe(403); // Regular user denied +}); + +testWithAuth('admin can access admin endpoint', async ({ apiRequest, authToken, authOptions }) => { + // Override to admin role + authOptions.userIdentifier = 'admin'; + + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/admin/users', + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(status).toBe(200); // Admin allowed + expect(body).toBeInstanceOf(Array); +}); +``` + +**Note:** Auth-session requires provider setup in global-setup.ts. See [auth-session configuration](https://seontechnologies.github.io/playwright-utils/auth-session.html). + +**Playwright Utils Benefits for Compliance:** +- Multi-user auth testing (regular, admin, etc.) +- Token persistence (faster test execution) +- Consistent auth patterns (audit trail) +- Automatic cleanup + +### Set Higher Quality Thresholds + +**Enterprise quality targets:** +- Test coverage: >85% (vs 80% for non-enterprise) +- Quality score: >85 (vs 75 for non-enterprise) +- P0 coverage: 100% (non-negotiable) +- P1 coverage: >95% (vs 90% for non-enterprise) + +**Rationale:** Enterprise systems affect more users, higher stakes. + +### Document Everything + +**Auditors need:** +- Why decisions were made (rationale) +- Who approved (signatures) +- When (timestamps) +- What evidence (test results, scan reports) + +**Use TEA's structured outputs:** +- Reports have timestamps +- Decisions have rationale +- Evidence is referenced +- Audit trail is automatic + +### Budget for Compliance Testing + +**Enterprise testing costs more:** +- Penetration testing: $10k-50k +- Security audits: $5k-20k +- Performance testing tools: $500-5k/month +- Compliance consulting: $200-500/hour + +**Plan accordingly:** +- Budget in project cost +- Schedule early (3+ months for SOC 2) +- Don't skip (non-negotiable for compliance) + +### Use External Validators + +**Don't self-certify:** +- Penetration testing: Hire external firm +- Security audits: Independent auditor +- Compliance: Certification body +- Performance: Load testing service + +**TEA's role:** Prepare for external validation, don't replace it. + +## Related Guides + +**Workflow Guides:** +- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) - Deep dive on NFRs +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Gate decisions with evidence +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Quality audits +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Compliance-focused planning + +**Use-Case Guides:** +- [Using TEA with Existing Tests](/docs/how-to/brownfield/use-tea-with-existing-tests.md) - Brownfield patterns + +**Customization:** +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Production-ready utilities + +## Understanding the Concepts + +- [Engagement Models](/docs/explanation/tea/engagement-models.md) - Enterprise model explained +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Probability × impact scoring +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Enterprise quality thresholds +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Complete TEA lifecycle + +## Reference + +- [TEA Command Reference](/docs/reference/tea/commands.md) - All 8 workflows +- [TEA Configuration](/docs/reference/tea/configuration.md) - Enterprise config options +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Testing patterns +- [Glossary](/docs/reference/glossary/index.md#test-architect-tea-concepts) - TEA terminology + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-test-review.md b/docs/how-to/workflows/run-test-review.md new file mode 100644 index 00000000..924b12a7 --- /dev/null +++ b/docs/how-to/workflows/run-test-review.md @@ -0,0 +1,605 @@ +--- +title: "How to Run Test Review with TEA" +description: Audit test quality using TEA's comprehensive knowledge base and get 0-100 scoring +--- + +# How to Run Test Review with TEA + +Use TEA's `*test-review` workflow to audit test quality with objective scoring and actionable feedback. TEA reviews tests against its knowledge base of best practices. + +## When to Use This + +- Want to validate test quality objectively +- Need quality metrics for release gates +- Preparing for production deployment +- Reviewing team-written tests +- Auditing AI-generated tests +- Onboarding new team members (show good patterns) + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Tests written (to review) +- Test framework configured + +## Steps + +### 1. Load TEA Agent + +Start a fresh chat and load TEA: + +``` +*tea +``` + +### 2. Run the Test Review Workflow + +``` +*test-review +``` + +### 3. Specify Review Scope + +TEA will ask what to review. + +#### Option A: Single File + +Review one test file: + +``` +tests/e2e/checkout.spec.ts +``` + +**Best for:** +- Reviewing specific failing tests +- Quick feedback on new tests +- Learning from specific examples + +#### Option B: Directory + +Review all tests in a directory: + +``` +tests/e2e/ +``` + +**Best for:** +- Reviewing E2E test suite +- Comparing test quality across files +- Finding patterns of issues + +#### Option C: Entire Suite + +Review all tests: + +``` +tests/ +``` + +**Best for:** +- Release gate quality check +- Comprehensive audit +- Establishing baseline metrics + +### 4. Review the Quality Report + +TEA generates a comprehensive quality report with scoring. + +#### Report Structure (`test-review.md`): + +```markdown +# Test Quality Review Report + +**Date:** 2026-01-13 +**Scope:** tests/e2e/ +**Overall Score:** 76/100 + +## Summary + +- **Tests Reviewed:** 12 +- **Passing Quality:** 9 tests (75%) +- **Needs Improvement:** 3 tests (25%) +- **Critical Issues:** 2 +- **Recommendations:** 6 + +## Critical Issues + +### 1. Hard Waits Detected + +**File:** `tests/e2e/checkout.spec.ts:45` +**Issue:** Using `page.waitForTimeout(3000)` +**Impact:** Test is flaky and unnecessarily slow +**Severity:** Critical + +**Current Code:** +```typescript +await page.click('button[type="submit"]'); +await page.waitForTimeout(3000); // ❌ Hard wait +await expect(page.locator('.success')).toBeVisible(); +``` + +**Fix:** +```typescript +await page.click('button[type="submit"]'); +// Wait for the API response that triggers success message +await page.waitForResponse(resp => + resp.url().includes('/api/checkout') && resp.ok() +); +await expect(page.locator('.success')).toBeVisible(); +``` + +**Why This Matters:** +- Hard waits are fixed timeouts that don't wait for actual conditions +- Tests fail intermittently on slower machines +- Wastes time waiting even when response is fast +- Network-first patterns are more reliable + +--- + +### 2. Conditional Flow Control + +**File:** `tests/e2e/profile.spec.ts:28` +**Issue:** Using if/else to handle optional elements +**Impact:** Non-deterministic test behavior +**Severity:** Critical + +**Current Code:** +```typescript +if (await page.locator('.banner').isVisible()) { + await page.click('.dismiss'); +} +// ❌ Test behavior changes based on banner presence +``` + +**Fix:** +```typescript +// Option 1: Make banner presence deterministic +await expect(page.locator('.banner')).toBeVisible(); +await page.click('.dismiss'); + +// Option 2: Test both scenarios separately +test('should show banner for new users', async ({ page }) => { + // Test with banner +}); + +test('should not show banner for returning users', async ({ page }) => { + // Test without banner +}); +``` + +**Why This Matters:** +- Tests should be deterministic (same result every run) +- Conditionals hide bugs (what if banner should always show?) +- Makes debugging harder +- Violates test isolation principle + +## Recommendations + +### 1. Extract Repeated Setup + +**File:** `tests/e2e/profile.spec.ts` +**Issue:** Login code duplicated in every test +**Severity:** Medium +**Impact:** Maintenance burden, test verbosity + +**Current:** +```typescript +test('test 1', async ({ page }) => { + await page.goto('/login'); + await page.fill('[name="email"]', 'test@example.com'); + await page.fill('[name="password"]', 'password'); + await page.click('button[type="submit"]'); + // Test logic... +}); + +test('test 2', async ({ page }) => { + // Same login code repeated +}); +``` + +**Fix (Vanilla Playwright):** +```typescript +// Create fixture in tests/support/fixtures/auth.ts +import { test as base, Page } from '@playwright/test'; + +export const test = base.extend<{ authenticatedPage: Page }>({ + authenticatedPage: async ({ page }, use) => { + await page.goto('/login'); + await page.getByLabel('Email').fill('test@example.com'); + await page.getByLabel('Password').fill('password'); + await page.getByRole('button', { name: 'Sign in' }).click(); + await page.waitForURL(/\/dashboard/); + await use(page); + } +}); + +// Use in tests +test('test 1', async ({ authenticatedPage }) => { + // Already logged in +}); +``` + +**Better (With Playwright Utils):** +```typescript +// Use built-in auth-session fixture +import { test as base } from '@playwright/test'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; + +export const test = base.extend(createAuthFixtures()); + +// Use in tests - even simpler +test('test 1', async ({ page, authToken }) => { + // authToken already available (persisted, reused) + await page.goto('/dashboard'); + // Already authenticated via authToken +}); +``` + +**Playwright Utils Benefits:** +- Token persisted to disk (faster subsequent runs) +- Multi-user support out of the box +- Automatic token renewal if expired +- No manual login flow needed + +--- + +### 2. Add Network Assertions + +**File:** `tests/e2e/api-calls.spec.ts` +**Issue:** No verification of API responses +**Severity:** Low +**Impact:** Tests don't catch API errors + +**Current:** +```typescript +await page.click('button[name="save"]'); +await expect(page.locator('.success')).toBeVisible(); +// ❌ What if API returned 500 but UI shows cached success? +``` + +**Enhancement:** +```typescript +const responsePromise = page.waitForResponse( + resp => resp.url().includes('/api/profile') && resp.status() === 200 +); +await page.click('button[name="save"]'); +const response = await responsePromise; + +// Verify API response +const data = await response.json(); +expect(data.success).toBe(true); + +// Verify UI +await expect(page.locator('.success')).toBeVisible(); +``` + +--- + +### 3. Improve Test Names + +**File:** `tests/e2e/checkout.spec.ts` +**Issue:** Vague test names +**Severity:** Low +**Impact:** Hard to understand test purpose + +**Current:** +```typescript +test('should work', async ({ page }) => { }); +test('test checkout', async ({ page }) => { }); +``` + +**Better:** +```typescript +test('should complete checkout with valid credit card', async ({ page }) => { }); +test('should show validation error for expired card', async ({ page }) => { }); +``` + +## Quality Scores by Category + +| Category | Score | Target | Status | +|----------|-------|--------|--------| +| **Determinism** | 26/35 | 30/35 | ⚠️ Needs Improvement | +| **Isolation** | 22/25 | 20/25 | ✅ Good | +| **Assertions** | 18/20 | 16/20 | ✅ Good | +| **Structure** | 7/10 | 8/10 | ⚠️ Minor Issues | +| **Performance** | 3/10 | 8/10 | ❌ Critical | + +### Scoring Breakdown + +**Determinism (35 points max):** +- No hard waits: 0/10 ❌ (found 3 instances) +- No conditionals: 8/10 ⚠️ (found 2 instances) +- No try-catch flow control: 10/10 ✅ +- Network-first patterns: 8/15 ⚠️ (some tests missing) + +**Isolation (25 points max):** +- Self-cleaning: 20/20 ✅ +- No global state: 5/5 ✅ +- Parallel-safe: 0/0 ✅ (not tested) + +**Assertions (20 points max):** +- Explicit in test body: 15/15 ✅ +- Specific and meaningful: 3/5 ⚠️ (some weak assertions) + +**Structure (10 points max):** +- Test size < 300 lines: 5/5 ✅ +- Clear names: 2/5 ⚠️ (some vague names) + +**Performance (10 points max):** +- Execution time < 1.5 min: 3/10 ❌ (3 tests exceed limit) + +## Files Reviewed + +| File | Score | Issues | Status | +|------|-------|--------|--------| +| `tests/e2e/checkout.spec.ts` | 65/100 | 4 | ❌ Needs Work | +| `tests/e2e/profile.spec.ts` | 72/100 | 3 | ⚠️ Needs Improvement | +| `tests/e2e/search.spec.ts` | 88/100 | 1 | ✅ Good | +| `tests/api/profile.spec.ts` | 92/100 | 0 | ✅ Excellent | + +## Next Steps + +### Immediate (Fix Critical Issues) +1. Remove hard waits in `checkout.spec.ts` (line 45, 67, 89) +2. Fix conditional in `profile.spec.ts` (line 28) +3. Optimize slow tests in `checkout.spec.ts` + +### Short-term (Apply Recommendations) +4. Extract login fixture from `profile.spec.ts` +5. Add network assertions to `api-calls.spec.ts` +6. Improve test names in `checkout.spec.ts` + +### Long-term (Continuous Improvement) +7. Re-run `*test-review` after fixes (target: 85/100) +8. Add performance budgets to CI +9. Document test patterns for team + +## Knowledge Base References + +TEA reviewed against these patterns: +- [test-quality.md](/docs/reference/tea/knowledge-base.md#test-quality) - Execution limits, isolation +- [network-first.md](/docs/reference/tea/knowledge-base.md#network-first) - Deterministic waits +- [timing-debugging.md](/docs/reference/tea/knowledge-base.md#timing-debugging) - Race conditions +- [selector-resilience.md](/docs/reference/tea/knowledge-base.md#selector-resilience) - Robust selectors +``` + +## Understanding the Scores + +### What Do Scores Mean? + +| Score Range | Interpretation | Action | +|-------------|----------------|--------| +| **90-100** | Excellent | Minimal changes needed, production-ready | +| **80-89** | Good | Minor improvements recommended | +| **70-79** | Acceptable | Address recommendations before release | +| **60-69** | Needs Improvement | Fix critical issues, apply recommendations | +| **< 60** | Critical | Significant refactoring needed | + +### Scoring Criteria + +**Determinism (35 points):** +- Tests produce same result every run +- No random failures (flakiness) +- No environment-dependent behavior + +**Isolation (25 points):** +- Tests don't depend on each other +- Can run in any order +- Clean up after themselves + +**Assertions (20 points):** +- Verify actual behavior +- Specific and meaningful +- Not abstracted away in helpers + +**Structure (10 points):** +- Readable and maintainable +- Appropriate size +- Clear naming + +**Performance (10 points):** +- Fast execution +- Efficient selectors +- No unnecessary waits + +## What You Get + +### Quality Report +- Overall score (0-100) +- Category scores (Determinism, Isolation, etc.) +- File-by-file breakdown + +### Critical Issues +- Specific line numbers +- Code examples (current vs fixed) +- Why it matters explanation +- Impact assessment + +### Recommendations +- Actionable improvements +- Code examples +- Priority/severity levels + +### Next Steps +- Immediate actions (fix critical) +- Short-term improvements +- Long-term quality goals + +## Tips + +### Review Before Release + +Make test review part of release checklist: + +```markdown +## Release Checklist +- [ ] All tests passing +- [ ] Test review score > 80 +- [ ] Critical issues resolved +- [ ] Performance within budget +``` + +### Review After AI Generation + +Always review AI-generated tests: + +``` +1. Run *atdd or *automate +2. Run *test-review on generated tests +3. Fix critical issues +4. Commit tests +``` + +### Set Quality Gates + +Use scores as quality gates: + +```yaml +# .github/workflows/test.yml +- name: Review test quality + run: | + # Run test review + # Parse score from report + if [ $SCORE -lt 80 ]; then + echo "Test quality below threshold" + exit 1 + fi +``` + +### Review Regularly + +Schedule periodic reviews: + +- **Per story:** Optional (spot check new tests) +- **Per epic:** Recommended (ensure consistency) +- **Per release:** Recommended for quality gates (required if using formal gate process) +- **Quarterly:** Audit entire suite + +### Focus Reviews + +For large suites, review incrementally: + +**Week 1:** Review E2E tests +**Week 2:** Review API tests +**Week 3:** Review component tests (Cypress CT or Vitest) +**Week 4:** Apply fixes across all suites + +**Component Testing Note:** TEA reviews component tests using framework-specific knowledge: +- **Cypress:** Reviews Cypress Component Testing specs (*.cy.tsx) +- **Playwright:** Reviews Vitest component tests (*.test.tsx) + +### Use Reviews for Learning + +Share reports with team: + +``` +Team Meeting: +- Review test-review.md +- Discuss critical issues +- Agree on patterns +- Update team guidelines +``` + +### Compare Over Time + +Track improvement: + +```markdown +## Quality Trend + +| Date | Score | Critical Issues | Notes | +|------|-------|-----------------|-------| +| 2026-01-01 | 65 | 5 | Baseline | +| 2026-01-15 | 72 | 2 | Fixed hard waits | +| 2026-02-01 | 84 | 0 | All critical resolved | +``` + +## Common Issues + +### Low Determinism Score + +**Symptoms:** +- Tests fail randomly +- "Works on my machine" +- CI failures that don't reproduce locally + +**Common Causes:** +- Hard waits (`waitForTimeout`) +- Conditional flow control (`if/else`) +- Try-catch for flow control +- Missing network-first patterns + +**Fix:** Review determinism section, apply network-first patterns + +### Low Performance Score + +**Symptoms:** +- Tests take > 1.5 minutes each +- Test suite takes hours +- CI times out + +**Common Causes:** +- Unnecessary waits (hard timeouts) +- Inefficient selectors (XPath, complex CSS) +- Not using parallelization +- Heavy setup in every test + +**Fix:** Optimize waits, improve selectors, use fixtures + +### Low Isolation Score + +**Symptoms:** +- Tests fail when run in different order +- Tests fail in parallel +- Test data conflicts + +**Common Causes:** +- Shared global state +- Tests don't clean up +- Hard-coded test data +- Database not reset between tests + +**Fix:** Use fixtures, clean up in afterEach, use unique test data + +### "Too Many Issues to Fix" + +**Problem:** Report shows 50+ issues, overwhelming. + +**Solution:** Prioritize: +1. Fix all critical issues first +2. Apply top 3 recommendations +3. Re-run review +4. Iterate + +Don't try to fix everything at once. + +### Reviews Take Too Long + +**Problem:** Reviewing entire suite takes hours. + +**Solution:** Review incrementally: +- Review new tests in PR review +- Schedule directory reviews weekly +- Full suite review quarterly + +## Related Guides + +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate tests to review +- [How to Run Automate](/docs/how-to/workflows/run-automate.md) - Expand coverage to review +- [How to Run Trace](/docs/how-to/workflows/run-trace.md) - Coverage complements quality + +## Understanding the Concepts + +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - What makes tests good +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Avoiding flakiness +- [Fixture Architecture](/docs/explanation/tea/fixture-architecture.md) - Reusable patterns + +## Reference + +- [Command: *test-review](/docs/reference/tea/commands.md#test-review) - Full command reference +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) - Patterns TEA reviews against + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/run-trace.md b/docs/how-to/workflows/run-trace.md new file mode 100644 index 00000000..f35403da --- /dev/null +++ b/docs/how-to/workflows/run-trace.md @@ -0,0 +1,883 @@ +--- +title: "How to Run Trace with TEA" +description: Map requirements to tests and make quality gate decisions using TEA's trace workflow +--- + +# How to Run Trace with TEA + +Use TEA's `*trace` workflow for requirements traceability and quality gate decisions. This is a two-phase workflow: Phase 1 analyzes coverage, Phase 2 makes the go/no-go decision. + +## When to Use This + +### Phase 1: Requirements Traceability +- Map acceptance criteria to implemented tests +- Identify coverage gaps +- Prioritize missing tests +- Refresh coverage after each story/epic + +### Phase 2: Quality Gate Decision +- Make go/no-go decision for release +- Validate coverage meets thresholds +- Document gate decision with evidence +- Support business-approved waivers + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Requirements defined (stories, acceptance criteria, test design) +- Tests implemented +- For brownfield: Existing codebase with tests + +## Steps + +### 1. Run the Trace Workflow + +``` +*trace +``` + +### 2. Specify Phase + +TEA will ask which phase you're running. + +**Phase 1: Requirements Traceability** +- Analyze coverage +- Identify gaps +- Generate recommendations + +**Phase 2: Quality Gate Decision** +- Make PASS/CONCERNS/FAIL/WAIVED decision +- Requires Phase 1 complete + +**Typical flow:** Run Phase 1 first, review gaps, then run Phase 2 for gate decision. + +--- + +## Phase 1: Requirements Traceability + +### 3. Provide Requirements Source + +TEA will ask where requirements are defined. + +**Options:** + +| Source | Example | Best For | +|--------|---------|----------| +| **Story file** | `story-profile-management.md` | Single story coverage | +| **Test design** | `test-design-epic-1.md` | Epic coverage | +| **PRD** | `PRD.md` | System-level coverage | +| **Multiple** | All of the above | Comprehensive analysis | + +**Example Response:** +``` +Requirements: +- story-profile-management.md (acceptance criteria) +- test-design-epic-1.md (test priorities) +``` + +### 4. Specify Test Location + +TEA will ask where tests are located. + +**Example:** +``` +Test location: tests/ +Include: +- tests/api/ +- tests/e2e/ +``` + +### 5. Specify Focus Areas (Optional) + +**Example:** +``` +Focus on: +- Profile CRUD operations +- Validation scenarios +- Authorization checks +``` + +### 6. Review Coverage Matrix + +TEA generates a comprehensive traceability matrix. + +#### Traceability Matrix (`traceability-matrix.md`): + +```markdown +# Requirements Traceability Matrix + +**Date:** 2026-01-13 +**Scope:** Epic 1 - User Profile Management +**Phase:** Phase 1 (Traceability Analysis) + +## Coverage Summary + +| Metric | Count | Percentage | +|--------|-------|------------| +| **Total Requirements** | 15 | 100% | +| **Full Coverage** | 11 | 73% | +| **Partial Coverage** | 3 | 20% | +| **No Coverage** | 1 | 7% | + +### By Priority + +| Priority | Total | Covered | Percentage | +|----------|-------|---------|------------| +| **P0** | 5 | 5 | 100% ✅ | +| **P1** | 6 | 5 | 83% ⚠️ | +| **P2** | 3 | 1 | 33% ⚠️ | +| **P3** | 1 | 0 | 0% ✅ (acceptable) | + +--- + +## Detailed Traceability + +### ✅ Requirement 1: User can view their profile (P0) + +**Acceptance Criteria:** +- User navigates to /profile +- Profile displays name, email, avatar +- Data is current (not cached) + +**Test Coverage:** FULL ✅ + +**Tests:** +- `tests/e2e/profile-view.spec.ts:15` - "should display profile page with current data" + - ✅ Navigates to /profile + - ✅ Verifies name, email visible + - ✅ Verifies avatar displayed + - ✅ Validates data freshness via API assertion + +- `tests/api/profile.spec.ts:8` - "should fetch user profile via API" + - ✅ Calls GET /api/profile + - ✅ Validates response schema + - ✅ Confirms all fields present + +--- + +### ⚠️ Requirement 2: User can edit profile (P0) + +**Acceptance Criteria:** +- User clicks "Edit Profile" +- Can modify name, email, bio +- Can upload avatar +- Changes are persisted +- Success message shown + +**Test Coverage:** PARTIAL ⚠️ + +**Tests:** +- `tests/e2e/profile-edit.spec.ts:22` - "should edit and save profile" + - ✅ Clicks edit button + - ✅ Modifies name and email + - ⚠️ **Does NOT test bio field** + - ❌ **Does NOT test avatar upload** + - ✅ Verifies persistence + - ✅ Verifies success message + +- `tests/api/profile.spec.ts:25` - "should update profile via PATCH" + - ✅ Calls PATCH /api/profile + - ✅ Validates update response + - ⚠️ **Only tests name/email, not bio/avatar** + +**Missing Coverage:** +- Bio field not tested in E2E or API +- Avatar upload not tested + +**Gap Severity:** HIGH (P0 requirement, critical path) + +--- + +### ✅ Requirement 3: Invalid email shows validation error (P1) + +**Acceptance Criteria:** +- Enter invalid email format +- See error message +- Cannot save changes + +**Test Coverage:** FULL ✅ + +**Tests:** +- `tests/e2e/profile-edit.spec.ts:45` - "should show validation error for invalid email" +- `tests/api/profile.spec.ts:50` - "should return 400 for invalid email" + +--- + +### ❌ Requirement 15: Profile export as PDF (P2) + +**Acceptance Criteria:** +- User clicks "Export Profile" +- PDF downloads with profile data + +**Test Coverage:** NONE ❌ + +**Gap Analysis:** +- **Priority:** P2 (medium) +- **Risk:** Low (non-critical feature) +- **Recommendation:** Add in next iteration (not blocking for release) + +--- + +## Gap Prioritization + +### Critical Gaps (Must Fix Before Release) + +| Gap | Requirement | Priority | Risk | Recommendation | +|-----|-------------|----------|------|----------------| +| 1 | Bio field not tested | P0 | High | Add E2E + API tests | +| 2 | Avatar upload not tested | P0 | High | Add E2E + API tests | + +**Estimated Effort:** 3 hours +**Owner:** QA team +**Deadline:** Before release + +### Non-Critical Gaps (Can Defer) + +| Gap | Requirement | Priority | Risk | Recommendation | +|-----|-------------|----------|------|----------------| +| 3 | Profile export not tested | P2 | Low | Add in v1.3 release | + +**Estimated Effort:** 2 hours +**Owner:** QA team +**Deadline:** Next release (February) + +--- + +## Recommendations + +### 1. Add Bio Field Tests + +**Tests Needed (Vanilla Playwright):** +```typescript +// tests/e2e/profile-edit.spec.ts +test('should edit bio field', async ({ page }) => { + await page.goto('/profile'); + await page.getByRole('button', { name: 'Edit' }).click(); + await page.getByLabel('Bio').fill('New bio text'); + await page.getByRole('button', { name: 'Save' }).click(); + await expect(page.getByText('New bio text')).toBeVisible(); +}); + +// tests/api/profile.spec.ts +test('should update bio via API', async ({ request }) => { + const response = await request.patch('/api/profile', { + data: { bio: 'Updated bio' } + }); + expect(response.ok()).toBeTruthy(); + const { bio } = await response.json(); + expect(bio).toBe('Updated bio'); +}); +``` + +**With Playwright Utils:** +```typescript +// tests/e2e/profile-edit.spec.ts +import { test } from '../support/fixtures'; // Composed with authToken + +test('should edit bio field', async ({ page, authToken }) => { + await page.goto('/profile'); + await page.getByRole('button', { name: 'Edit' }).click(); + await page.getByLabel('Bio').fill('New bio text'); + await page.getByRole('button', { name: 'Save' }).click(); + await expect(page.getByText('New bio text')).toBeVisible(); +}); + +// tests/api/profile.spec.ts +import { test as base, expect } from '@playwright/test'; +import { test as apiRequestFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { createAuthFixtures } from '@seontechnologies/playwright-utils/auth-session'; +import { mergeTests } from '@playwright/test'; + +// Merge API request + auth fixtures +const authFixtureTest = base.extend(createAuthFixtures()); +const test = mergeTests(apiRequestFixture, authFixtureTest); + +test('should update bio via API', async ({ apiRequest, authToken }) => { + const { status, body } = await apiRequest({ + method: 'PATCH', + path: '/api/profile', + body: { bio: 'Updated bio' }, // 'body' not 'data' + headers: { Authorization: `Bearer ${authToken}` } + }); + + expect(status).toBe(200); + expect(body.bio).toBe('Updated bio'); +}); +``` + +**Note:** `authToken` requires auth-session fixture setup. See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md#auth-session). + +### 2. Add Avatar Upload Tests + +**Tests Needed:** +```typescript +// tests/e2e/profile-edit.spec.ts +test('should upload avatar image', async ({ page }) => { + await page.goto('/profile'); + await page.getByRole('button', { name: 'Edit' }).click(); + + // Upload file + await page.setInputFiles('[type="file"]', 'fixtures/avatar.png'); + await page.getByRole('button', { name: 'Save' }).click(); + + // Verify uploaded image displays + await expect(page.locator('img[alt="Profile avatar"]')).toBeVisible(); +}); + +// tests/api/profile.spec.ts +import { test, expect } from '@playwright/test'; +import fs from 'fs/promises'; + +test('should accept valid image upload', async ({ request }) => { + const response = await request.post('/api/profile/avatar', { + multipart: { + file: { + name: 'avatar.png', + mimeType: 'image/png', + buffer: await fs.readFile('fixtures/avatar.png') + } + } + }); + expect(response.ok()).toBeTruthy(); +}); +``` + +--- + +## Next Steps + +After reviewing traceability: + +1. **Fix critical gaps** - Add tests for P0/P1 requirements +2. **Run *test-review** - Ensure new tests meet quality standards +3. **Run Phase 2** - Make gate decision after gaps addressed +``` + +--- + +## Phase 2: Quality Gate Decision + +After Phase 1 coverage analysis is complete, run Phase 2 for the gate decision. + +**Prerequisites:** +- Phase 1 traceability matrix complete +- Test execution results available (must have test results) + +**Note:** Phase 2 will skip if test execution results aren't provided. The workflow requires actual test run results to make gate decisions. + +### 7. Run Phase 2 + +``` +*trace +``` + +Select "Phase 2: Quality Gate Decision" + +### 8. Provide Additional Context + +TEA will ask for: + +**Gate Type:** +- Story gate (small release) +- Epic gate (larger release) +- Release gate (production deployment) +- Hotfix gate (emergency fix) + +**Decision Mode:** +- **Deterministic** - Rule-based (coverage %, quality scores) +- **Manual** - Team decision with TEA guidance + +**Example:** +``` +Gate type: Epic gate +Decision mode: Deterministic +``` + +### 9. Provide Supporting Evidence + +TEA will request: + +**Phase 1 Results:** +``` +traceability-matrix.md (from Phase 1) +``` + +**Test Quality (Optional):** +``` +test-review.md (from *test-review) +``` + +**NFR Assessment (Optional):** +``` +nfr-assessment.md (from *nfr-assess) +``` + +### 10. Review Gate Decision + +TEA makes evidence-based gate decision and writes to separate file. + +#### Gate Decision (`gate-decision-{gate_type}-{story_id}.md`): + +```markdown +--- + +# Phase 2: Quality Gate Decision + +**Gate Type:** Epic Gate +**Decision:** PASS ✅ +**Date:** 2026-01-13 +**Approvers:** Product Manager, Tech Lead, QA Lead + +## Decision Summary + +**Verdict:** Ready to release + +**Evidence:** +- P0 coverage: 100% (5/5 requirements) +- P1 coverage: 100% (6/6 requirements) +- P2 coverage: 33% (1/3 requirements) - acceptable +- Test quality score: 84/100 +- NFR assessment: PASS + +## Coverage Analysis + +| Priority | Required Coverage | Actual Coverage | Status | +|----------|------------------|-----------------|--------| +| **P0** | 100% | 100% | ✅ PASS | +| **P1** | 90% | 100% | ✅ PASS | +| **P2** | 50% | 33% | ⚠️ Below (acceptable) | +| **P3** | 20% | 0% | ✅ PASS (low priority) | + +**Rationale:** +- All critical path (P0) requirements fully tested +- All high-value (P1) requirements fully tested +- P2 gap (profile export) is low risk and deferred to next release + +## Quality Metrics + +| Metric | Threshold | Actual | Status | +|--------|-----------|--------|--------| +| P0/P1 Coverage | >95% | 100% | ✅ | +| Test Quality Score | >80 | 84 | ✅ | +| NFR Status | PASS | PASS | ✅ | + +## Risks and Mitigations + +### Accepted Risks + +**Risk 1: Profile export not tested (P2)** +- **Impact:** Medium (users can't export profile) +- **Mitigation:** Feature flag disabled by default +- **Plan:** Add tests in v1.3 release (February) +- **Monitoring:** Track feature flag usage + +## Approvals + +- [x] **Product Manager** - Business requirements met (Approved: 2026-01-13) +- [x] **Tech Lead** - Technical quality acceptable (Approved: 2026-01-13) +- [x] **QA Lead** - Test coverage sufficient (Approved: 2026-01-13) + +## Next Steps + +### Deployment +1. Merge to main branch +2. Deploy to staging +3. Run smoke tests in staging +4. Deploy to production +5. Monitor for 24 hours + +### Monitoring +- Set alerts for profile endpoint (P99 > 200ms) +- Track error rates (target: <0.1%) +- Monitor profile export feature flag usage + +### Future Work +- Add profile export tests (v1.3) +- Expand P2 coverage to 50% +``` + +### Gate Decision Rules + +TEA uses deterministic rules when decision_mode = "deterministic": + +| P0 Coverage | P1 Coverage | Overall Coverage | Decision | +|-------------|-------------|------------------|----------| +| 100% | ≥90% | ≥80% | **PASS** ✅ | +| 100% | 80-89% | ≥80% | **CONCERNS** ⚠️ | +| <100% | Any | Any | **FAIL** ❌ | +| Any | <80% | Any | **FAIL** ❌ | +| Any | Any | <80% | **FAIL** ❌ | +| Any | Any | Any | **WAIVED** ⏭️ (with approval) | + +**Detailed Rules:** +- **PASS:** P0=100%, P1≥90%, Overall≥80% +- **CONCERNS:** P0=100%, P1 80-89%, Overall≥80% (below threshold but not critical) +- **FAIL:** P0<100% OR P1<80% OR Overall<80% (critical gaps) + +**PASS** ✅: All criteria met, ready to release + +**CONCERNS** ⚠️: Some criteria not met, but: +- Mitigation plan exists +- Risk is acceptable +- Team approves proceeding +- Monitoring in place + +**FAIL** ❌: Critical criteria not met: +- P0 requirements not tested +- Critical security vulnerabilities +- System is broken +- Cannot deploy + +**WAIVED** ⏭️: Business approves proceeding despite concerns: +- Documented business justification +- Accepted risks quantified +- Approver signatures +- Future plans documented + +### Example CONCERNS Decision + +```markdown +## Decision Summary + +**Verdict:** CONCERNS ⚠️ - Proceed with monitoring + +**Evidence:** +- P0 coverage: 100% +- P1 coverage: 85% (below 90% target) +- Test quality: 78/100 (below 80 target) + +**Gaps:** +- 1 P1 requirement not tested (avatar upload) +- Test quality score slightly below threshold + +**Mitigation:** +- Avatar upload not critical for v1.2 launch +- Test quality issues are minor (no flakiness) +- Monitoring alerts configured + +**Approvals:** +- Product Manager: APPROVED (business priority to launch) +- Tech Lead: APPROVED (technical risk acceptable) +``` + +### Example FAIL Decision + +```markdown +## Decision Summary + +**Verdict:** FAIL ❌ - Cannot release + +**Evidence:** +- P0 coverage: 60% (below 95% threshold) +- Critical security vulnerability (CVE-2024-12345) +- Test quality: 55/100 + +**Blockers:** +1. **Login flow not tested** (P0 requirement) + - Critical path completely untested + - Must add E2E and API tests + +2. **SQL injection vulnerability** + - Critical security issue + - Must fix before deployment + +**Actions Required:** +1. Add login tests (QA team, 2 days) +2. Fix SQL injection (backend team, 1 day) +3. Re-run security scan (DevOps, 1 hour) +4. Re-run *trace after fixes + +**Cannot proceed until all blockers resolved.** +``` + +## What You Get + +### Phase 1: Traceability Matrix +- Requirement-to-test mapping +- Coverage classification (FULL/PARTIAL/NONE) +- Gap identification with priorities +- Actionable recommendations + +### Phase 2: Gate Decision +- Go/no-go verdict (PASS/CONCERNS/FAIL/WAIVED) +- Evidence summary +- Approval signatures +- Next steps and monitoring plan + +## Usage Patterns + +### Greenfield Projects + +**Phase 3:** +``` +After architecture complete: +1. Run *test-design (system-level) +2. Run *trace Phase 1 (baseline) +3. Use for implementation-readiness gate +``` + +**Phase 4:** +``` +After each epic/story: +1. Run *trace Phase 1 (refresh coverage) +2. Identify gaps +3. Add missing tests +``` + +**Release Gate:** +``` +Before deployment: +1. Run *trace Phase 1 (final coverage check) +2. Run *trace Phase 2 (make gate decision) +3. Get approvals +4. Deploy (if PASS or WAIVED) +``` + +### Brownfield Projects + +**Phase 2:** +``` +Before planning new work: +1. Run *trace Phase 1 (establish baseline) +2. Understand existing coverage +3. Plan testing strategy +``` + +**Phase 4:** +``` +After each epic/story: +1. Run *trace Phase 1 (refresh) +2. Compare to baseline +3. Track coverage improvement +``` + +**Release Gate:** +``` +Before deployment: +1. Run *trace Phase 1 (final check) +2. Run *trace Phase 2 (gate decision) +3. Compare to baseline +4. Deploy if coverage maintained or improved +``` + +## Tips + +### Run Phase 1 Frequently + +Don't wait until release gate: + +``` +After Story 1: *trace Phase 1 (identify gaps early) +After Story 2: *trace Phase 1 (refresh) +After Story 3: *trace Phase 1 (refresh) +Before Release: *trace Phase 1 + Phase 2 (final gate) +``` + +**Benefit:** Catch gaps early when they're cheap to fix. + +### Use Coverage Trends + +Track improvement over time: + +```markdown +## Coverage Trend + +| Date | Epic | P0/P1 Coverage | Quality Score | Status | +|------|------|----------------|---------------|--------| +| 2026-01-01 | Baseline | 45% | - | Starting point | +| 2026-01-08 | Epic 1 | 78% | 72 | Improving | +| 2026-01-15 | Epic 2 | 92% | 84 | Near target | +| 2026-01-20 | Epic 3 | 100% | 88 | Ready! | +``` + +### Set Coverage Targets by Priority + +Don't aim for 100% across all priorities: + +**Recommended Targets:** +- **P0:** 100% (critical path must be tested) +- **P1:** 90% (high-value scenarios) +- **P2:** 50% (nice-to-have features) +- **P3:** 20% (low-value edge cases) + +### Use Classification Strategically + +**FULL** ✅: Requirement completely tested +- E2E test covers full user workflow +- API test validates backend behavior +- All acceptance criteria covered + +**PARTIAL** ⚠️: Some aspects tested +- E2E test exists but missing scenarios +- API test exists but incomplete +- Some acceptance criteria not covered + +**NONE** ❌: No tests exist +- Requirement identified but not tested +- May be intentional (low priority) or oversight + +**Classification helps prioritize:** +- Fix NONE coverage for P0/P1 requirements first +- Enhance PARTIAL coverage for P0 requirements +- Accept PARTIAL or NONE for P2/P3 if time-constrained + +### Automate Gate Decisions + +Use traceability in CI: + +```yaml +# .github/workflows/gate-check.yml +- name: Check coverage + run: | + # Run trace Phase 1 + # Parse coverage percentages + if [ $P0_COVERAGE -lt 95 ]; then + echo "P0 coverage below 95%" + exit 1 + fi +``` + +### Document Waivers Clearly + +If proceeding with WAIVED: + +**Required:** +```markdown +## Waiver Documentation + +**Waived By:** VP Engineering, Product Lead +**Date:** 2026-01-15 +**Gate Type:** Release Gate v1.2 + +**Justification:** +Business critical to launch by Q1 for investor demo. +Performance concerns acceptable for initial user base. + +**Conditions:** +- Set monitoring alerts for P99 > 300ms +- Plan optimization for v1.3 (due February 28) +- Monitor user feedback closely + +**Accepted Risks:** +- 1% of users may experience 350ms latency +- Avatar upload feature incomplete +- Profile export deferred to next release + +**Quantified Impact:** +- Affects <100 users at current scale +- Workaround exists (manual export) +- Monitoring will catch issues early + +**Approvals:** +- VP Engineering: [Signature] Date: 2026-01-15 +- Product Lead: [Signature] Date: 2026-01-15 +- QA Lead: [Signature] Date: 2026-01-15 +``` + +## Common Issues + +### Too Many Gaps to Fix + +**Problem:** Phase 1 shows 50 uncovered requirements. + +**Solution:** Prioritize ruthlessly: +1. Fix all P0 gaps (critical path) +2. Fix high-risk P1 gaps +3. Accept low-risk P1 gaps with mitigation +4. Defer all P2/P3 gaps + +**Don't try to fix everything** - focus on what matters for release. + +### Can't Find Test Coverage + +**Problem:** Tests exist but TEA can't map them to requirements. + +**Cause:** Tests don't reference requirements. + +**Solution:** Add traceability comments: +```typescript +test('should display profile', async ({ page }) => { + // Covers: Requirement 1 - User can view profile + // Acceptance criteria: Navigate to /profile, see name/email + await page.goto('/profile'); + await expect(page.getByText('Test User')).toBeVisible(); +}); +``` + +Or use test IDs: +```typescript +test('[REQ-1] should display profile', async ({ page }) => { + // Test code... +}); +``` + +### Unclear What "FULL" vs "PARTIAL" Means + +**FULL** ✅: All acceptance criteria tested +``` +Requirement: User can edit profile +Acceptance criteria: + - Can modify name ✅ Tested + - Can modify email ✅ Tested + - Can upload avatar ✅ Tested + - Changes persist ✅ Tested +Result: FULL coverage +``` + +**PARTIAL** ⚠️: Some criteria tested, some not +``` +Requirement: User can edit profile +Acceptance criteria: + - Can modify name ✅ Tested + - Can modify email ✅ Tested + - Can upload avatar ❌ Not tested + - Changes persist ✅ Tested +Result: PARTIAL coverage (3/4 criteria) +``` + +### Gate Decision Unclear + +**Problem:** Not sure if PASS or CONCERNS is appropriate. + +**Guideline:** + +**Use PASS** ✅ if: +- All P0 requirements 100% covered +- P1 requirements >90% covered +- No critical issues +- NFRs met + +**Use CONCERNS** ⚠️ if: +- P1 coverage 85-90% (close to threshold) +- Minor quality issues (score 70-79) +- NFRs have mitigation plans +- Team agrees risk is acceptable + +**Use FAIL** ❌ if: +- P0 coverage <100% (critical path gaps) +- P1 coverage <85% +- Critical security/performance issues +- No mitigation possible + +**When in doubt, use CONCERNS** and document the risk. + +## Related Guides + +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Provides requirements for traceability +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Quality scores feed gate +- [How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) - NFR status feeds gate + +## Understanding the Concepts + +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - Why P0 vs P3 matters +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Gate decisions in context + +## Reference + +- [Command: *trace](/docs/reference/tea/commands.md#trace) - Full command reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config options + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/how-to/workflows/setup-ci.md b/docs/how-to/workflows/setup-ci.md new file mode 100644 index 00000000..fc68a08b --- /dev/null +++ b/docs/how-to/workflows/setup-ci.md @@ -0,0 +1,765 @@ +--- +title: "How to Set Up CI Pipeline with TEA" +description: Configure automated test execution with selective testing and burn-in loops using TEA +--- + +# How to Set Up CI Pipeline with TEA + +Use TEA's `*ci` workflow to scaffold production-ready CI/CD configuration for automated test execution with selective testing, parallel sharding, and flakiness detection. + +## When to Use This + +- Need to automate test execution in CI/CD +- Want selective testing (only run affected tests) +- Need parallel execution for faster feedback +- Want burn-in loops for flakiness detection +- Setting up new CI/CD pipeline +- Optimizing existing CI/CD workflow + +## Prerequisites + +- BMad Method installed +- TEA agent available +- Test framework configured (run `*framework` first) +- Tests written (have something to run in CI) +- CI/CD platform access (GitHub Actions, GitLab CI, etc.) + +## Steps + +### 1. Load TEA Agent + +Start a fresh chat and load TEA: + +``` +*tea +``` + +### 2. Run the CI Workflow + +``` +*ci +``` + +### 3. Select CI/CD Platform + +TEA will ask which platform you're using. + +**Supported Platforms:** +- **GitHub Actions** (most common) +- **GitLab CI** +- **Circle CI** +- **Jenkins** +- **Other** (TEA provides generic template) + +**Example:** +``` +GitHub Actions +``` + +### 4. Configure Test Strategy + +TEA will ask about your test execution strategy. + +#### Repository Structure + +**Question:** "What's your repository structure?" + +**Options:** +- **Single app** - One application in root +- **Monorepo** - Multiple apps/packages +- **Monorepo with affected detection** - Only test changed packages + +**Example:** +``` +Monorepo with multiple apps +Need selective testing for changed packages only +``` + +#### Parallel Execution + +**Question:** "Want to shard tests for parallel execution?" + +**Options:** +- **No sharding** - Run tests sequentially +- **Shard by workers** - Split across N workers +- **Shard by file** - Each file runs in parallel + +**Example:** +``` +Yes, shard across 4 workers for faster execution +``` + +**Why Shard?** +- **4 workers:** 20-minute suite → 5 minutes +- **Better resource usage:** Utilize CI runners efficiently +- **Faster feedback:** Developers wait less + +#### Burn-In Loops + +**Question:** "Want burn-in loops for flakiness detection?" + +**Options:** +- **No burn-in** - Run tests once +- **PR burn-in** - Run tests multiple times on PRs +- **Nightly burn-in** - Dedicated flakiness detection job + +**Example:** +``` +Yes, run tests 5 times on PRs to catch flaky tests early +``` + +**Why Burn-In?** +- Catches flaky tests before they merge +- Prevents intermittent CI failures +- Builds confidence in test suite + +### 5. Review Generated CI Configuration + +TEA generates platform-specific workflow files. + +#### GitHub Actions (`.github/workflows/test.yml`): + +```yaml +name: Test Suite + +on: + pull_request: + push: + branches: [main, develop] + schedule: + - cron: '0 2 * * *' # Nightly at 2 AM + +jobs: + # Main test job with sharding + test: + name: Test (Shard ${{ matrix.shard }}) + runs-on: ubuntu-latest + timeout-minutes: 15 + + strategy: + fail-fast: false + matrix: + shard: [1, 2, 3, 4] + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: '.nvmrc' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Install Playwright browsers + run: npx playwright install --with-deps + + - name: Run tests + run: npx playwright test --shard=${{ matrix.shard }}/4 + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-results-${{ matrix.shard }} + path: test-results/ + retention-days: 7 + + - name: Upload test report + if: always() + uses: actions/upload-artifact@v4 + with: + name: playwright-report-${{ matrix.shard }} + path: playwright-report/ + retention-days: 7 + + # Burn-in job for flakiness detection (PRs only) + burn-in: + name: Burn-In (Flakiness Detection) + runs-on: ubuntu-latest + if: github.event_name == 'pull_request' + timeout-minutes: 30 + + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: '.nvmrc' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Install Playwright browsers + run: npx playwright install --with-deps + + - name: Run burn-in loop + run: | + for i in {1..5}; do + echo "=== Burn-in iteration $i/5 ===" + npx playwright test --grep-invert "@skip" || exit 1 + done + + - name: Upload burn-in results + if: failure() + uses: actions/upload-artifact@v4 + with: + name: burn-in-failures + path: test-results/ + + # Selective testing (changed files only) + selective: + name: Selective Tests + runs-on: ubuntu-latest + if: github.event_name == 'pull_request' + + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for git diff + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: '.nvmrc' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Install Playwright browsers + run: npx playwright install --with-deps + + - name: Run selective tests + run: npm run test:changed +``` + +#### GitLab CI (`.gitlab-ci.yml`): + +```yaml +variables: + NODE_VERSION: "18" + +stages: + - test + - burn-in + +# Test job with parallel execution +test: + stage: test + image: node:$NODE_VERSION + parallel: 4 + script: + - npm ci + - npx playwright install --with-deps + - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL + artifacts: + when: always + paths: + - test-results/ + - playwright-report/ + expire_in: 7 days + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + +# Burn-in job for flakiness detection +burn-in: + stage: burn-in + image: node:$NODE_VERSION + script: + - npm ci + - npx playwright install --with-deps + - | + for i in {1..5}; do + echo "=== Burn-in iteration $i/5 ===" + npx playwright test || exit 1 + done + artifacts: + when: on_failure + paths: + - test-results/ + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" +``` + +#### Helper Scripts + +TEA generates shell scripts for CI and local development. + +**Test Scripts** (`package.json`): + +```json +{ + "scripts": { + "test": "playwright test", + "test:headed": "playwright test --headed", + "test:debug": "playwright test --debug", + "test:smoke": "playwright test --grep @smoke", + "test:critical": "playwright test --grep @critical", + "test:changed": "./scripts/test-changed.sh", + "test:burn-in": "./scripts/burn-in.sh", + "test:report": "playwright show-report", + "ci:local": "./scripts/ci-local.sh" + } +} +``` + +**Selective Testing Script** (`scripts/test-changed.sh`): + +```bash +#!/bin/bash +# Run only tests for changed files + +CHANGED_FILES=$(git diff --name-only origin/main...HEAD) + +if echo "$CHANGED_FILES" | grep -q "src/.*\.ts$"; then + echo "Running affected tests..." + npm run test:e2e -- --grep="$(echo $CHANGED_FILES | sed 's/src\///g' | sed 's/\.ts//g')" +else + echo "No test-affecting changes detected" +fi +``` + +**Burn-In Script** (`scripts/burn-in.sh`): + +```bash +#!/bin/bash +# Run tests multiple times to detect flakiness + +ITERATIONS=${BURN_IN_ITERATIONS:-5} +FAILURES=0 + +for i in $(seq 1 $ITERATIONS); do + echo "=== Burn-in iteration $i/$ITERATIONS ===" + + if npm test; then + echo "✓ Iteration $i passed" + else + echo "✗ Iteration $i failed" + FAILURES=$((FAILURES + 1)) + fi +done + +if [ $FAILURES -gt 0 ]; then + echo "❌ Tests failed in $FAILURES/$ITERATIONS iterations" + exit 1 +fi + +echo "✅ All $ITERATIONS iterations passed" +``` + +**Local CI Mirror Script** (`scripts/ci-local.sh`): + +```bash +#!/bin/bash +# Mirror CI execution locally for debugging + +echo "🔍 Running CI pipeline locally..." + +# Lint +npm run lint || exit 1 + +# Tests +npm run test || exit 1 + +# Burn-in (reduced iterations for local) +for i in {1..3}; do + echo "🔥 Burn-in $i/3" + npm test || exit 1 +done + +echo "✅ Local CI pipeline passed" +``` + +**Make scripts executable:** +```bash +chmod +x scripts/*.sh +``` + +**Alternative: Smart Burn-In with Playwright Utils** + +If `tea_use_playwright_utils: true`, you can use git diff-based burn-in: + +```typescript +// scripts/burn-in-changed.ts +import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'; + +async function main() { + await runBurnIn({ + configPath: 'playwright.burn-in.config.ts', + baseBranch: 'main' + }); +} + +main().catch(console.error); +``` + +```typescript +// playwright.burn-in.config.ts +import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'; + +const config: BurnInConfig = { + skipBurnInPatterns: ['**/config/**', '**/*.md', '**/*types*'], + burnInTestPercentage: 0.3, // Run 30% of affected tests + burnIn: { repeatEach: 5, retries: 1 } +}; + +export default config; +``` + +**Benefits over shell script:** +- Only runs tests affected by git changes (faster) +- Smart filtering (skips config, docs, types) +- Volume control (run percentage, not all tests) + +**Example:** Changed 1 file → runs 3 affected tests 5 times = 15 runs (not 500 tests × 5 = 2500 runs) + +### 6. Configure Secrets + +TEA provides a secrets checklist. + +**Required Secrets** (add to CI/CD platform): + +```markdown +## GitHub Actions Secrets + +Repository Settings → Secrets and variables → Actions + +### Required +- None (tests run without external auth) + +### Optional +- `TEST_USER_EMAIL` - Test user credentials +- `TEST_USER_PASSWORD` - Test user password +- `API_BASE_URL` - API endpoint for tests +- `DATABASE_URL` - Test database (if needed) +``` + +**How to Add Secrets:** + +**GitHub Actions:** +1. Go to repo Settings → Secrets → Actions +2. Click "New repository secret" +3. Add name and value +4. Use in workflow: `${{ secrets.TEST_USER_EMAIL }}` + +**GitLab CI:** +1. Go to Project Settings → CI/CD → Variables +2. Add variable name and value +3. Use in workflow: `$TEST_USER_EMAIL` + +### 7. Test the CI Pipeline + +#### Push and Verify + +**Commit the workflow file:** +```bash +git add .github/workflows/test.yml +git commit -m "ci: add automated test pipeline" +git push +``` + +**Watch the CI run:** +- GitHub Actions: Go to Actions tab +- GitLab CI: Go to CI/CD → Pipelines +- Circle CI: Go to Pipelines + +**Expected Result:** +``` +✓ test (shard 1/4) - 3m 24s +✓ test (shard 2/4) - 3m 18s +✓ test (shard 3/4) - 3m 31s +✓ test (shard 4/4) - 3m 15s +✓ burn-in - 15m 42s +``` + +#### Test on Pull Request + +**Create test PR:** +```bash +git checkout -b test-ci-setup +echo "# Test" > test.md +git add test.md +git commit -m "test: verify CI setup" +git push -u origin test-ci-setup +``` + +**Open PR and verify:** +- Tests run automatically +- Burn-in runs (if configured for PRs) +- Selective tests run (if applicable) +- All checks pass ✓ + +## What You Get + +### Automated Test Execution +- **On every PR** - Catch issues before merge +- **On every push to main** - Protect production +- **Nightly** - Comprehensive regression testing + +### Parallel Execution +- **4x faster feedback** - Shard across multiple workers +- **Efficient resource usage** - Maximize CI runner utilization + +### Selective Testing +- **Run only affected tests** - Git diff-based selection +- **Faster PR feedback** - Don't run entire suite every time + +### Flakiness Detection +- **Burn-in loops** - Run tests multiple times +- **Early detection** - Catch flaky tests in PRs +- **Confidence building** - Know tests are reliable + +### Artifact Collection +- **Test results** - Saved for 7 days +- **Screenshots** - On test failures +- **Videos** - Full test recordings +- **Traces** - Playwright trace files for debugging + +## Tips + +### Start Simple, Add Complexity + +**Week 1:** Basic pipeline +```yaml +- Run tests on PR +- Single worker (no sharding) +``` + +**Week 2:** Add parallelization +```yaml +- Shard across 4 workers +- Faster feedback +``` + +**Week 3:** Add selective testing +```yaml +- Git diff-based selection +- Skip unaffected tests +``` + +**Week 4:** Add burn-in +```yaml +- Detect flaky tests +- Run on PR and nightly +``` + +### Optimize for Feedback Speed + +**Goal:** PR feedback in < 5 minutes + +**Strategies:** +- Shard tests across workers (4 workers = 4x faster) +- Use selective testing (run 20% of tests, not 100%) +- Cache dependencies (`actions/cache`, `cache: 'npm'`) +- Run smoke tests first, full suite after + +**Example fast workflow:** +```yaml +jobs: + smoke: + # Run critical path tests (2 min) + run: npm run test:smoke + + full: + needs: smoke + # Run full suite only if smoke passes (10 min) + run: npm test +``` + +### Use Test Tags + +Tag tests for selective execution: + +```typescript +// Critical path tests (always run) +test('@critical should login', async ({ page }) => { }); + +// Smoke tests (run first) +test('@smoke should load homepage', async ({ page }) => { }); + +// Slow tests (run nightly only) +test('@slow should process large file', async ({ page }) => { }); + +// Skip in CI +test('@local-only should use local service', async ({ page }) => { }); +``` + +**In CI:** +```bash +# PR: Run critical and smoke only +npx playwright test --grep "@critical|@smoke" + +# Nightly: Run everything except local-only +npx playwright test --grep-invert "@local-only" +``` + +### Monitor CI Performance + +Track metrics: + +```markdown +## CI Metrics + +| Metric | Target | Current | Status | +|--------|--------|---------|--------| +| PR feedback time | < 5 min | 3m 24s | ✅ | +| Full suite time | < 15 min | 12m 18s | ✅ | +| Flakiness rate | < 1% | 0.3% | ✅ | +| CI cost/month | < $100 | $75 | ✅ | +``` + +### Handle Flaky Tests + +When burn-in detects flakiness: + +1. **Quarantine flaky test:** +```typescript +test.skip('flaky test - investigating', async ({ page }) => { + // TODO: Fix flakiness +}); +``` + +2. **Investigate with trace viewer:** +```bash +npx playwright show-trace test-results/trace.zip +``` + +3. **Fix root cause:** +- Add network-first patterns +- Remove hard waits +- Fix race conditions + +4. **Verify fix:** +```bash +npm run test:burn-in -- tests/flaky.spec.ts --repeat 20 +``` + +### Secure Secrets + +**Don't commit secrets to code:** +```yaml +# ❌ Bad +- run: API_KEY=sk-1234... npm test + +# ✅ Good +- run: npm test + env: + API_KEY: ${{ secrets.API_KEY }} +``` + +**Use environment-specific secrets:** +- `STAGING_API_URL` +- `PROD_API_URL` +- `TEST_API_URL` + +### Cache Aggressively + +Speed up CI with caching: + +```yaml +# Cache node_modules +- uses: actions/setup-node@v4 + with: + cache: 'npm' + +# Cache Playwright browsers +- name: Cache Playwright browsers + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ hashFiles('package-lock.json') }} +``` + +## Common Issues + +### Tests Pass Locally, Fail in CI + +**Symptoms:** +- Green locally, red in CI +- "Works on my machine" + +**Common Causes:** +- Different Node version +- Different browser version +- Missing environment variables +- Timezone differences +- Race conditions (CI slower) + +**Solutions:** +```yaml +# Pin Node version +- uses: actions/setup-node@v4 + with: + node-version-file: '.nvmrc' + +# Pin browser versions +- run: npx playwright install --with-deps chromium@1.40.0 + +# Set timezone + env: + TZ: 'America/New_York' +``` + +### CI Takes Too Long + +**Problem:** CI takes 30+ minutes, developers wait too long. + +**Solutions:** +1. **Shard tests:** 4 workers = 4x faster +2. **Selective testing:** Only run affected tests on PR +3. **Smoke tests first:** Run critical path (2 min), full suite after +4. **Cache dependencies:** `npm ci` with cache +5. **Optimize tests:** Remove slow tests, hard waits + +### Burn-In Always Fails + +**Problem:** Burn-in job fails every time. + +**Cause:** Test suite is flaky. + +**Solution:** +1. Identify flaky tests (check which iteration fails) +2. Fix flaky tests using `*test-review` +3. Re-run burn-in on specific files: +```bash +npm run test:burn-in tests/flaky.spec.ts +``` + +### Out of CI Minutes + +**Problem:** Using too many CI minutes, hitting plan limit. + +**Solutions:** +1. Run full suite only on main branch +2. Use selective testing on PRs +3. Run expensive tests nightly only +4. Self-host runners (for GitHub Actions) + +## Related Guides + +- [How to Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) - Run first +- [How to Run Test Review](/docs/how-to/workflows/run-test-review.md) - Audit CI tests +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) - Burn-in utility + +## Understanding the Concepts + +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) - Why determinism matters +- [Network-First Patterns](/docs/explanation/tea/network-first-patterns.md) - Avoid CI flakiness + +## Reference + +- [Command: *ci](/docs/reference/tea/commands.md#ci) - Full command reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - CI-related config options + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/reference/glossary/index.md b/docs/reference/glossary/index.md index 6e4d4586..5191e6ed 100644 --- a/docs/reference/glossary/index.md +++ b/docs/reference/glossary/index.md @@ -6,117 +6,154 @@ Terminology reference for the BMad Method. ## Core Concepts -| Term | Definition | -|------|------------| -| **Agent** | Specialized AI persona with specific expertise (PM, Architect, SM, DEV, TEA) that guides users through workflows and creates deliverables. | -| **BMad** | Breakthrough Method of Agile AI Driven Development — AI-driven agile framework with specialized agents, guided workflows, and scale-adaptive intelligence. | -| **BMad Method** | Complete methodology for AI-assisted software development, encompassing planning, architecture, implementation, and quality assurance workflows that adapt to project complexity. | -| **BMM** | BMad Method Module — core orchestration system providing comprehensive lifecycle management through specialized agents and workflows. | -| **Scale-Adaptive System** | Intelligent workflow orchestration that adjusts planning depth and documentation requirements based on project needs through three planning tracks. | -| **Workflow** | Multi-step guided process that orchestrates AI agent activities to produce specific deliverables. Workflows are interactive and adapt to user context. | +| Term | Definition | +| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Agent** | Specialized AI persona with specific expertise (PM, Architect, SM, DEV, TEA) that guides users through workflows and creates deliverables. | +| **BMad** | Breakthrough Method of Agile AI Driven Development — AI-driven agile framework with specialized agents, guided workflows, and scale-adaptive intelligence. | +| **BMad Method** | Complete methodology for AI-assisted software development, encompassing planning, architecture, implementation, and quality assurance workflows that adapt to project complexity. | +| **BMM** | BMad Method Module — core orchestration system providing comprehensive lifecycle management through specialized agents and workflows. | +| **Scale-Adaptive System** | Intelligent workflow orchestration that adjusts planning depth and documentation requirements based on project needs through three planning tracks. | +| **Workflow** | Multi-step guided process that orchestrates AI agent activities to produce specific deliverables. Workflows are interactive and adapt to user context. | ## Scale and Complexity -| Term | Definition | -|------|------------| -| **BMad Method Track** | Full product planning track using PRD + Architecture + UX. Best for products, platforms, and complex features. Typical range: 10-50+ stories. | +| Term | Definition | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **BMad Method Track** | Full product planning track using PRD + Architecture + UX. Best for products, platforms, and complex features. Typical range: 10-50+ stories. | | **Enterprise Method Track** | Extended planning track adding Security Architecture, DevOps Strategy, and Test Strategy. Best for compliance needs and multi-tenant systems. Typical range: 30+ stories. | -| **Planning Track** | Methodology path (Quick Flow, BMad Method, or Enterprise) chosen based on planning needs and complexity, not story count alone. | -| **Quick Flow Track** | Fast implementation track using tech-spec only. Best for bug fixes, small features, and clear-scope changes. Typical range: 1-15 stories. | +| **Planning Track** | Methodology path (Quick Flow, BMad Method, or Enterprise) chosen based on planning needs and complexity, not story count alone. | +| **Quick Flow Track** | Fast implementation track using tech-spec only. Best for bug fixes, small features, and clear-scope changes. Typical range: 1-15 stories. | ## Planning Documents -| Term | Definition | -|------|------------| +| Term | Definition | +| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | **Architecture Document** | *BMad Method/Enterprise.* System-wide design document defining structure, components, data models, integration patterns, security, and deployment. | -| **Epics** | High-level feature groupings containing multiple related stories. Typically 5-15 stories each representing cohesive functionality. | -| **Game Brief** | *BMGD.* Document capturing game's core vision, pillars, target audience, and scope. Foundation for the GDD. | -| **GDD** | *BMGD.* Game Design Document — comprehensive document detailing all aspects of game design: mechanics, systems, content, and more. | -| **PRD** | *BMad Method/Enterprise.* Product Requirements Document containing vision, goals, FRs, NFRs, and success criteria. Focuses on WHAT to build. | -| **Product Brief** | *Phase 1.* Optional strategic document capturing product vision, market context, and high-level requirements before detailed planning. | -| **Tech-Spec** | *Quick Flow only.* Comprehensive technical plan with problem statement, solution approach, file-level changes, and testing strategy. | +| **Epics** | High-level feature groupings containing multiple related stories. Typically 5-15 stories each representing cohesive functionality. | +| **Game Brief** | *BMGD.* Document capturing game's core vision, pillars, target audience, and scope. Foundation for the GDD. | +| **GDD** | *BMGD.* Game Design Document — comprehensive document detailing all aspects of game design: mechanics, systems, content, and more. | +| **PRD** | *BMad Method/Enterprise.* Product Requirements Document containing vision, goals, FRs, NFRs, and success criteria. Focuses on WHAT to build. | +| **Product Brief** | *Phase 1.* Optional strategic document capturing product vision, market context, and high-level requirements before detailed planning. | +| **Tech-Spec** | *Quick Flow only.* Comprehensive technical plan with problem statement, solution approach, file-level changes, and testing strategy. | ## Workflow and Phases -| Term | Definition | -|------|------------| -| **Phase 0: Documentation** | *Brownfield.* Conditional prerequisite phase creating codebase documentation before planning. Only required if existing docs are insufficient. | -| **Phase 1: Analysis** | Discovery phase including brainstorming, research, and product brief creation. Optional for Quick Flow, recommended for BMad Method. | -| **Phase 2: Planning** | Required phase creating formal requirements. Routes to tech-spec (Quick Flow) or PRD (BMad Method/Enterprise). | -| **Phase 3: Solutioning** | *BMad Method/Enterprise.* Architecture design phase including creation, validation, and gate checks. | -| **Phase 4: Implementation** | Required sprint-based development through story-by-story iteration using sprint-planning, create-story, dev-story, and code-review workflows. | -| **Quick Spec Flow** | Fast-track workflow for Quick Flow projects going straight from idea to tech-spec to implementation. | -| **Workflow Init** | Initialization workflow creating bmm-workflow-status.yaml, detecting project type, and determining planning track. | -| **Workflow Status** | Universal entry point checking for existing status file, displaying progress, and recommending next action. | +| Term | Definition | +| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| **Phase 0: Documentation** | *Brownfield.* Conditional prerequisite phase creating codebase documentation before planning. Only required if existing docs are insufficient. | +| **Phase 1: Analysis** | Discovery phase including brainstorming, research, and product brief creation. Optional for Quick Flow, recommended for BMad Method. | +| **Phase 2: Planning** | Required phase creating formal requirements. Routes to tech-spec (Quick Flow) or PRD (BMad Method/Enterprise). | +| **Phase 3: Solutioning** | *BMad Method/Enterprise.* Architecture design phase including creation, validation, and gate checks. | +| **Phase 4: Implementation** | Required sprint-based development through story-by-story iteration using sprint-planning, create-story, dev-story, and code-review workflows. | +| **Quick Spec Flow** | Fast-track workflow for Quick Flow projects going straight from idea to tech-spec to implementation. | +| **Workflow Init** | Initialization workflow creating bmm-workflow-status.yaml, detecting project type, and determining planning track. | +| **Workflow Status** | Universal entry point checking for existing status file, displaying progress, and recommending next action. | ## Agents and Roles -| Term | Definition | -|------|------------| -| **Analyst** | Agent that initializes workflows, conducts research, creates product briefs, and tracks progress. Often the entry point for new projects. | -| **Architect** | Agent designing system architecture, creating architecture documents, and validating designs. Primary agent for Phase 3. | -| **BMad Master** | Meta-level orchestrator from BMad Core facilitating party mode and providing high-level guidance across all modules. | -| **DEV** | Developer agent implementing stories, writing code, running tests, and performing code reviews. Primary implementer in Phase 4. | -| **Game Architect** | *BMGD.* Agent designing game system architecture and validating game-specific technical designs. | -| **Game Designer** | *BMGD.* Agent creating game design documents (GDD) and running game-specific workflows. | -| **Party Mode** | Multi-agent collaboration feature where agents discuss challenges together. BMad Master orchestrates, selecting 2-3 relevant agents per message. | -| **PM** | Product Manager agent creating PRDs and tech-specs. Primary agent for Phase 2 planning. | -| **SM** | Scrum Master agent managing sprints, creating stories, and coordinating implementation. Primary orchestrator for Phase 4. | -| **TEA** | Test Architect agent responsible for test strategy, quality gates, and NFR assessment. Integrates throughout all phases. | -| **Technical Writer** | Agent specialized in creating technical documentation, diagrams, and maintaining documentation standards. | -| **UX Designer** | Agent creating UX design documents, interaction patterns, and visual specifications for UI-heavy projects. | +| Term | Definition | +| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Analyst** | Agent that initializes workflows, conducts research, creates product briefs, and tracks progress. Often the entry point for new projects. | +| **Architect** | Agent designing system architecture, creating architecture documents, and validating designs. Primary agent for Phase 3. | +| **BMad Master** | Meta-level orchestrator from BMad Core facilitating party mode and providing high-level guidance across all modules. | +| **DEV** | Developer agent implementing stories, writing code, running tests, and performing code reviews. Primary implementer in Phase 4. | +| **Game Architect** | *BMGD.* Agent designing game system architecture and validating game-specific technical designs. | +| **Game Designer** | *BMGD.* Agent creating game design documents (GDD) and running game-specific workflows. | +| **Party Mode** | Multi-agent collaboration feature where agents discuss challenges together. BMad Master orchestrates, selecting 2-3 relevant agents per message. | +| **PM** | Product Manager agent creating PRDs and tech-specs. Primary agent for Phase 2 planning. | +| **SM** | Scrum Master agent managing sprints, creating stories, and coordinating implementation. Primary orchestrator for Phase 4. | +| **TEA** | Test Architect agent responsible for test strategy, quality gates, and NFR assessment. Integrates throughout all phases. | +| **Technical Writer** | Agent specialized in creating technical documentation, diagrams, and maintaining documentation standards. | +| **UX Designer** | Agent creating UX design documents, interaction patterns, and visual specifications for UI-heavy projects. | ## Status and Tracking -| Term | Definition | -|------|------------| -| **bmm-workflow-status.yaml** | *Phases 1-3.* Tracking file showing current phase, completed workflows, and next recommended actions. | -| **DoD** | Definition of Done — criteria for marking a story complete: implementation done, tests passing, code reviewed, docs updated. | -| **Epic Status Progression** | `backlog → in-progress → done` — lifecycle states for epics during implementation. | -| **Gate Check** | Validation workflow (implementation-readiness) ensuring PRD, Architecture, and Epics are aligned before Phase 4. | -| **Retrospective** | Workflow after each epic capturing learnings and improvements for continuous improvement. | -| **sprint-status.yaml** | *Phase 4.* Single source of truth for implementation tracking containing all epics, stories, and their statuses. | -| **Story Status Progression** | `backlog → ready-for-dev → in-progress → review → done` — lifecycle states for stories. | +| Term | Definition | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| **bmm-workflow-status.yaml** | *Phases 1-3.* Tracking file showing current phase, completed workflows, and next recommended actions. | +| **DoD** | Definition of Done — criteria for marking a story complete: implementation done, tests passing, code reviewed, docs updated. | +| **Epic Status Progression** | `backlog → in-progress → done` — lifecycle states for epics during implementation. | +| **Gate Check** | Validation workflow (implementation-readiness) ensuring PRD, Architecture, and Epics are aligned before Phase 4. | +| **Retrospective** | Workflow after each epic capturing learnings and improvements for continuous improvement. | +| **sprint-status.yaml** | *Phase 4.* Single source of truth for implementation tracking containing all epics, stories, and their statuses. | +| **Story Status Progression** | `backlog → ready-for-dev → in-progress → review → done` — lifecycle states for stories. | ## Project Types -| Term | Definition | -|------|------------| -| **Brownfield** | Existing project with established codebase and patterns. Requires understanding existing architecture and planning integration. | -| **Convention Detection** | *Quick Flow.* Feature auto-detecting existing code style, naming conventions, and frameworks from brownfield codebases. | -| **document-project** | *Brownfield.* Workflow analyzing and documenting existing codebase with three scan levels: quick, deep, exhaustive. | -| **Feature Flags** | *Brownfield.* Implementation technique for gradual rollout, easy rollback, and A/B testing of new functionality. | -| **Greenfield** | New project starting from scratch with freedom to establish patterns, choose stack, and design from clean slate. | -| **Integration Points** | *Brownfield.* Specific locations where new code connects with existing systems. Must be documented in tech-specs. | +| Term | Definition | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | +| **Brownfield** | Existing project with established codebase and patterns. Requires understanding existing architecture and planning integration. | +| **Convention Detection** | *Quick Flow.* Feature auto-detecting existing code style, naming conventions, and frameworks from brownfield codebases. | +| **document-project** | *Brownfield.* Workflow analyzing and documenting existing codebase with three scan levels: quick, deep, exhaustive. | +| **Feature Flags** | *Brownfield.* Implementation technique for gradual rollout, easy rollback, and A/B testing of new functionality. | +| **Greenfield** | New project starting from scratch with freedom to establish patterns, choose stack, and design from clean slate. | +| **Integration Points** | *Brownfield.* Specific locations where new code connects with existing systems. Must be documented in tech-specs. | ## Implementation Terms -| Term | Definition | -|------|------------| +| Term | Definition | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | **Context Engineering** | Loading domain-specific standards into AI context automatically via manifests, ensuring consistent outputs regardless of prompt variation. | -| **Correct Course** | Workflow for navigating significant changes when implementation is off-track. Analyzes impact and recommends adjustments. | -| **Shard / Sharding** | Splitting large planning documents into section-based files for LLM optimization. Phase 4 workflows load only needed sections. | -| **Sprint** | Time-boxed period of development work, typically 1-2 weeks. | -| **Sprint Planning** | Workflow initializing Phase 4 by creating sprint-status.yaml and extracting epics/stories from planning docs. | -| **Story** | Single unit of implementable work with clear acceptance criteria, typically 2-8 hours of effort. Grouped into epics. | -| **Story Context** | Implementation guidance embedded in story files during create-story, referencing existing patterns and approaches. | -| **Story File** | Markdown file containing story description, acceptance criteria, technical notes, and testing requirements. | -| **Track Selection** | Automatic analysis by workflow-init suggesting appropriate track based on complexity indicators. User can override. | +| **Correct Course** | Workflow for navigating significant changes when implementation is off-track. Analyzes impact and recommends adjustments. | +| **Shard / Sharding** | Splitting large planning documents into section-based files for LLM optimization. Phase 4 workflows load only needed sections. | +| **Sprint** | Time-boxed period of development work, typically 1-2 weeks. | +| **Sprint Planning** | Workflow initializing Phase 4 by creating sprint-status.yaml and extracting epics/stories from planning docs. | +| **Story** | Single unit of implementable work with clear acceptance criteria, typically 2-8 hours of effort. Grouped into epics. | +| **Story Context** | Implementation guidance embedded in story files during create-story, referencing existing patterns and approaches. | +| **Story File** | Markdown file containing story description, acceptance criteria, technical notes, and testing requirements. | +| **Track Selection** | Automatic analysis by workflow-init suggesting appropriate track based on complexity indicators. User can override. | ## Game Development Terms -| Term | Definition | -|------|------------| -| **Core Fantasy** | *BMGD.* The emotional experience players seek from your game — what they want to FEEL. | -| **Core Loop** | *BMGD.* Fundamental cycle of actions players repeat throughout gameplay. The heart of your game. | -| **Design Pillar** | *BMGD.* Core principle guiding all design decisions. Typically 3-5 pillars define a game's identity. | -| **Environmental Storytelling** | *BMGD.* Narrative communicated through the game world itself rather than explicit dialogue. | -| **Game Type** | *BMGD.* Genre classification determining which specialized GDD sections are included. | -| **MDA Framework** | *BMGD.* Mechanics → Dynamics → Aesthetics — framework for analyzing and designing games. | -| **Meta-Progression** | *BMGD.* Persistent progression carrying between individual runs or sessions. | -| **Metroidvania** | *BMGD.* Genre featuring interconnected world exploration with ability-gated progression. | -| **Narrative Complexity** | *BMGD.* How central story is to the game: Critical, Heavy, Moderate, or Light. | -| **Permadeath** | *BMGD.* Game mechanic where character death is permanent, typically requiring a new run. | -| **Player Agency** | *BMGD.* Degree to which players can make meaningful choices affecting outcomes. | -| **Procedural Generation** | *BMGD.* Algorithmic creation of game content (levels, items, characters) rather than hand-crafted. | -| **Roguelike** | *BMGD.* Genre featuring procedural generation, permadeath, and run-based progression. | +| Term | Definition | +| ------------------------------ | ---------------------------------------------------------------------------------------------------- | +| **Core Fantasy** | *BMGD.* The emotional experience players seek from your game — what they want to FEEL. | +| **Core Loop** | *BMGD.* Fundamental cycle of actions players repeat throughout gameplay. The heart of your game. | +| **Design Pillar** | *BMGD.* Core principle guiding all design decisions. Typically 3-5 pillars define a game's identity. | +| **Environmental Storytelling** | *BMGD.* Narrative communicated through the game world itself rather than explicit dialogue. | +| **Game Type** | *BMGD.* Genre classification determining which specialized GDD sections are included. | +| **MDA Framework** | *BMGD.* Mechanics → Dynamics → Aesthetics — framework for analyzing and designing games. | +| **Meta-Progression** | *BMGD.* Persistent progression carrying between individual runs or sessions. | +| **Metroidvania** | *BMGD.* Genre featuring interconnected world exploration with ability-gated progression. | +| **Narrative Complexity** | *BMGD.* How central story is to the game: Critical, Heavy, Moderate, or Light. | +| **Permadeath** | *BMGD.* Game mechanic where character death is permanent, typically requiring a new run. | +| **Player Agency** | *BMGD.* Degree to which players can make meaningful choices affecting outcomes. | +| **Procedural Generation** | *BMGD.* Algorithmic creation of game content (levels, items, characters) rather than hand-crafted. | +| **Roguelike** | *BMGD.* Genre featuring procedural generation, permadeath, and run-based progression. | + +## Test Architect (TEA) Concepts + +| Term | Definition | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **ATDD** | Acceptance Test-Driven Development — Generating failing acceptance tests BEFORE implementation (TDD red phase). | +| **Burn-in Testing** | Running tests multiple times (typically 5-10 iterations) to detect flakiness and intermittent failures. | +| **Component Testing** | Testing UI components in isolation using framework-specific tools (Cypress Component Testing or Vitest + React Testing Library). | +| **Coverage Traceability** | Mapping acceptance criteria to implemented tests with classification (FULL/PARTIAL/NONE) to identify gaps and measure completeness. | +| **Epic-Level Test Design** | Test planning per epic (Phase 4) focusing on risk assessment, priorities, and coverage strategy for that specific epic. | +| **Fixture Architecture** | Pattern of building pure functions first, then wrapping in framework-specific fixtures for testability, reusability, and composition. | +| **Gate Decision** | Go/no-go decision for release with four outcomes: PASS ✅ (ready), CONCERNS ⚠️ (proceed with mitigation), FAIL ❌ (blocked), WAIVED ⏭️ (approved despite issues). | +| **Knowledge Fragment** | Individual markdown file in TEA's knowledge base covering a specific testing pattern or practice (33 fragments total). | +| **MCP Enhancements** | Model Context Protocol servers enabling live browser verification during test generation (exploratory, recording, and healing modes). | +| **Network-First Pattern** | Testing pattern that waits for actual network responses instead of fixed timeouts to avoid race conditions and flakiness. | +| **NFR Assessment** | Validation of non-functional requirements (security, performance, reliability, maintainability) with evidence-based decisions. | +| **Playwright Utils** | Optional package (`@seontechnologies/playwright-utils`) providing production-ready fixtures and utilities for Playwright tests. | +| **Risk-Based Testing** | Testing approach where depth scales with business impact using probability × impact scoring (1-9 scale). | +| **System-Level Test Design** | Test planning at architecture level (Phase 3) focusing on testability review, ADR mapping, and test infrastructure needs. | +| **tea-index.csv** | Manifest file tracking all knowledge fragments, their descriptions, tags, and which workflows load them. | +| **TEA Integrated** | Full BMad Method integration with TEA workflows across all phases (Phase 2, 3, 4, and Release Gate). | +| **TEA Lite** | Beginner approach using just `*automate` workflow to test existing features (simplest way to use TEA). | +| **TEA Solo** | Standalone engagement model using TEA without full BMad Method integration (bring your own requirements). | +| **Test Priorities** | Classification system for test importance: P0 (critical path), P1 (high value), P2 (medium value), P3 (low value). | + +--- + +## See Also + +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Complete TEA capabilities +- [TEA Knowledge Base](/docs/reference/tea/knowledge-base.md) - Fragment index +- [TEA Command Reference](/docs/reference/tea/commands.md) - Workflow reference +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config options + +--- + +Generated with [BMad Method](https://bmad-method.org) diff --git a/docs/reference/tea/commands.md b/docs/reference/tea/commands.md new file mode 100644 index 00000000..89bc2d34 --- /dev/null +++ b/docs/reference/tea/commands.md @@ -0,0 +1,1371 @@ +--- +title: "TEA Command Reference" +description: Complete reference for all TEA (Test Architect) workflows and commands +--- + +# TEA Command Reference + +Complete reference for all 8 TEA (Test Architect) workflows. Use this for quick lookup of commands, parameters, and outputs. + +## Quick Index + +- [*framework](#framework) - Scaffold test framework +- [*ci](#ci) - Setup CI/CD pipeline +- [*test-design](#test-design) - Risk-based test planning +- [*atdd](#atdd) - Acceptance TDD (failing tests first) +- [*automate](#automate) - Test automation expansion +- [*test-review](#test-review) - Test quality audit +- [*nfr-assess](#nfr-assess) - Non-functional requirements assessment +- [*trace](#trace) - Coverage traceability and gate decisions + +**Note:** `*workflow-status` is a shared BMM workflow available to all agents, not TEA-specific. See [Core Workflows](/docs/reference/workflows/core-workflows.md). + +--- + +## *framework + +Scaffold production-ready test framework (Playwright or Cypress). + +### Purpose + +Initialize test infrastructure with best practices, environment configuration, and sample tests. + +### Phase + +Phase 3 (Solutioning) - Run once per project after architecture is complete. + +### Frequency + +Once per project (one-time setup). + +### When to Use + +- No existing test framework in your project +- Current test setup isn't production-ready +- Starting new project needing test infrastructure +- Want to adopt Playwright or Cypress with proper structure + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| Tech stack | "React web application" | Helps determine test approach | +| Test framework | "Playwright" | Playwright or Cypress | +| Testing scope | "E2E and API testing" | E2E, integration, unit, or mix | +| CI/CD platform | "GitHub Actions" | For future `*ci` setup | + +### Outputs + +**Generated Files:** +``` +tests/ +├── e2e/ # E2E test directory +│ └── example.spec.ts # Sample E2E test +├── api/ # API test directory (if requested) +│ └── example.spec.ts # Sample API test +├── support/ # Support directory +│ ├── fixtures/ # Shared fixtures +│ │ └── index.ts # Fixture composition +│ └── helpers/ # Pure utility functions +│ └── api-request.ts # Example helper +├── playwright.config.ts # Framework configuration +└── README.md # Testing documentation + +.env.example # Environment variable template +.nvmrc # Node version specification +``` + +**Configuration Includes:** +- Multiple environments (dev, staging, prod) +- Timeout standards +- Retry logic +- Artifact collection (screenshots, videos, traces) +- Reporter configuration + +**Sample Tests Include:** +- Network-first patterns (no hard waits) +- Proper fixture usage +- Explicit assertions +- Deterministic test structure + +**Framework-Specific Examples:** + +**Vanilla Playwright:** +```typescript +// tests/e2e/example.spec.ts +import { test, expect } from '@playwright/test'; + +test('example test', async ({ page, request }) => { + // Manual API call + const response = await request.get('/api/data'); + const data = await response.json(); + + await page.goto('/'); + await expect(page.locator('h1')).toContainText(data.title); +}); +``` + +**With Playwright Utils:** +```typescript +// tests/e2e/example.spec.ts +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { expect } from '@playwright/test'; + +test('example test', async ({ page, apiRequest }) => { + // Utility handles status/body separation + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/data' + }); + + expect(status).toBe(200); + await page.goto('/'); + await expect(page.locator('h1')).toContainText(body.title); +}); +``` + +### Optional Integrations + +**Playwright Utils:** +If `tea_use_playwright_utils: true` in config: +- Includes `@seontechnologies/playwright-utils` in scaffold +- Adds fixture composition examples +- Provides utility import examples + +See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) + +### Related Commands + +- `*ci` - Run after framework to setup CI/CD +- `*test-design` - Run concurrently or after for test planning + +### How-To Guide + +[How to Set Up a Test Framework](/docs/how-to/workflows/setup-test-framework.md) + +--- + +## *ci + +Setup CI/CD quality pipeline with selective testing and burn-in loops. + +### Purpose + +Scaffold production-ready CI/CD configuration for automated test execution. + +### Phase + +Phase 3 (Solutioning) - Run once per project after framework setup. + +### Frequency + +Once per project (one-time setup). + +### When to Use + +- Need to automate test execution in CI/CD +- Want selective testing (only run affected tests) +- Need burn-in loops for flakiness detection +- Setting up new CI/CD pipeline + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| CI/CD platform | "GitHub Actions" | GitHub Actions, GitLab CI, Circle CI, Jenkins | +| Repository structure | "Monorepo with multiple apps" | Affects test selection strategy | +| Sharding strategy | "Yes, run tests in parallel" | Shard across multiple workers | +| Burn-in loops | "Yes, for flakiness detection" | Run tests multiple times | + +### Outputs + +**Platform-Specific Workflow:** + +**GitHub Actions** (`.github/workflows/test.yml`): +```yaml +name: Test Suite + +on: + pull_request: + push: + branches: [main, develop] + +jobs: + test: + runs-on: ubuntu-latest + strategy: + matrix: + shard: [1, 2, 3, 4] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version-file: '.nvmrc' + + - name: Install dependencies + run: npm ci + + - name: Run tests + run: npx playwright test --shard=${{ matrix.shard }}/4 + + - name: Upload artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-results-${{ matrix.shard }} + path: test-results/ + + burn-in: + runs-on: ubuntu-latest + if: github.event_name == 'pull_request' + steps: + - name: Run burn-in loop + run: | + for i in {1..5}; do + npx playwright test --grep-invert @skip + done +``` + +**Also Generates:** +- **Test scripts** (`package.json`) - Selective testing commands +- **Secrets checklist** - Required environment variables +- **Sharding configuration** - Parallel execution setup +- **Artifact collection** - Save screenshots, videos, traces + +### Selective Testing + +Generated CI includes selective test execution: + +```bash +# Run only tests affected by changes +npm run test:selective + +# Run specific tags +npm run test:smoke # @smoke tagged tests +npm run test:critical # @critical tagged tests +``` + +### Burn-in Loops + +Detects flaky tests by running multiple times: + +```bash +# Run tests 5 times to detect flakiness +npm run test:burn-in + +# Run specific test file 10 times +npm run test:burn-in -- tests/e2e/checkout.spec.ts --repeat 10 +``` + +### Optional Integrations + +**Playwright Utils:** +If `tea_use_playwright_utils: true`: +- Includes `burn-in` utility for smart test selection +- Adds git diff-based selective testing +- Provides test prioritization + +See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) + +### Related Commands + +- `*framework` - Run before CI setup +- `*test-review` - Run to ensure CI-ready tests + +### How-To Guide + +[How to Set Up CI Pipeline](/docs/how-to/workflows/setup-ci.md) + +--- + +## *test-design + +Create comprehensive test scenarios with risk assessment and coverage strategies. + +### Purpose + +Risk-based test planning that identifies what to test, at what level, and with what priority. + +### Phase + +**Dual Mode:** +- **Phase 3:** System-level (architecture testability review) +- **Phase 4:** Epic-level (per-epic test planning) + +### Frequency + +- **System-level:** Once per project (or when architecture changes) +- **Epic-level:** Per epic in implementation cycle + +### When to Use + +**System-level:** +- After architecture is complete +- Before implementation-readiness gate +- To validate architecture testability +- When ADRs (Architecture Decision Records) are updated + +**Epic-level:** +- At the start of each epic +- Before implementing stories in the epic +- To identify epic-specific testing needs +- When planning sprint work + +### Inputs + +TEA will ask: + +**Mode Selection:** +| Question | Example Answer | Notes | +|----------|----------------|-------| +| System or epic level? | "Epic-level" | Determines scope | + +**System-Level Inputs:** +- Architecture document location +- ADRs (if available) +- PRD with FRs/NFRs +- Technology stack decisions + +**Epic-Level Inputs:** +- Epic description and goals +- Stories with acceptance criteria +- Related PRD sections +- Integration points with existing system + +### Outputs + +**System-Level Output** (`test-design-system.md`): + +```markdown +# Test Design - System Level + +## Architecture Testability Review + +### Strengths +- Microservices architecture enables API-level testing +- Event-driven design allows message interception +- Clear service boundaries support contract testing + +### Concerns +- Complex distributed tracing may be hard to test +- No test environment for third-party integrations +- Database migrations need test data management + +## ADR → Test Mapping + +| ADR | Decision | Test Impact | Test Strategy | +|-----|----------|-------------|---------------| +| ADR-001 | Use PostgreSQL | Data integrity critical | API tests + DB assertions | +| ADR-002 | Event sourcing | Event replay testing | Integration tests for event handlers | +| ADR-003 | OAuth2 authentication | Auth flows complex | E2E + API tests for all flows | + +## Architecturally Significant Requirements (ASRs) + +### Performance +- API response time < 200ms (P99) +- Test strategy: Load testing with k6, API response time assertions + +### Security +- All endpoints require authentication +- Test strategy: Security testing suite, unauthorized access scenarios + +### Reliability +- 99.9% uptime requirement +- Test strategy: Chaos engineering, failover testing + +## Environment Needs + +- **Dev:** Local Docker compose setup +- **Staging:** Replica of production +- **Production:** Read-only test accounts + +## Test Infrastructure Recommendations + +- [ ] Set up contract testing with Pact +- [ ] Create test data factories +- [ ] Implement API mocking for third-party services +- [ ] Add performance test suite +``` + +**Epic-Level Output** (`test-design-epic-N.md`): + +```markdown +# Test Design - Epic 1: User Profile Management + +## Risk Assessment + +| Risk Category | Probability | Impact | Score | Mitigation | +|---------------|-------------|--------|-------|------------| +| DATA | 3 (High) | 3 (High) | 9 | Validate all profile updates, test data corruption scenarios | +| SEC | 2 (Medium) | 3 (High) | 6 | Test authorization (users can only edit own profiles) | +| BUS | 2 (Medium) | 2 (Medium) | 4 | Verify profile data appears correctly across app | +| PERF | 1 (Low) | 2 (Medium) | 2 | Profile load should be < 500ms | + +## Test Priorities + +### P0 - Critical Path (Must Test) +- User can view their own profile +- User can update profile fields +- Changes are persisted correctly +- **Coverage Target:** 100% (all scenarios) + +### P1 - High Value (Should Test) +- Validation prevents invalid data (email format, etc.) +- Unauthorized users cannot edit profiles +- Profile updates trigger notifications +- **Coverage Target:** 80% (major scenarios) + +### P2 - Medium Value (Nice to Test) +- Profile picture upload and display +- Profile history/audit log +- **Coverage Target:** 50% (happy path) + +### P3 - Low Value (Optional) +- Advanced profile customization +- Profile export functionality +- **Coverage Target:** 20% (smoke test) + +## Coverage Strategy + +### E2E Tests (5 tests) +- View profile page (P0) +- Edit and save profile (P0) +- Profile validation errors (P1) +- Unauthorized access prevented (P1) +- Profile picture upload (P2) + +### API Tests (8 tests) +- GET /api/profile returns profile (P0) +- PATCH /api/profile updates profile (P0) +- Validation for each field (P1) +- Authorization checks (P1) +- Profile picture upload API (P2) +- Profile history endpoint (P2) + +### Component Tests (3 tests) +- ProfileForm component renders (P1) +- ProfileForm validation (P1) +- ProfilePictureUpload component (P2) + +## Integration Risks + +- Profile data stored in PostgreSQL - ensure transaction integrity +- Profile updates trigger notification service - test event propagation +- Profile pictures stored in S3 - test upload/download flows + +## Regression Hotspots (Brownfield) + +N/A - New feature + +## Implementation Order + +1. API tests for profile CRUD (P0) +2. E2E test for viewing profile (P0) +3. E2E test for editing profile (P0) +4. Validation tests API + E2E (P1) +5. Authorization tests (P1) +6. Profile picture tests (P2) +``` + +### Optional: Exploratory Mode + +If MCP enhancements enabled (`tea_use_mcp_enhancements: true` in config): + +When prompted, select "exploratory mode" to: +- Open live browser for UI discovery +- Validate test scenarios against real behavior +- Capture accurate selectors interactively + +See [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) + +### Related Commands + +- `*atdd` - Generate tests based on test design +- `*automate` - Generate tests based on test design +- `*framework` - Run first if no test infrastructure + +### How-To Guide + +[How to Run Test Design](/docs/how-to/workflows/run-test-design.md) + +--- + +## *atdd + +Generate failing acceptance tests BEFORE implementation (TDD red phase). + +### Purpose + +Create failing tests that guide feature implementation following test-driven development. + +### Phase + +Phase 4 (Implementation) - Before implementing each story. + +### Frequency + +Per story (optional - only if practicing TDD). + +### When to Use + +- Feature doesn't exist yet +- Want tests to guide implementation +- Practicing test-driven development +- Want clear success criteria before coding + +**Don't use if:** +- Feature already exists (use `*automate` instead) +- Want tests that pass immediately + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| Story/feature details | "User profile page with CRUD ops" | What are you building? | +| Acceptance criteria | "User can view, edit, save profile" | What defines "done"? | +| Reference docs | "test-design-epic-1.md, story-123.md" | Optional context | +| Test levels | "API + E2E tests, focus P0/P1" | Which test types? | + +### Outputs + +**Failing Tests:** +- API tests (`tests/api/`) - Backend endpoint tests +- E2E tests (`tests/e2e/`) - Full user workflow tests +- Component tests (`tests/component/`) - UI component tests (if requested) +- All tests fail initially (red phase) + +### Component Testing by Framework + +TEA generates component tests using framework-appropriate tools: + +| Your Framework | Component Testing Tool | What TEA Generates | +|----------------|----------------------|-------------------| +| **Cypress** | Cypress Component Testing | Cypress component specs (*.cy.tsx) | +| **Playwright** | Vitest + React Testing Library | Vitest component tests (*.test.tsx) | + +**Note:** Component tests use separate tooling: +- Cypress: Run with `cypress run-ct` +- Vitest: Run with `vitest` or `npm run test:unit` + +**Implementation Checklist:** +```markdown +## Implementation Checklist + +### Backend +- [ ] Create endpoints +- [ ] Add validation +- [ ] Write unit tests + +### Frontend +- [ ] Create components +- [ ] Add form handling +- [ ] Handle errors + +### Tests +- [x] API tests generated (failing) +- [x] E2E tests generated (failing) +- [ ] Make tests pass +``` + +**Test Structure:** + +```typescript +// tests/api/profile.spec.ts +import { test, expect } from '@playwright/test'; + +test('should fetch user profile', async ({ request }) => { + const response = await request.get('/api/profile'); + expect(response.status()).toBe(200); // FAILS - endpoint doesn't exist yet +}); + +// tests/e2e/profile.spec.ts +test('should display profile page', async ({ page }) => { + await page.goto('/profile'); + await expect(page.getByText('Profile')).toBeVisible(); // FAILS - page doesn't exist +}); +``` + +### Optional: Recording Mode + +If MCP enhancements enabled (`tea_use_mcp_enhancements: true` in config): + +When prompted, select "recording mode" to: +- Verify selectors against actual UI with live browser +- Capture network requests in real-time +- Generate accurate locators from actual DOM + +Note: Recording mode assumes feature partially exists (use for refinement). + +### TDD Workflow + +1. **Red**: Run `*atdd` → tests fail +2. **Green**: Implement feature → tests pass +3. **Refactor**: Improve code → tests still pass + +### Related Commands + +- `*test-design` - Run first for better test generation +- `*automate` - Use after implementation for additional tests +- `*test-review` - Audit generated test quality + +### How-To Guide + +[How to Run ATDD](/docs/how-to/workflows/run-atdd.md) + +--- + +## *automate + +Expand test automation coverage after story implementation. + +### Purpose + +Generate comprehensive tests for existing features, avoiding duplicate coverage. + +### Phase + +Phase 4 (Implementation) - After implementing each story. + +### Frequency + +Per story or feature (after implementation complete). + +### When to Use + +- Feature already exists and works +- Want to add test coverage +- Need tests that pass immediately +- Expanding existing test suite + +**Don't use if:** +- Feature doesn't exist yet (use `*atdd` instead) +- Want failing tests to guide development + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| What are you testing? | "TodoMVC React app" | Feature/app description | +| Reference docs | "test-design-epic-1.md" | Optional test design | +| Specific scenarios | "Cover P0 and P1 from test design" | Focus areas | +| Existing tests | "tests/e2e/basic.spec.ts" | Avoid duplication | + +### Modes + +**BMad-Integrated Mode:** +- Works with story, tech-spec, PRD, test-design +- Comprehensive context for test generation +- Recommended for BMad Method projects + +**Standalone Mode:** +- Analyzes codebase independently +- Works without BMad artifacts +- Good for TEA Solo usage + +### Outputs + +**Comprehensive Test Suite:** + +``` +tests/ +├── e2e/ +│ ├── profile-view.spec.ts # View profile tests +│ ├── profile-edit.spec.ts # Edit profile tests +│ └── profile-validation.spec.ts # Validation tests +├── api/ +│ ├── profile-crud.spec.ts # CRUD operations +│ └── profile-auth.spec.ts # Authorization tests +└── component/ + ├── ProfileForm.test.tsx # Component tests (Vitest for Playwright) + └── ProfileForm.cy.tsx # Component tests (Cypress CT) +``` + +**Component Testing Note:** Framework-dependent - Cypress users get Cypress CT, Playwright users get Vitest tests. + +**Test Quality Features:** +- Network-first patterns (waits for responses, not timeouts) +- Explicit assertions (no conditionals) +- Self-cleaning (tests clean up after themselves) +- Deterministic (no flakiness) + +**Additional Artifacts:** +- **Updated fixtures** - Shared test utilities +- **Updated factories** - Test data generation +- **README updates** - How to run new tests +- **Definition of Done summary** - Quality checklist + +### Prioritization + +TEA generates tests based on: +- Test design priorities (P0 → P1 → P2 → P3) +- Risk assessment scores +- Existing test coverage (avoids duplication) + +**Example:** +``` +Generated 12 tests: +- 4 P0 tests (critical path) +- 5 P1 tests (high value) +- 3 P2 tests (medium value) +- Skipped P3 (low value) +``` + +### Optional: Healing Mode + +If MCP enhancements enabled (`tea_use_mcp_enhancements: true` in config): + +When prompted, select "healing mode" to: +- Fix broken selectors with visual debugging +- Update outdated assertions interactively +- Enhance tests with trace viewer insights + +See [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) + +### Optional: Recording Mode + +If MCP enhancements enabled: + +When prompted, select "recording mode" to verify tests with live browser for accurate selectors. + +### Related Commands + +- `*test-design` - Run first for prioritized test generation +- `*atdd` - Use before implementation (TDD approach) +- `*test-review` - Audit generated test quality + +### How-To Guide + +[How to Run Automate](/docs/how-to/workflows/run-automate.md) + +--- + +## *test-review + +Review test quality using comprehensive knowledge base and best practices. + +### Purpose + +Audit test suite quality with 0-100 scoring and actionable feedback. + +### Phase + +- **Phase 4:** Optional per-story review +- **Release Gate:** Final audit before release + +### Frequency + +- Per story (optional) +- Per epic (recommended) +- Before release (recommended for quality gates, required if using formal gate process) + +### When to Use + +- Want to validate test quality +- Need objective quality metrics +- Preparing for release gate +- Reviewing team-written tests +- Auditing AI-generated tests + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| Review scope | "tests/e2e/ directory" | File, directory, or entire suite | +| Focus areas | "Check for flakiness patterns" | Optional specific concerns | +| Strictness level | "Standard" | Relaxed, standard, or strict | + +### Review Criteria + +TEA reviews against knowledge base patterns: + +**Determinism (35 points):** +- No hard waits (`waitForTimeout`) +- No conditionals (if/else) for flow control +- No try-catch for flow control +- Network-first patterns used + +**Isolation (25 points):** +- Self-cleaning (cleanup after test) +- No global state dependencies +- Can run in parallel +- Independent of execution order + +**Assertions (20 points):** +- Explicit in test body (not abstracted) +- Specific and meaningful +- Covers actual behavior +- No weak assertions (`toBeTrue`, `toBeDefined`) + +**Structure (10 points):** +- Test size < 300 lines +- Clear describe/test names +- Proper setup/teardown +- Single responsibility per test + +**Performance (10 points):** +- Execution time < 1.5 minutes +- Efficient selectors +- Minimal redundant actions + +### Outputs + +**Quality Report** (`test-review.md`): + +```markdown +# Test Quality Review Report + +**Date:** 2026-01-13 +**Scope:** tests/e2e/ +**Score:** 78/100 + +## Summary + +- **Tests Reviewed:** 15 +- **Passing Quality:** 12 tests (80%) +- **Needs Improvement:** 3 tests (20%) +- **Critical Issues:** 2 +- **Recommendations:** 8 + +## Critical Issues + +### 1. Hard Waits Detected (tests/e2e/checkout.spec.ts:45) + +**Issue:** Using `waitForTimeout(3000)` +**Impact:** Flakiness, slow execution +**Fix:** +```typescript +// ❌ Bad +await page.waitForTimeout(3000); + +// ✅ Good +await page.waitForResponse(resp => resp.url().includes('/api/checkout')); +``` + +### 2. Conditional Flow Control (tests/e2e/profile.spec.ts:28) + +**Issue:** Using if/else to handle optional elements +**Impact:** Non-deterministic behavior +**Fix:** +```typescript +// ❌ Bad +if (await page.locator('.banner').isVisible()) { + await page.click('.dismiss'); +} + +// ✅ Good +// Make test deterministic - either banner always shows or doesn't +await expect(page.locator('.banner')).toBeVisible(); +await page.click('.dismiss'); +``` + +## Recommendations + +1. **Extract repeated setup** (tests/e2e/login.spec.ts) - Consider using fixtures +2. **Add network assertions** (tests/e2e/api-calls.spec.ts) - Verify API responses +3. **Improve test names** (tests/e2e/checkout.spec.ts) - Use descriptive names +4. **Reduce test size** (tests/e2e/full-flow.spec.ts) - Split into smaller tests + +## Quality Scores by Category + +| Category | Score | Status | +|----------|-------|--------| +| Determinism | 28/35 | ⚠️ Needs Improvement | +| Isolation | 22/25 | ✅ Good | +| Assertions | 18/20 | ✅ Good | +| Structure | 7/10 | ⚠️ Needs Improvement | +| Performance | 3/10 | ❌ Critical | + +## Next Steps + +1. Fix critical issues (hard waits, conditionals) +2. Address performance concerns (slow tests) +3. Apply recommendations +4. Re-run `*test-review` to verify improvements +``` + +### Review Scope Options + +**Single File:** +``` +*test-review tests/e2e/checkout.spec.ts +``` + +**Directory:** +``` +*test-review tests/e2e/ +``` + +**Entire Suite:** +``` +*test-review tests/ +``` + +### Related Commands + +- `*atdd` - Review tests generated by ATDD +- `*automate` - Review tests generated by automate +- `*trace` - Coverage analysis complements quality review + +### How-To Guide + +[How to Review Test Quality](/docs/how-to/workflows/run-test-review.md) + +--- + +## *nfr-assess + +Validate non-functional requirements before release. + +### Purpose + +Assess security, performance, reliability, and maintainability with evidence-based decisions. + +### Phase + +- **Phase 2:** Optional (enterprise, capture NFRs early) +- **Release Gate:** Validate before release + +### Frequency + +- Per epic (optional) +- Per release (mandatory for enterprise/compliance) + +### When to Use + +- Enterprise projects with compliance needs +- Projects with strict NFRs +- Before production release +- When NFRs are critical to success + +### Inputs + +TEA will ask: + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| NFR focus areas | "Security, Performance" | Categories to assess | +| Thresholds | "API < 200ms P99, 0 critical vulns" | Specific requirements | +| Evidence location | "Load test results in /reports" | Where to find data | + +### NFR Categories + +**Security:** +- Authentication/authorization +- Data encryption +- Vulnerability scanning +- Security headers +- Input validation + +**Performance:** +- Response time (P50, P95, P99) +- Throughput (requests/second) +- Resource usage (CPU, memory) +- Database query performance +- Frontend load time + +**Reliability:** +- Error handling +- Recovery mechanisms +- Availability/uptime +- Failover testing +- Data backup/restore + +**Maintainability:** +- Code quality metrics +- Test coverage +- Technical debt tracking +- Documentation completeness +- Dependency health + +### Outputs + +**NFR Assessment Report** (`nfr-assessment.md`): + +```markdown +# Non-Functional Requirements Assessment + +**Date:** 2026-01-13 +**Epic:** User Profile Management +**Decision:** CONCERNS ⚠️ + +## Summary + +- **Security:** PASS ✅ +- **Performance:** CONCERNS ⚠️ +- **Reliability:** PASS ✅ +- **Maintainability:** PASS ✅ + +## Security Assessment + +**Status:** PASS ✅ + +**Requirements:** +- All endpoints require authentication: ✅ Verified +- Data encryption at rest: ✅ PostgreSQL TDE enabled +- Input validation: ✅ Zod schemas on all endpoints +- No critical vulnerabilities: ✅ npm audit clean + +**Evidence:** +- Security scan report: `/reports/security-scan.pdf` +- Auth tests: 15/15 passing +- Penetration test results: No critical findings + +## Performance Assessment + +**Status:** CONCERNS ⚠️ + +**Requirements:** +| Metric | Target | Actual | Status | +|--------|--------|--------|--------| +| API response (P99) | < 200ms | 350ms | ❌ | +| API response (P95) | < 150ms | 180ms | ⚠️ | +| Throughput | > 1000 rps | 850 rps | ⚠️ | +| Frontend load | < 2s | 1.8s | ✅ | + +**Issues:** +1. **P99 latency exceeds target** - Database queries not optimized +2. **Throughput below target** - Missing database indexes + +**Mitigation Plan:** +- Add indexes to profile queries (owner: backend team, deadline: before release) +- Implement query caching (owner: backend team, deadline: before release) +- Re-run load tests after optimization + +**Evidence:** +- Load test report: `/reports/k6-load-test.json` +- APM data: Datadog dashboard link + +## Reliability Assessment + +**Status:** PASS ✅ + +**Requirements:** +- Error handling: ✅ All endpoints return structured errors +- Recovery: ✅ Graceful degradation tested +- Database failover: ✅ Tested successfully + +**Evidence:** +- Chaos engineering test results +- Error rate in staging: 0.01% (target < 0.1%) + +## Maintainability Assessment + +**Status:** PASS ✅ + +**Requirements:** +- Test coverage: ✅ 85% (target > 80%) +- Code quality: ✅ SonarQube grade A +- Documentation: ✅ API docs complete + +**Evidence:** +- Coverage report: `/reports/coverage/index.html` +- SonarQube: Link to project dashboard + +## Gate Decision + +**Decision:** CONCERNS ⚠️ + +**Rationale:** +- Performance metrics below target (P99, throughput) +- Mitigation plan in place with clear owners and deadlines +- Security and reliability meet requirements + +**Actions Required:** +1. Optimize database queries (backend team, 3 days) +2. Re-run performance tests (QA team, 1 day) +3. Update this assessment with new results + +**Waiver Option:** +If business approves deploying with current performance: +- Document waiver justification +- Set monitoring alerts for P99 latency +- Plan optimization for next release +``` + +### Decision Rules + +**PASS** ✅: All NFRs met, no concerns +**CONCERNS** ⚠️: Some NFRs not met, mitigation plan exists +**FAIL** ❌: Critical NFRs not met, blocks release +**WAIVED** ⏭️: Business-approved waiver with documented justification + +### Critical Principle + +**Never guess thresholds.** If you don't know the NFR target, mark as CONCERNS and request clarification. + +### Related Commands + +- `*trace` - Coverage traceability complements NFR assessment +- `*test-review` - Quality assessment complements NFR + +### How-To Guide + +[How to Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) + +--- + +## *trace + +Map requirements to tests (Phase 1) and make quality gate decision (Phase 2). + +### Purpose + +Two-phase workflow: traceability analysis followed by go/no-go gate decision. + +### Phase + +- **Phase 2 (Baseline):** Brownfield projects establishing baseline +- **Phase 4 (Refresh):** After each story/epic to update coverage +- **Release Gate:** Final gate decision before deployment + +### Frequency + +- Brownfield Phase 2: Once (baseline) +- Phase 4: Per story or epic (refresh) +- Release Gate: Once (final gate decision) + +### Two-Phase Workflow + +### Phase 1: Requirements Traceability + +Map acceptance criteria to implemented tests. + +**Inputs:** + +| Question | Example Answer | Notes | +|----------|----------------|-------| +| Requirements source | "story-123.md, test-design-epic-1.md" | Where are acceptance criteria? | +| Test location | "tests/" | Where are tests? | +| Focus areas | "Profile CRUD operations" | Optional scope | + +**Outputs:** + +**Coverage Matrix** (`traceability-matrix.md`): + +```markdown +# Requirements Traceability Matrix + +**Date:** 2026-01-13 +**Scope:** Epic 1 - User Profile Management + +## Coverage Summary + +- **Total Requirements:** 12 +- **Full Coverage:** 8 (67%) +- **Partial Coverage:** 3 (25%) +- **No Coverage:** 1 (8%) + +## Detailed Traceability + +### Requirement 1: User can view their profile + +**Acceptance Criteria:** +- User navigates to /profile +- Profile displays name, email, avatar + +**Test Coverage:** FULL ✅ + +**Tests:** +- `tests/e2e/profile-view.spec.ts:10` - "should display profile page" +- `tests/api/profile.spec.ts:5` - "should fetch user profile" + +### Requirement 2: User can edit profile + +**Acceptance Criteria:** +- User clicks "Edit Profile" +- Can modify name and email +- Can upload avatar +- Changes are saved + +**Test Coverage:** PARTIAL ⚠️ + +**Tests:** +- `tests/e2e/profile-edit.spec.ts:15` - "should edit and save profile" (name/email only) +- `tests/api/profile.spec.ts:20` - "should update profile via API" + +**Missing Coverage:** +- Avatar upload not tested + +### Requirement 3: Invalid email shows error + +**Acceptance Criteria:** +- Enter invalid email format +- See error message +- Cannot save + +**Test Coverage:** FULL ✅ + +**Tests:** +- `tests/e2e/profile-edit.spec.ts:35` - "should show validation error" +- `tests/api/profile.spec.ts:40` - "should validate email format" + +### Requirement 12: Profile history + +**Acceptance Criteria:** +- View audit log of profile changes + +**Test Coverage:** NONE ❌ + +**Gap Analysis:** +- Priority: P2 (medium) +- Risk: Low (audit feature, not critical path) +- Recommendation: Add in next iteration + +## Gap Prioritization + +| Gap | Priority | Risk | Recommendation | +|-----|----------|------|----------------| +| Avatar upload not tested | P1 | Medium | Add before release | +| Profile history not tested | P2 | Low | Add in next iteration | + +## Recommendations + +1. **Add avatar upload tests** (High priority) + - E2E test for upload flow + - API test for image validation + +2. **Add profile history tests** (Medium priority) + - Can defer to next release + - Low risk (non-critical feature) +``` + +### Phase 2: Quality Gate Decision + +Make go/no-go decision for release. + +**Inputs:** +- Phase 1 traceability results +- Test review results (if available) +- NFR assessment results (if available) +- Business context (deadlines, criticality) + +**Gate Decision Rules:** + +| Coverage | Test Quality | NFRs | Decision | +|----------|--------------|------|----------| +| >95% P0/P1 | >80 score | PASS | PASS ✅ | +| >85% P0/P1 | >70 score | CONCERNS | CONCERNS ⚠️ | +| <85% P0/P1 | <70 score | FAIL | FAIL ❌ | +| Any | Any | Any | WAIVED ⏭️ (with approval) | + +**Outputs:** + +**Gate Decision** (written to traceability-matrix.md or separate gate file): + +```yaml +decision: PASS +date: 2026-01-13 +epic: User Profile Management +release: v1.2.0 + +summary: + total_requirements: 12 + covered_requirements: 11 + coverage_percentage: 92% + critical_gaps: 0 + test_quality_score: 82 + +criteria: + p0_coverage: 100% + p1_coverage: 90% + test_quality: 82 + nfr_assessment: PASS + +rationale: | + All P0 requirements have full test coverage. + One P1 gap (avatar upload) has low risk. + Test quality exceeds 80% threshold. + NFR assessment passed. + +actions: + - Add avatar upload tests in next iteration (P1) + - Monitor profile performance in production + +approvers: + - name: Product Manager + approved: true + date: 2026-01-13 + - name: Tech Lead + approved: true + date: 2026-01-13 + +next_steps: + - Deploy to staging + - Run smoke tests + - Deploy to production +``` + +### Usage Patterns + +**Greenfield:** +- Run Phase 1 after Phase 3 (system-level test design) +- Run Phase 1 refresh in Phase 4 (per epic) +- Run Phase 2 at release gate + +**Brownfield:** +- Run Phase 1 in Phase 2 (establish baseline) +- Run Phase 1 refresh in Phase 4 (per epic) +- Run Phase 2 at release gate + +### Related Commands + +- `*test-design` - Provides requirements for traceability +- `*test-review` - Quality scores feed gate decision +- `*nfr-assess` - NFR results feed gate decision + +### How-To Guide + +[How to Run Trace](/docs/how-to/workflows/run-trace.md) + +--- + +## Summary Table + +| Command | Phase | Frequency | Purpose | Output | +|---------|-------|-----------|---------|--------| +| `*framework` | 3 | Once | Scaffold test framework | Config, sample tests | +| `*ci` | 3 | Once | Setup CI/CD pipeline | Workflow files, scripts | +| `*test-design` | 3, 4 | Once (system), Per epic | Risk-based test planning | Test design documents | +| `*atdd` | 4 | Per story (optional) | Generate failing tests (TDD) | Failing tests, checklist | +| `*automate` | 4 | Per story/feature | Generate passing tests | Passing tests, fixtures | +| `*test-review` | 4, Gate | Per epic/release | Audit test quality | Quality report (0-100) | +| `*nfr-assess` | 2, Gate | Per release | Validate NFRs | NFR assessment report | +| `*trace` | 2, 4, Gate | Baseline, refresh, gate | Coverage + gate decision | Coverage matrix, gate YAML | + +--- + +## See Also + +### How-To Guides +- [Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) +- [Run Test Design](/docs/how-to/workflows/run-test-design.md) +- [Run ATDD](/docs/how-to/workflows/run-atdd.md) +- [Run Automate](/docs/how-to/workflows/run-automate.md) +- [Run Test Review](/docs/how-to/workflows/run-test-review.md) +- [Set Up CI Pipeline](/docs/how-to/workflows/setup-ci.md) +- [Run NFR Assessment](/docs/how-to/workflows/run-nfr-assess.md) +- [Run Trace](/docs/how-to/workflows/run-trace.md) + +### Explanation +- [TEA Overview](/docs/explanation/features/tea-overview.md) +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) +- [Test Quality Standards](/docs/explanation/tea/test-quality-standards.md) + +### Reference +- [TEA Configuration](/docs/reference/tea/configuration.md) +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) +- [Glossary](/docs/reference/glossary/index.md) + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/reference/tea/configuration.md b/docs/reference/tea/configuration.md new file mode 100644 index 00000000..60688b4a --- /dev/null +++ b/docs/reference/tea/configuration.md @@ -0,0 +1,757 @@ +--- +title: "TEA Configuration Reference" +description: Complete reference for TEA configuration options and file locations +--- + +# TEA Configuration Reference + +Complete reference for all TEA (Test Architect) configuration options. + +## Configuration File Locations + +### User Configuration (Installer-Generated) + +**Location:** `_bmad/bmm/config.yaml` + +**Purpose:** Project-specific configuration values for your repository + +**Created By:** `npx bmad-method@alpha install` command + +**Status:** Gitignored (not committed to repository) + +**Usage:** Edit this file to change TEA behavior in your project + +**Example:** +```yaml +# _bmad/bmm/config.yaml +project_name: my-awesome-app +user_skill_level: intermediate +output_folder: _bmad-output +tea_use_playwright_utils: true +tea_use_mcp_enhancements: false +``` + +### Canonical Schema (Source of Truth) + +**Location:** `src/modules/bmm/module.yaml` + +**Purpose:** Defines available configuration keys, defaults, and installer prompts + +**Created By:** BMAD maintainers (part of BMAD repo) + +**Status:** Versioned in BMAD repository + +**Usage:** Reference only (do not edit unless contributing to BMAD) + +**Note:** The installer reads `module.yaml` to prompt for config values, then writes user choices to `_bmad/bmm/config.yaml` in your project. + +--- + +## TEA Configuration Options + +### tea_use_playwright_utils + +Enable Playwright Utils integration for production-ready fixtures and utilities. + +**Schema Location:** `src/modules/bmm/module.yaml:52-56` + +**User Config:** `_bmad/bmm/config.yaml` + +**Type:** `boolean` + +**Default:** `false` (set via installer prompt during installation) + +**Installer Prompt:** +``` +Are you using playwright-utils (@seontechnologies/playwright-utils) in your project? +You must install packages yourself, or use test architect's *framework command. +``` + +**Purpose:** Enables TEA to: +- Include playwright-utils in `*framework` scaffold +- Generate tests using playwright-utils fixtures +- Review tests against playwright-utils patterns +- Configure CI with burn-in and selective testing utilities + +**Affects Workflows:** +- `*framework` - Includes playwright-utils imports and fixture examples +- `*atdd` - Uses fixtures like `apiRequest`, `authSession` in generated tests +- `*automate` - Leverages utilities for test patterns +- `*test-review` - Reviews against playwright-utils best practices +- `*ci` - Includes burn-in utility and selective testing + +**Example (Enable):** +```yaml +tea_use_playwright_utils: true +``` + +**Example (Disable):** +```yaml +tea_use_playwright_utils: false +``` + +**Prerequisites:** +```bash +npm install -D @seontechnologies/playwright-utils +``` + +**Related:** +- [Integrate Playwright Utils Guide](/docs/how-to/customization/integrate-playwright-utils.md) +- [Playwright Utils on npm](https://www.npmjs.com/package/@seontechnologies/playwright-utils) + +--- + +### tea_use_mcp_enhancements + +Enable Playwright MCP servers for live browser verification during test generation. + +**Schema Location:** `src/modules/bmm/module.yaml:47-50` + +**User Config:** `_bmad/bmm/config.yaml` + +**Type:** `boolean` + +**Default:** `false` + +**Installer Prompt:** +``` +Test Architect Playwright MCP capabilities (healing, exploratory, verification) are optionally available. +You will have to setup your MCPs yourself; refer to https://docs.bmad-method.org/explanation/features/tea-overview for configuration examples. +Would you like to enable MCP enhancements in Test Architect? +``` + +**Purpose:** Enables TEA to use Model Context Protocol servers for: +- Live browser automation during test design +- Selector verification with actual DOM +- Interactive UI discovery +- Visual debugging and healing + +**Affects Workflows:** +- `*test-design` - Enables exploratory mode (browser-based UI discovery) +- `*atdd` - Enables recording mode (verify selectors with live browser) +- `*automate` - Enables healing mode (fix tests with visual debugging) + +**MCP Servers Required:** + +**Two Playwright MCP servers** (actively maintained, continuously updated): + +- `playwright` - Browser automation (`npx @playwright/mcp@latest`) +- `playwright-test` - Test runner with failure analysis (`npx playwright run-test-mcp-server`) + +**Configuration example**: + +```json +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": ["@playwright/mcp@latest"] + }, + "playwright-test": { + "command": "npx", + "args": ["playwright", "run-test-mcp-server"] + } + } +} +``` + +**Configuration Location (IDE-Specific):** + +**Cursor:** +``` +~/.cursor/config.json or workspace .cursor/config.json +``` + +**VS Code with Claude:** +``` +~/Library/Application Support/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json +``` + +**Example (Enable):** +```yaml +tea_use_mcp_enhancements: true +``` + +**Example (Disable):** +```yaml +tea_use_mcp_enhancements: false +``` + +**Prerequisites:** +1. MCP servers installed in IDE configuration +2. `@playwright/mcp` package available globally or locally +3. Browser binaries installed (`npx playwright install`) + +**Related:** +- [Enable MCP Enhancements Guide](/docs/how-to/customization/enable-tea-mcp-enhancements.md) +- [TEA Overview - MCP Section](/docs/explanation/features/tea-overview.md#playwright-mcp-enhancements) +- [Playwright MCP on npm](https://www.npmjs.com/package/@playwright/mcp) + +--- + +## Core BMM Configuration (Inherited by TEA) + +TEA also uses core BMM configuration options from `_bmad/bmm/config.yaml`: + +### output_folder + +**Type:** `string` + +**Default:** `_bmad-output` + +**Purpose:** Where TEA writes output files (test designs, reports, traceability matrices) + +**Example:** +```yaml +output_folder: _bmad-output +``` + +**TEA Output Files:** +- `test-design-system.md` (from *test-design system-level) +- `test-design-epic-N.md` (from *test-design epic-level) +- `test-review.md` (from *test-review) +- `traceability-matrix.md` (from *trace Phase 1) +- `gate-decision-{gate_type}-{story_id}.md` (from *trace Phase 2) +- `nfr-assessment.md` (from *nfr-assess) +- `automation-summary.md` (from *automate) +- `atdd-checklist-{story_id}.md` (from *atdd) + +--- + +### user_skill_level + +**Type:** `enum` + +**Options:** `beginner` | `intermediate` | `expert` + +**Default:** `intermediate` + +**Purpose:** Affects how TEA explains concepts in chat responses + +**Example:** +```yaml +user_skill_level: beginner +``` + +**Impact on TEA:** +- **Beginner:** More detailed explanations, links to concepts, verbose guidance +- **Intermediate:** Balanced explanations, assumes basic knowledge +- **Expert:** Concise, technical, minimal hand-holding + +--- + +### project_name + +**Type:** `string` + +**Default:** Directory name + +**Purpose:** Used in TEA-generated documentation and reports + +**Example:** +```yaml +project_name: my-awesome-app +``` + +**Used in:** +- Report headers +- Documentation titles +- CI configuration comments + +--- + +### communication_language + +**Type:** `string` + +**Default:** `english` + +**Purpose:** Language for TEA chat responses + +**Example:** +```yaml +communication_language: english +``` + +**Supported:** Any language (TEA responds in specified language) + +--- + +### document_output_language + +**Type:** `string` + +**Default:** `english` + +**Purpose:** Language for TEA-generated documents (test designs, reports) + +**Example:** +```yaml +document_output_language: english +``` + +**Note:** Can differ from `communication_language` - chat in Spanish, generate docs in English. + +--- + +## Environment Variables + +TEA workflows may use environment variables for test configuration. + +### Test Framework Variables + +**Playwright:** +```bash +# .env +BASE_URL=https://todomvc.com/examples/react/ +API_BASE_URL=https://api.example.com +TEST_USER_EMAIL=test@example.com +TEST_USER_PASSWORD=password123 +``` + +**Cypress:** +```bash +# cypress.env.json or .env +CYPRESS_BASE_URL=https://example.com +CYPRESS_API_URL=https://api.example.com +``` + +### CI/CD Variables + +Set in CI platform (GitHub Actions secrets, GitLab CI variables): + +```yaml +# .github/workflows/test.yml +env: + BASE_URL: ${{ secrets.STAGING_URL }} + API_KEY: ${{ secrets.API_KEY }} + TEST_USER_EMAIL: ${{ secrets.TEST_USER }} +``` + +--- + +## Configuration Patterns + +### Development vs Production + +**Separate configs for environments:** + +```yaml +# _bmad/bmm/config.yaml +output_folder: _bmad-output + +# .env.development +BASE_URL=http://localhost:3000 +API_BASE_URL=http://localhost:4000 + +# .env.staging +BASE_URL=https://staging.example.com +API_BASE_URL=https://api-staging.example.com + +# .env.production (read-only tests only!) +BASE_URL=https://example.com +API_BASE_URL=https://api.example.com +``` + +### Team vs Individual + +**Team config (committed):** +```yaml +# _bmad/bmm/config.yaml.example (committed to repo) +project_name: team-project +output_folder: _bmad-output +tea_use_playwright_utils: true +tea_use_mcp_enhancements: false +``` + +**Individual config (gitignored):** +```yaml +# _bmad/bmm/config.yaml (gitignored) +user_name: John Doe +user_skill_level: expert +tea_use_mcp_enhancements: true # Individual preference +``` + +### Monorepo Configuration + +**Root config:** +```yaml +# _bmad/bmm/config.yaml (root) +project_name: monorepo-parent +output_folder: _bmad-output +``` + +**Package-specific:** +```yaml +# packages/web-app/_bmad/bmm/config.yaml +project_name: web-app +output_folder: ../../_bmad-output/web-app +tea_use_playwright_utils: true + +# packages/mobile-app/_bmad/bmm/config.yaml +project_name: mobile-app +output_folder: ../../_bmad-output/mobile-app +tea_use_playwright_utils: false +``` + +--- + +## Configuration Best Practices + +### 1. Use Version Control Wisely + +**Commit:** +``` +_bmad/bmm/config.yaml.example # Template for team +.nvmrc # Node version +package.json # Dependencies +``` + +**Gitignore:** +``` +_bmad/bmm/config.yaml # User-specific values +.env # Secrets +.env.local # Local overrides +``` + +### 2. Document Required Setup + +**In your README:** +```markdown +## Setup + +1. Install BMad: + npx bmad-method@alpha install + +2. Copy config template: + cp _bmad/bmm/config.yaml.example _bmad/bmm/config.yaml + +3. Edit config with your values: + - Set user_name + - Enable tea_use_playwright_utils if using playwright-utils + - Enable tea_use_mcp_enhancements if MCPs configured +``` + +### 3. Validate Configuration + +**Check config is valid:** +```bash +# Check TEA config is set +cat _bmad/bmm/config.yaml | grep tea_use + +# Verify playwright-utils installed (if enabled) +npm list @seontechnologies/playwright-utils + +# Verify MCP servers configured (if enabled) +# Check your IDE's MCP settings +``` + +### 4. Keep Config Minimal + +**Don't over-configure:** +```yaml +# ❌ Bad - overriding everything unnecessarily +project_name: my-project +user_name: John Doe +user_skill_level: expert +output_folder: custom/path +planning_artifacts: custom/planning +implementation_artifacts: custom/implementation +project_knowledge: custom/docs +tea_use_playwright_utils: true +tea_use_mcp_enhancements: true +communication_language: english +document_output_language: english +# Overriding 11 config options when most can use defaults + +# ✅ Good - only essential overrides +tea_use_playwright_utils: true +output_folder: docs/testing +# Only override what differs from defaults +``` + +**Use defaults when possible** - only override what you actually need to change. + +--- + +## Troubleshooting + +### Configuration Not Loaded + +**Problem:** TEA doesn't use my config values. + +**Causes:** +1. Config file in wrong location +2. YAML syntax error +3. Typo in config key + +**Solution:** +```bash +# Check file exists +ls -la _bmad/bmm/config.yaml + +# Validate YAML syntax +npm install -g js-yaml +js-yaml _bmad/bmm/config.yaml + +# Check for typos (compare to module.yaml) +diff _bmad/bmm/config.yaml src/modules/bmm/module.yaml +``` + +### Playwright Utils Not Working + +**Problem:** `tea_use_playwright_utils: true` but TEA doesn't use utilities. + +**Causes:** +1. Package not installed +2. Config file not saved +3. Workflow run before config update + +**Solution:** +```bash +# Verify package installed +npm list @seontechnologies/playwright-utils + +# Check config value +grep tea_use_playwright_utils _bmad/bmm/config.yaml + +# Re-run workflow in fresh chat +# (TEA loads config at workflow start) +``` + +### MCP Enhancements Not Working + +**Problem:** `tea_use_mcp_enhancements: true` but no browser opens. + +**Causes:** +1. MCP servers not configured in IDE +2. MCP package not installed +3. Browser binaries missing + +**Solution:** +```bash +# Check MCP package available +npx @playwright/mcp@latest --version + +# Install browsers +npx playwright install + +# Verify IDE MCP config +# Check ~/.cursor/config.json or VS Code settings +``` + +### Config Changes Not Applied + +**Problem:** Updated config but TEA still uses old values. + +**Cause:** TEA loads config at workflow start. + +**Solution:** +1. Save `_bmad/bmm/config.yaml` +2. Start fresh chat +3. Run TEA workflow +4. Config will be reloaded + +**TEA doesn't reload config mid-chat** - always start fresh chat after config changes. + +--- + +## Configuration Examples + +### Minimal Setup (Defaults) + +```yaml +# _bmad/bmm/config.yaml +project_name: my-project +user_skill_level: intermediate +output_folder: _bmad-output +tea_use_playwright_utils: false +tea_use_mcp_enhancements: false +``` + +**Best for:** +- New projects +- Learning TEA +- Simple testing needs + +--- + +### Advanced Setup (All Features) + +```yaml +# _bmad/bmm/config.yaml +project_name: enterprise-app +user_skill_level: expert +output_folder: docs/testing +planning_artifacts: docs/planning +implementation_artifacts: docs/implementation +project_knowledge: docs +tea_use_playwright_utils: true +tea_use_mcp_enhancements: true +``` + +**Prerequisites:** +```bash +npm install -D @seontechnologies/playwright-utils +# Configure MCP servers in IDE +``` + +**Best for:** +- Enterprise projects +- Teams with established testing practices +- Projects needing advanced TEA features + +--- + +### Monorepo Setup + +**Root config:** +```yaml +# _bmad/bmm/config.yaml (root) +project_name: monorepo +output_folder: _bmad-output +tea_use_playwright_utils: true +``` + +**Package configs:** +```yaml +# apps/web/_bmad/bmm/config.yaml +project_name: web-app +output_folder: ../../_bmad-output/web + +# apps/api/_bmad/bmm/config.yaml +project_name: api-service +output_folder: ../../_bmad-output/api +tea_use_playwright_utils: false # API tests don't need it +``` + +--- + +### Team Template + +**Commit this template:** +```yaml +# _bmad/bmm/config.yaml.example +# Copy to config.yaml and fill in your values + +project_name: your-project-name +user_name: Your Name +user_skill_level: intermediate # beginner | intermediate | expert +output_folder: _bmad-output +planning_artifacts: _bmad-output/planning-artifacts +implementation_artifacts: _bmad-output/implementation-artifacts +project_knowledge: docs + +# TEA Configuration +tea_use_playwright_utils: false # Set true if using @seontechnologies/playwright-utils +tea_use_mcp_enhancements: false # Set true if MCP servers configured in IDE + +# Languages +communication_language: english +document_output_language: english +``` + +**Team instructions:** +```markdown +## Setup for New Team Members + +1. Clone repo +2. Copy config template: + cp _bmad/bmm/config.yaml.example _bmad/bmm/config.yaml +3. Edit with your name and preferences +4. Install dependencies: + npm install +5. (Optional) Enable playwright-utils: + npm install -D @seontechnologies/playwright-utils + Set tea_use_playwright_utils: true +``` + +--- + +## FAQ + +### When should I enable playwright-utils? + +**Enable if:** +- You're using or planning to use `@seontechnologies/playwright-utils` +- You want production-ready fixtures and utilities +- Your team benefits from standardized patterns +- You need utilities like `apiRequest`, `authSession`, `networkRecorder` + +**Skip if:** +- You're just learning TEA (keep it simple) +- You have your own fixture library +- You don't need the utilities + +### When should I enable MCP enhancements? + +**Enable if:** +- You want live browser verification during test generation +- You're debugging complex UI issues +- You want exploratory mode in `*test-design` +- You want recording mode in `*atdd` for accurate selectors + +**Skip if:** +- You're new to TEA (adds complexity) +- You don't have MCP servers configured +- Your tests work fine without it + +### Can I change config after installation? + +**Yes!** Edit `_bmad/bmm/config.yaml` anytime. + +**Important:** Start fresh chat after config changes (TEA loads config at workflow start). + +### Can I have different configs per branch? + +**Yes:** +```bash +# feature branch +git checkout feature/new-testing +# Edit config for experimentation +vim _bmad/bmm/config.yaml + +# main branch +git checkout main +# Config reverts to main branch values +``` + +Config is gitignored, so each branch can have different values. + +### How do I share config with team? + +**Use config.yaml.example:** +```bash +# Commit template +cp _bmad/bmm/config.yaml _bmad/bmm/config.yaml.example +git add _bmad/bmm/config.yaml.example +git commit -m "docs: add BMad config template" +``` + +**Team members copy template:** +```bash +cp _bmad/bmm/config.yaml.example _bmad/bmm/config.yaml +# Edit with their values +``` + +--- + +## See Also + +### How-To Guides +- [Set Up Test Framework](/docs/how-to/workflows/setup-test-framework.md) +- [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) +- [Enable MCP Enhancements](/docs/how-to/customization/enable-tea-mcp-enhancements.md) + +### Reference +- [TEA Command Reference](/docs/reference/tea/commands.md) +- [Knowledge Base Index](/docs/reference/tea/knowledge-base.md) +- [Glossary](/docs/reference/glossary/index.md) + +### Explanation +- [TEA Overview](/docs/explanation/features/tea-overview.md) +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/reference/tea/knowledge-base.md b/docs/reference/tea/knowledge-base.md new file mode 100644 index 00000000..24fb7877 --- /dev/null +++ b/docs/reference/tea/knowledge-base.md @@ -0,0 +1,583 @@ +--- +title: "TEA Knowledge Base Index" +description: Complete index of TEA's 33 knowledge fragments for context engineering +--- + +# TEA Knowledge Base Index + +TEA uses 33 specialized knowledge fragments for context engineering. These fragments are loaded dynamically based on workflow needs via the `tea-index.csv` manifest. + +## What is Context Engineering? + +**Context engineering** is the practice of loading domain-specific standards into AI context automatically rather than relying on prompts alone. + +Instead of asking AI to "write good tests" every time, TEA: +1. Reads `tea-index.csv` to identify relevant fragments for the workflow +2. Loads only the fragments needed (keeps context focused) +3. Operates with domain-specific standards, not generic knowledge +4. Produces consistent, production-ready tests across projects + +**Example:** +``` +User runs: *test-design + +TEA reads tea-index.csv: +- Loads: test-quality.md, test-priorities-matrix.md, risk-governance.md +- Skips: network-recorder.md, burn-in.md (not needed for test design) + +Result: Focused context, consistent quality standards +``` + +## How Knowledge Loading Works + +### 1. Workflow Trigger +User runs a TEA workflow (e.g., `*test-design`) + +### 2. Manifest Lookup +TEA reads `src/modules/bmm/testarch/tea-index.csv`: +```csv +id,name,description,tags,fragment_file +test-quality,Test Quality,Execution limits and isolation rules,quality;standards,knowledge/test-quality.md +risk-governance,Risk Governance,Risk scoring and gate decisions,risk;governance,knowledge/risk-governance.md +``` + +### 3. Dynamic Loading +Only fragments needed for the workflow are loaded into context + +### 4. Consistent Output +AI operates with established patterns, producing consistent results + +## Fragment Categories + +### Architecture & Fixtures + +Core patterns for test infrastructure and fixture composition. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [fixture-architecture](../../../src/modules/bmm/testarch/knowledge/fixture-architecture.md) | Pure function → Fixture → mergeTests composition with auto-cleanup | Testability, composition, reusability | +| [network-first](../../../src/modules/bmm/testarch/knowledge/network-first.md) | Intercept-before-navigate workflow, HAR capture, deterministic waits | Flakiness prevention, network patterns | +| [playwright-config](../../../src/modules/bmm/testarch/knowledge/playwright-config.md) | Environment switching, timeout standards, artifact outputs | Configuration, environments, CI | +| [fixtures-composition](../../../src/modules/bmm/testarch/knowledge/fixtures-composition.md) | mergeTests composition patterns for combining utilities | Fixture merging, utility composition | + +**Used in:** *framework, *test-design, *atdd, *automate, *test-review + +--- + +### Data & Setup + +Patterns for test data generation, authentication, and setup. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [data-factories](../../../src/modules/bmm/testarch/knowledge/data-factories.md) | Factory patterns with faker, overrides, API seeding, cleanup | Test data, factories, cleanup | +| [email-auth](../../../src/modules/bmm/testarch/knowledge/email-auth.md) | Magic link extraction, state preservation, negative flows | Authentication, email testing | +| [auth-session](../../../src/modules/bmm/testarch/knowledge/auth-session.md) | Token persistence, multi-user, API/browser authentication | Auth patterns, session management | + +**Used in:** *framework, *atdd, *automate, *test-review + +--- + +### Network & Reliability + +Network interception, error handling, and reliability patterns. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [network-recorder](../../../src/modules/bmm/testarch/knowledge/network-recorder.md) | HAR record/playback, CRUD detection for offline testing | Offline testing, network replay | +| [intercept-network-call](../../../src/modules/bmm/testarch/knowledge/intercept-network-call.md) | Network spy/stub, JSON parsing for UI tests | Mocking, interception, stubbing | +| [error-handling](../../../src/modules/bmm/testarch/knowledge/error-handling.md) | Scoped exception handling, retry validation, telemetry logging | Error patterns, resilience | +| [network-error-monitor](../../../src/modules/bmm/testarch/knowledge/network-error-monitor.md) | HTTP 4xx/5xx detection for UI tests | Error detection, monitoring | + +**Used in:** *atdd, *automate, *test-review + +--- + +### Test Execution & CI + +CI/CD patterns, burn-in testing, and selective test execution. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [ci-burn-in](../../../src/modules/bmm/testarch/knowledge/ci-burn-in.md) | Staged jobs, shard orchestration, burn-in loops | CI/CD, flakiness detection | +| [burn-in](../../../src/modules/bmm/testarch/knowledge/burn-in.md) | Smart test selection, git diff for CI optimization | Test selection, performance | +| [selective-testing](../../../src/modules/bmm/testarch/knowledge/selective-testing.md) | Tag/grep usage, spec filters, diff-based runs | Test filtering, optimization | + +**Used in:** *ci, *test-review + +--- + +### Quality & Standards + +Test quality standards, test level selection, and TDD patterns. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [test-quality](../../../src/modules/bmm/testarch/knowledge/test-quality.md) | Execution limits, isolation rules, green criteria | DoD, best practices, anti-patterns | +| [test-levels-framework](../../../src/modules/bmm/testarch/knowledge/test-levels-framework.md) | Guidelines for unit, integration, E2E selection | Test pyramid, level selection | +| [test-priorities-matrix](../../../src/modules/bmm/testarch/knowledge/test-priorities-matrix.md) | P0-P3 criteria, coverage targets, execution ordering | Prioritization, risk-based testing | +| [test-healing-patterns](../../../src/modules/bmm/testarch/knowledge/test-healing-patterns.md) | Common failure patterns and automated fixes | Debugging, healing, fixes | +| [component-tdd](../../../src/modules/bmm/testarch/knowledge/component-tdd.md) | Red→green→refactor workflow, provider isolation | TDD, component testing | + +**Used in:** *test-design, *atdd, *automate, *test-review, *trace + +--- + +### Risk & Gates + +Risk assessment, governance, and gate decision frameworks. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [risk-governance](../../../src/modules/bmm/testarch/knowledge/risk-governance.md) | Scoring matrix, category ownership, gate decision rules | Risk assessment, governance | +| [probability-impact](../../../src/modules/bmm/testarch/knowledge/probability-impact.md) | Probability × impact scale for scoring matrix | Risk scoring, impact analysis | +| [nfr-criteria](../../../src/modules/bmm/testarch/knowledge/nfr-criteria.md) | Security, performance, reliability, maintainability status | NFRs, compliance, enterprise | + +**Used in:** *test-design, *nfr-assess, *trace + +--- + +### Selectors & Timing + +Selector resilience, race condition debugging, and visual debugging. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [selector-resilience](../../../src/modules/bmm/testarch/knowledge/selector-resilience.md) | Robust selector strategies and debugging | Selectors, locators, resilience | +| [timing-debugging](../../../src/modules/bmm/testarch/knowledge/timing-debugging.md) | Race condition identification and deterministic fixes | Race conditions, timing issues | +| [visual-debugging](../../../src/modules/bmm/testarch/knowledge/visual-debugging.md) | Trace viewer usage, artifact expectations | Debugging, trace viewer, artifacts | + +**Used in:** *atdd, *automate, *test-review + +--- + +### Feature Flags & Testing Patterns + +Feature flag testing, contract testing, and API testing patterns. + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| [feature-flags](../../../src/modules/bmm/testarch/knowledge/feature-flags.md) | Enum management, targeting helpers, cleanup, checklists | Feature flags, toggles | +| [contract-testing](../../../src/modules/bmm/testarch/knowledge/contract-testing.md) | Pact publishing, provider verification, resilience | Contract testing, Pact | +| [api-testing-patterns](../../../src/modules/bmm/testarch/knowledge/api-testing-patterns.md) | Pure API patterns without browser | API testing, backend testing | + +**Used in:** *test-design, *atdd, *automate + +--- + +### Playwright-Utils Integration + +Patterns for using `@seontechnologies/playwright-utils` package (11 utilities). + +| Fragment | Description | Key Topics | +|----------|-------------|-----------| +| overview | Playwright Utils installation, design principles, fixture patterns | Getting started, principles, setup | +| [api-request](../../../src/modules/bmm/testarch/knowledge/api-request.md) | Typed HTTP client, schema validation, retry logic | API calls, HTTP, validation | +| [auth-session](../../../src/modules/bmm/testarch/knowledge/auth-session.md) | Token persistence, multi-user, API/browser authentication | Auth patterns, session management | +| [network-recorder](../../../src/modules/bmm/testarch/knowledge/network-recorder.md) | HAR record/playback, CRUD detection for offline testing | Offline testing, network replay | +| [intercept-network-call](../../../src/modules/bmm/testarch/knowledge/intercept-network-call.md) | Network spy/stub, JSON parsing for UI tests | Mocking, interception, stubbing | +| [recurse](../../../src/modules/bmm/testarch/knowledge/recurse.md) | Async polling for API responses, background jobs | Polling, eventual consistency | +| [log](../../../src/modules/bmm/testarch/knowledge/log.md) | Structured logging for API and UI tests | Logging, debugging, reporting | +| [file-utils](../../../src/modules/bmm/testarch/knowledge/file-utils.md) | CSV/XLSX/PDF/ZIP handling with download support | File validation, exports | +| [burn-in](../../../src/modules/bmm/testarch/knowledge/burn-in.md) | Smart test selection with git diff analysis | CI optimization, selective testing | +| [network-error-monitor](../../../src/modules/bmm/testarch/knowledge/network-error-monitor.md) | Auto-detect HTTP 4xx/5xx errors during tests | Error monitoring, silent failures | +| [fixtures-composition](../../../src/modules/bmm/testarch/knowledge/fixtures-composition.md) | mergeTests composition patterns for combining utilities | Fixture merging, utility composition | + +**Note:** All 11 playwright-utils fragments are in the same `knowledge/` directory as other fragments. + +**Used in:** *framework (if `tea_use_playwright_utils: true`), *atdd, *automate, *test-review, *ci + +**Official Docs:** https://seontechnologies.github.io/playwright-utils/ + +--- + +## Fragment Manifest (tea-index.csv) + +**Location:** `src/modules/bmm/testarch/tea-index.csv` + +**Purpose:** Tracks all knowledge fragments and their usage in workflows + +**Structure:** +```csv +id,name,description,tags,fragment_file +test-quality,Test Quality,Execution limits and isolation rules,quality;standards,knowledge/test-quality.md +risk-governance,Risk Governance,Risk scoring and gate decisions,risk;governance,knowledge/risk-governance.md +``` + +**Columns:** +- `id` - Unique fragment identifier (kebab-case) +- `name` - Human-readable fragment name +- `description` - What the fragment covers +- `tags` - Searchable tags (semicolon-separated) +- `fragment_file` - Relative path to fragment markdown file + +## Fragment Locations + +**Knowledge Base Directory:** +``` +src/modules/bmm/testarch/knowledge/ +├── api-request.md +├── api-testing-patterns.md +├── auth-session.md +├── burn-in.md +├── ci-burn-in.md +├── component-tdd.md +├── contract-testing.md +├── data-factories.md +├── email-auth.md +├── error-handling.md +├── feature-flags.md +├── file-utils.md +├── fixture-architecture.md +├── fixtures-composition.md +├── intercept-network-call.md +├── log.md +├── network-error-monitor.md +├── network-first.md +├── network-recorder.md +├── nfr-criteria.md +├── playwright-config.md +├── probability-impact.md +├── recurse.md +├── risk-governance.md +├── selector-resilience.md +├── selective-testing.md +├── test-healing-patterns.md +├── test-levels-framework.md +├── test-priorities-matrix.md +├── test-quality.md +├── timing-debugging.md +└── visual-debugging.md +``` + +**All fragments in single directory** (no subfolders) + +**Manifest:** +``` +src/modules/bmm/testarch/tea-index.csv +``` + +## Workflow Fragment Loading + +Each TEA workflow loads specific fragments: + +### *framework +**Key Fragments:** +- fixture-architecture.md +- playwright-config.md +- fixtures-composition.md + +**Purpose:** Test infrastructure patterns and fixture composition + +**Note:** Loads additional fragments based on framework choice (Playwright/Cypress) and config (`tea_use_playwright_utils`). + +--- + +### *test-design +**Key Fragments:** +- test-quality.md +- test-priorities-matrix.md +- test-levels-framework.md +- risk-governance.md +- probability-impact.md + +**Purpose:** Risk assessment and test planning standards + +**Note:** Loads additional fragments based on mode (system-level vs epic-level) and focus areas. + +--- + +### *atdd +**Key Fragments:** +- test-quality.md +- component-tdd.md +- fixture-architecture.md +- network-first.md +- data-factories.md +- selector-resilience.md +- timing-debugging.md +- test-healing-patterns.md + +**Purpose:** TDD patterns and test generation standards + +**Note:** Loads auth, network, and utility fragments based on feature requirements. + +--- + +### *automate +**Key Fragments:** +- test-quality.md +- test-levels-framework.md +- test-priorities-matrix.md +- fixture-architecture.md +- network-first.md +- selector-resilience.md +- test-healing-patterns.md +- timing-debugging.md + +**Purpose:** Comprehensive test generation with quality standards + +**Note:** Loads additional fragments for data factories, auth, network utilities based on test needs. + +--- + +### *test-review +**Key Fragments:** +- test-quality.md +- test-healing-patterns.md +- selector-resilience.md +- timing-debugging.md +- visual-debugging.md +- network-first.md +- test-levels-framework.md +- fixture-architecture.md + +**Purpose:** Comprehensive quality review against all standards + +**Note:** Loads all applicable playwright-utils fragments when `tea_use_playwright_utils: true`. + +--- + +### *ci +**Key Fragments:** +- ci-burn-in.md +- burn-in.md +- selective-testing.md +- playwright-config.md + +**Purpose:** CI/CD best practices and optimization + +--- + +### *nfr-assess +**Key Fragments:** +- nfr-criteria.md +- risk-governance.md +- probability-impact.md + +**Purpose:** NFR assessment frameworks and decision rules + +--- + +### *trace +**Key Fragments:** +- test-priorities-matrix.md +- risk-governance.md +- test-quality.md + +**Purpose:** Traceability and gate decision standards + +**Note:** Loads nfr-criteria.md if NFR assessment is part of gate decision. + +--- + +## Key Fragments Explained + +### test-quality.md + +**What it covers:** +- Execution time limits (< 1.5 minutes) +- Test size limits (< 300 lines) +- No hard waits (waitForTimeout banned) +- No conditionals for flow control +- No try-catch for flow control +- Assertions must be explicit +- Self-cleaning tests for parallel execution + +**Why it matters:** +This is the Definition of Done for test quality. All TEA workflows reference this for quality standards. + +**Code examples:** 12+ + +--- + +### network-first.md + +**What it covers:** +- Intercept-before-navigate pattern +- Wait for network responses, not timeouts +- HAR capture for offline testing +- Deterministic waiting strategies + +**Why it matters:** +Prevents 90% of test flakiness. Core pattern for reliable E2E tests. + +**Code examples:** 15+ + +--- + +### fixture-architecture.md + +**What it covers:** +- Build pure functions first +- Wrap in framework fixtures second +- Compose with mergeTests +- Enable reusability and testability + +**Why it matters:** +Foundation of scalable test architecture. Makes utilities reusable and unit-testable. + +**Code examples:** 10+ + +--- + +### risk-governance.md + +**What it covers:** +- Risk scoring matrix (Probability × Impact) +- Risk categories (TECH, SEC, PERF, DATA, BUS, OPS) +- Gate decision rules (PASS/CONCERNS/FAIL/WAIVED) +- Mitigation planning + +**Why it matters:** +Objective, data-driven release decisions. Removes politics from quality gates. + +**Code examples:** 5 + +--- + +### test-priorities-matrix.md + +**What it covers:** +- P0: Critical path (100% coverage required) +- P1: High value (90% coverage target) +- P2: Medium value (50% coverage target) +- P3: Low value (20% coverage target) +- Execution ordering (P0 → P1 → P2 → P3) + +**Why it matters:** +Focus testing effort on what matters. Don't waste time on P3 edge cases. + +**Code examples:** 8 + +--- + +## Using Fragments Directly + +### As a Learning Resource + +Read fragments to learn patterns: + +```bash +# Read fixture architecture pattern +cat src/modules/bmm/testarch/knowledge/fixture-architecture.md + +# Read network-first pattern +cat src/modules/bmm/testarch/knowledge/network-first.md +``` + +### As Team Guidelines + +Use fragments as team documentation: + +```markdown +# Team Testing Guidelines + +## Fixture Architecture +See: src/modules/bmm/testarch/knowledge/fixture-architecture.md + +All fixtures must follow the pure function → fixture wrapper pattern. + +## Network Patterns +See: src/modules/bmm/testarch/knowledge/network-first.md + +All tests must use network-first patterns. No hard waits allowed. +``` + +### As Code Review Checklist + +Reference fragments in code review: + +```markdown +## PR Review Checklist + +- [ ] Tests follow test-quality.md standards (no hard waits, < 300 lines) +- [ ] Selectors follow selector-resilience.md (prefer getByRole) +- [ ] Network patterns follow network-first.md (wait for responses) +- [ ] Fixtures follow fixture-architecture.md (pure functions) +``` + +## Fragment Statistics + +**Total Fragments:** 33 +**Total Size:** ~600 KB (all fragments combined) +**Average Fragment Size:** ~18 KB +**Largest Fragment:** contract-testing.md (~28 KB) +**Smallest Fragment:** burn-in.md (~7 KB) + +**By Category:** +- Architecture & Fixtures: 4 fragments +- Data & Setup: 3 fragments +- Network & Reliability: 4 fragments +- Test Execution & CI: 3 fragments +- Quality & Standards: 5 fragments +- Risk & Gates: 3 fragments +- Selectors & Timing: 3 fragments +- Feature Flags & Patterns: 3 fragments +- Playwright-Utils Integration: 8 fragments + +**Note:** Statistics may drift with updates. All fragments are in the same `knowledge/` directory. + +## Contributing to Knowledge Base + +### Adding New Fragments + +1. Create fragment in `src/modules/bmm/testarch/knowledge/` +2. Follow existing format (Principle, Rationale, Pattern Examples) +3. Add to `tea-index.csv` with metadata +4. Update workflow instructions to load fragment +5. Test with TEA workflow + +### Updating Existing Fragments + +1. Edit fragment markdown file +2. Update `tea-index.csv` if metadata changes (line count, examples) +3. Test with affected workflows +4. Ensure no breaking changes to patterns + +### Fragment Quality Standards + +**Good fragment:** +- Principle stated clearly +- Rationale explains why +- Multiple pattern examples with code +- Good vs bad comparisons +- Self-contained (links to other fragments minimal) + +**Example structure:** +```markdown +# Fragment Name + +## Principle +[One sentence - what is this pattern?] + +## Rationale +[Why use this instead of alternatives?] + +## Pattern Examples + +### Example 1: Basic Usage +[Code example with explanation] + +### Example 2: Advanced Pattern +[Code example with explanation] + +## Anti-Patterns + +### Don't Do This +[Bad code example] +[Why it's bad] + +## Related Patterns +- [Other fragment](../other-fragment.md) +``` + +## Related + +- [TEA Overview](/docs/explanation/features/tea-overview.md) - How knowledge base fits in TEA +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Context engineering philosophy +- [TEA Command Reference](/docs/reference/tea/commands.md) - Workflows that use fragments + +--- + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect) diff --git a/docs/tutorials/getting-started/tea-lite-quickstart.md b/docs/tutorials/getting-started/tea-lite-quickstart.md new file mode 100644 index 00000000..c4818085 --- /dev/null +++ b/docs/tutorials/getting-started/tea-lite-quickstart.md @@ -0,0 +1,465 @@ +--- +title: "Getting Started with TEA (Test Architect) - TEA Lite" +description: Learn TEA fundamentals by generating and running tests for an existing demo app in 30 minutes +--- + +# Getting Started with TEA (Test Architect) - TEA Lite + +Welcome! **TEA Lite** is the simplest way to get started with TEA - just use `*automate` to generate tests for existing features. Perfect for beginners who want to learn TEA fundamentals quickly. + +## What You'll Build + +By the end of this 30-minute tutorial, you'll have: +- A working Playwright test framework +- Your first risk-based test plan +- Passing tests for an existing demo app feature + +## Prerequisites + +- Node.js installed (v18 or later) +- 30 minutes of focused time +- We'll use TodoMVC (https://todomvc.com/examples/react/) as our demo app + +## TEA Approaches Explained + +Before we start, understand the three ways to use TEA: + +- **TEA Lite** (this tutorial): Beginner using just `*automate` to test existing features +- **TEA Solo**: Using TEA standalone without full BMad Method integration +- **TEA Integrated**: Full BMad Method with all TEA workflows across phases + +This tutorial focuses on **TEA Lite** - the fastest way to see TEA in action. + +--- + +## Step 0: Setup (2 minutes) + +We'll test TodoMVC, a standard demo app used across testing documentation. + +**Demo App:** https://todomvc.com/examples/react/ + +No installation needed - TodoMVC runs in your browser. Open the link above and: +1. Add a few todos (type and press Enter) +2. Mark some as complete (click checkbox) +3. Try the "All", "Active", "Completed" filters + +You've just explored the features we'll test! + +--- + +## Step 1: Install BMad and Scaffold Framework (10 minutes) + +### Install BMad Method + +```bash +npx bmad-method@alpha install +``` + +When prompted: +- **Select modules:** Choose "BMM: BMad Method" (press Space, then Enter) +- **Project name:** Keep default or enter your project name +- **Experience level:** Choose "beginner" for this tutorial +- **Planning artifacts folder:** Keep default +- **Implementation artifacts folder:** Keep default +- **Project knowledge folder:** Keep default +- **Enable TEA Playwright MCP enhancements?** Choose "No" for now (we'll explore this later) +- **Using playwright-utils?** Choose "No" for now (we'll explore this later) + +BMad is now installed! You'll see a `_bmad/` folder in your project. + +### Load TEA Agent + +Start a new chat with your AI assistant (Claude, etc.) and type: + +``` +*tea +``` + +This loads the Test Architect agent. You'll see TEA's menu with available workflows. + +### Scaffold Test Framework + +In your chat, run: + +``` +*framework +``` + +TEA will ask you questions: + +**Q: What's your tech stack?** +A: "We're testing a React web application (TodoMVC)" + +**Q: Which test framework?** +A: "Playwright" + +**Q: Testing scope?** +A: "E2E testing for web application" + +**Q: CI/CD platform?** +A: "GitHub Actions" (or your preference) + +TEA will generate: +- `tests/` directory with Playwright config +- `playwright.config.ts` with base configuration +- Sample test structure +- `.env.example` for environment variables +- `.nvmrc` for Node version + +**Verify the setup:** + +```bash +npm install +npx playwright install +``` + +You now have a production-ready test framework! + +--- + +## Step 2: Your First Test Design (5 minutes) + +Test design is where TEA shines - risk-based planning before writing tests. + +### Run Test Design + +In your chat with TEA, run: + +``` +*test-design +``` + +**Q: System-level or epic-level?** +A: "Epic-level - I want to test TodoMVC's basic functionality" + +**Q: What feature are you testing?** +A: "TodoMVC's core CRUD operations - creating, completing, and deleting todos" + +**Q: Any specific risks or concerns?** +A: "We want to ensure the filter buttons (All, Active, Completed) work correctly" + +TEA will analyze and create `test-design-epic-1.md` with: + +1. **Risk Assessment** + - Probability × Impact scoring + - Risk categories (TECH, SEC, PERF, DATA, BUS, OPS) + - High-risk areas identified + +2. **Test Priorities** + - P0: Critical path (creating and displaying todos) + - P1: High value (completing todos, filters) + - P2: Medium value (deleting todos) + - P3: Low value (edge cases) + +3. **Coverage Strategy** + - E2E tests for user workflows + - Which scenarios need testing + - Suggested test structure + +**Review the test design file** - notice how TEA provides a systematic approach to what needs testing and why. + +--- + +## Step 3: Generate Tests for Existing Features (5 minutes) + +Now the magic happens - TEA generates tests based on your test design. + +### Run Automate + +In your chat with TEA, run: + +``` +*automate +``` + +**Q: What are you testing?** +A: "TodoMVC React app at https://todomvc.com/examples/react/ - focus on the test design we just created" + +**Q: Reference existing docs?** +A: "Yes, use test-design-epic-1.md" + +**Q: Any specific test scenarios?** +A: "Cover the P0 and P1 scenarios from the test design" + +TEA will generate: + +**`tests/e2e/todomvc.spec.ts`** with tests like: +```typescript +import { test, expect } from '@playwright/test'; + +test.describe('TodoMVC - Core Functionality', () => { + test.beforeEach(async ({ page }) => { + await page.goto('https://todomvc.com/examples/react/'); + }); + + test('should create a new todo', async ({ page }) => { + // TodoMVC uses a simple input without placeholder or test IDs + const todoInput = page.locator('.new-todo'); + await todoInput.fill('Buy groceries'); + await todoInput.press('Enter'); + + // Verify todo appears in list + await expect(page.locator('.todo-list li')).toContainText('Buy groceries'); + }); + + test('should mark todo as complete', async ({ page }) => { + // Create a todo + const todoInput = page.locator('.new-todo'); + await todoInput.fill('Complete tutorial'); + await todoInput.press('Enter'); + + // Mark as complete using the toggle checkbox + await page.locator('.todo-list li .toggle').click(); + + // Verify completed state + await expect(page.locator('.todo-list li')).toHaveClass(/completed/); + }); + + test('should filter todos by status', async ({ page }) => { + // Create multiple todos + const todoInput = page.locator('.new-todo'); + await todoInput.fill('Buy groceries'); + await todoInput.press('Enter'); + await todoInput.fill('Write tests'); + await todoInput.press('Enter'); + + // Complete the first todo ("Buy groceries") + await page.locator('.todo-list li .toggle').first().click(); + + // Test Active filter (shows only incomplete todos) + await page.locator('.filters a[href="#/active"]').click(); + await expect(page.locator('.todo-list li')).toHaveCount(1); + await expect(page.locator('.todo-list li')).toContainText('Write tests'); + + // Test Completed filter (shows only completed todos) + await page.locator('.filters a[href="#/completed"]').click(); + await expect(page.locator('.todo-list li')).toHaveCount(1); + await expect(page.locator('.todo-list li')).toContainText('Buy groceries'); + }); +}); +``` + +TEA also creates: +- **`tests/README.md`** - How to run tests, project conventions +- **Definition of Done summary** - What makes a test "good" + +### With Playwright Utils (Optional Enhancement) + +If you have `tea_use_playwright_utils: true` in your config, TEA generates tests using production-ready utilities: + +**Vanilla Playwright:** +```typescript +test('should mark todo as complete', async ({ page, request }) => { + // Manual API call + const response = await request.post('/api/todos', { + data: { title: 'Complete tutorial' } + }); + const todo = await response.json(); + + await page.goto('/'); + await page.locator(`.todo-list li:has-text("${todo.title}") .toggle`).click(); + await expect(page.locator('.todo-list li')).toHaveClass(/completed/); +}); +``` + +**With Playwright Utils:** +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { expect } from '@playwright/test'; + +test('should mark todo as complete', async ({ page, apiRequest }) => { + // Typed API call with cleaner syntax + const { status, body: todo } = await apiRequest({ + method: 'POST', + path: '/api/todos', + body: { title: 'Complete tutorial' } // 'body' not 'data' + }); + + expect(status).toBe(201); + await page.goto('/'); + await page.locator(`.todo-list li:has-text("${todo.title}") .toggle`).click(); + await expect(page.locator('.todo-list li')).toHaveClass(/completed/); +}); +``` + +**Benefits:** +- Type-safe API responses (`{ status, body }`) +- Automatic retry for 5xx errors +- Built-in schema validation +- Cleaner, more maintainable code + +See [Integrate Playwright Utils](/docs/how-to/customization/integrate-playwright-utils.md) to enable this. + +--- + +## Step 4: Run and Validate (5 minutes) + +Time to see your tests in action! + +### Run the Tests + +```bash +npx playwright test +``` + +You should see: +``` +Running 3 tests using 1 worker + + ✓ tests/e2e/todomvc.spec.ts:7:3 › should create a new todo (2s) + ✓ tests/e2e/todomvc.spec.ts:15:3 › should mark todo as complete (2s) + ✓ tests/e2e/todomvc.spec.ts:30:3 › should filter todos by status (3s) + + 3 passed (7s) +``` + +All green! Your tests are passing against the existing TodoMVC app. + +### View Test Report + +```bash +npx playwright show-report +``` + +Opens a beautiful HTML report showing: +- Test execution timeline +- Screenshots (if any failures) +- Trace viewer for debugging + +### What Just Happened? + +You used **TEA Lite** to: +1. Scaffold a production-ready test framework (`*framework`) +2. Create a risk-based test plan (`*test-design`) +3. Generate comprehensive tests (`*automate`) +4. Run tests against an existing application + +All in 30 minutes! + +--- + +## What You Learned + +Congratulations! You've completed the TEA Lite tutorial. You learned: + +### TEA Workflows +- `*framework` - Scaffold test infrastructure +- `*test-design` - Risk-based test planning +- `*automate` - Generate tests for existing features + +### TEA Principles +- **Risk-based testing** - Depth scales with impact (P0 vs P3) +- **Test design first** - Plan before generating +- **Network-first patterns** - Tests wait for actual responses (no hard waits) +- **Production-ready from day one** - Not toy examples + +### Key Takeaway + +TEA Lite (just `*automate`) is perfect for: +- Beginners learning TEA fundamentals +- Testing existing applications +- Quick test coverage expansion +- Teams wanting fast results + +--- + +## Understanding ATDD vs Automate + +This tutorial used `*automate` to generate tests for **existing features** (tests pass immediately). + +**When to use `*automate`:** +- Feature already exists +- Want to add test coverage +- Tests should pass on first run + +**When to use `*atdd`:** +- Feature doesn't exist yet (TDD workflow) +- Want failing tests BEFORE implementation +- Following red → green → refactor cycle + +See [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) for the TDD approach. + +--- + +## Next Steps + +### Level Up Your TEA Skills + +**How-To Guides** (task-oriented): +- [How to Run Test Design](/docs/how-to/workflows/run-test-design.md) - Deep dive into risk assessment +- [How to Run ATDD](/docs/how-to/workflows/run-atdd.md) - Generate failing tests first (TDD) +- [How to Set Up CI Pipeline](/docs/how-to/workflows/setup-ci.md) - Automate test execution +- [How to Review Test Quality](/docs/how-to/workflows/run-test-review.md) - Audit test quality + +**Explanation** (understanding-oriented): +- [TEA Overview](/docs/explanation/features/tea-overview.md) - Complete TEA capabilities +- [Testing as Engineering](/docs/explanation/philosophy/testing-as-engineering.md) - Design philosophy +- [Risk-Based Testing](/docs/explanation/tea/risk-based-testing.md) - How risk scoring works + +**Reference** (quick lookup): +- [TEA Command Reference](/docs/reference/tea/commands.md) - All 8 TEA workflows +- [TEA Configuration](/docs/reference/tea/configuration.md) - Config options +- [Glossary](/docs/reference/glossary/index.md) - TEA terminology + +### Try TEA Solo + +Ready for standalone usage without full BMad Method? Use TEA Solo: +- Run any TEA workflow independently +- Bring your own requirements +- Use on non-BMad projects + +See [TEA Overview](/docs/explanation/features/tea-overview.md) for engagement models. + +### Go Full TEA Integrated + +Want the complete quality operating model? Try TEA Integrated with BMad Method: +- Phase 2: Planning with NFR assessment +- Phase 3: Architecture testability review +- Phase 4: Per-epic test design → ATDD → automate +- Release Gate: Coverage traceability and gate decisions + +See [BMad Method Documentation](/) for the full workflow. + +--- + +## Troubleshooting + +### Tests Failing? + +**Problem:** Tests can't find elements +**Solution:** TodoMVC doesn't use test IDs or accessible roles consistently. The selectors in this tutorial use CSS classes that match TodoMVC's actual structure: +```typescript +// TodoMVC uses these CSS classes: +page.locator('.new-todo') // Input field +page.locator('.todo-list li') // Todo items +page.locator('.toggle') // Checkbox + +// If testing your own app, prefer accessible selectors: +page.getByRole('textbox') +page.getByRole('listitem') +page.getByRole('checkbox') +``` + +**Note:** In production code, use accessible selectors (`getByRole`, `getByLabel`, `getByText`) for better resilience. TodoMVC is used here for learning, not as a selector best practice example. + +**Problem:** Network timeout +**Solution:** Increase timeout in `playwright.config.ts`: +```typescript +use: { + timeout: 30000, // 30 seconds +} +``` + +### Need Help? + +- **Documentation:** https://docs.bmad-method.org +- **GitHub Issues:** https://github.com/bmad-code-org/bmad-method/issues +- **Discord:** Join the BMAD community + +--- + +## Feedback + +Found this tutorial helpful? Have suggestions? Open an issue on GitHub! + +Generated with [BMad Method](https://bmad-method.org) - TEA (Test Architect)