diff --git a/TESTING-GUIDE.md b/TESTING-GUIDE.md new file mode 100644 index 00000000..2831ae35 --- /dev/null +++ b/TESTING-GUIDE.md @@ -0,0 +1,293 @@ +# Testing Guide: Gap Analysis Features + +## Setup Complete ✅ + +Your platform project (`~/git/ai/platform`) is configured to use the dev version via symlink: + +``` +~/git/ai/platform/_bmad/bmm → ~/git/ai/BMAD-METHOD/src/modules/bmm +``` + +All changes from `feature/gap-analysis-dev-time` branch are **live and testable**. + +--- + +## Test Scenarios + +### Test 1: Basic Gap Analysis (Recommended First) + +**Goal:** Verify gap analysis runs and proposes task updates + +```bash +cd ~/git/ai/platform + +# Load PM or Dev agent +# Run: +/dev-story + +# Expected: +# → Loads next ready-for-dev story +# → Step 1.5 runs automatically +# → Shows "📊 Gap Analysis Complete" +# → Presents task updates +# → Asks: "Approve these task updates? [Y/A/n/e/s/r]" + +# Test each option: +# [Y] - Approve and proceed +# [A] - Auto-accept mode +# [n] - Keep draft tasks +# [e] - Edit manually +# [r] - Review details +# [s] - Skip story +``` + +**Success Criteria:** +- ✅ Gap analysis runs automatically +- ✅ Scans codebase with Glob/Grep/Read +- ✅ Proposes accurate task updates +- ✅ Updates story file when approved +- ✅ Adds "Gap Analysis" section to story + +--- + +### Test 2: Batch Planning Staleness Detection + +**Goal:** Verify gap analysis catches code from earlier stories + +```bash +# Create 3 stories in batch +/create-story # Story 1.1 +/create-story # Story 1.2 +/create-story # Story 1.3 + +# All will have "DRAFT TASKS" notation + +# Develop Story 1.1 +/dev-story +# Gap analysis: likely finds nothing (first story) +# Approve and implement + +# Develop Story 1.2 +/dev-story +# Gap analysis: should detect Story 1.1's code! +# Proposes task refinements: "Extend X" instead of "Create X" + +# Develop Story 1.3 +/dev-story +# Gap analysis: should detect Stories 1.1-1.2's code! +# Proposes even more refinements +``` + +**Success Criteria:** +- ✅ Story 1.1: Gap analysis finds minimal existing code +- ✅ Story 1.2: Gap analysis detects Story 1.1's implementations +- ✅ Story 1.3: Gap analysis detects Stories 1.1-1.2's work +- ✅ Tasks get refined based on cumulative codebase state + +--- + +### Test 3: Standalone Gap Analysis (Audit Tool) + +**Goal:** Audit completed stories without starting development + +```bash +# Load any agent +/gap-analysis + +# When prompted, try: + +# Option 1: Audit by status +Enter: "done" +# Should list all done stories, ask which to validate + +# Option 2: Audit specific story +Enter: "1-2-auth" +# Should validate that specific story + +# Option 3: Audit by file path +Enter: "docs/sprint-artifacts/1-2-auth.md" +# Should validate that story file + +# Expected output: +# → Scans codebase +# → Shows "What Exists" vs "What's Missing" +# → Detects false positives (marked done but code missing) +# → Presents options: [U]pdate, [A]udit report, [N]o changes, [R]eview, [Q]uit +``` + +**Success Criteria:** +- ✅ Can audit stories by status +- ✅ Can audit specific story +- ✅ Detects false positives +- ✅ Can update story file with findings +- ✅ Can generate audit reports + +--- + +### Test 4: Super-Dev-Story (Enhanced Quality) + +**Goal:** Verify comprehensive quality workflow + +```bash +# Load any agent +/super-dev-story + +# Expected flow: +# 1. Executes dev-story Steps 1-8 (including pre-dev gap analysis) +# 2. After all tasks complete... +# 3. Step 9.5: Post-dev gap analysis runs +# - Re-scans codebase +# - Verifies all checked tasks actually implemented +# - If gaps: adds tasks, loops back to step 5 +# 4. Step 9.6: Auto code review runs +# - Reviews all changed files +# - Finds security/quality issues +# - If critical/high: adds tasks, loops back to step 5 +# - If medium/low: asks to fix or document +# 5. Story marked "review" only after passing all gates +``` + +**Success Criteria:** +- ✅ Executes all dev-story steps +- ✅ Runs post-dev gap analysis +- ✅ Runs code review automatically +- ✅ Loops back if issues found +- ✅ Only marks done after validation passes + +--- + +### Test 5: Auto-Accept Mode + +**Goal:** Verify automation-friendly flow + +```bash +/dev-story + +# When gap analysis prompts: +Select: [A] Auto-accept + +# Continue developing more stories: +/dev-story # Story 2 +/dev-story # Story 3 + +# Expected: +# → Gap analysis runs for each +# → No prompts after first [A] +# → All refinements auto-applied +# → Still documented in Change Log +``` + +**Success Criteria:** +- ✅ First prompt allows [A] selection +- ✅ Future stories auto-apply without prompting +- ✅ All changes documented +- ✅ Can be used for CI/CD automation + +--- + +## Edge Cases to Test + +### Edge Case 1: Greenfield First Story + +```bash +# Brand new project, no code yet +/dev-story # Story 1.1 + +# Expected: +# → Gap analysis scans +# → Finds nothing (empty project) +# → Proposes no changes +# → Auto-proceeds to implementation +``` + +### Edge Case 2: Everything Already Exists + +```bash +# Story tasks say "Create X" +# But X already fully implemented + +# Expected: +# → Gap analysis detects X exists +# → Proposes removing "Create X" task +# → Story might be already complete! +``` + +### Edge Case 3: Partial Implementation + +```bash +# Story says "Create auth service" +# But partial auth service exists (50% complete) + +# Expected: +# → Gap analysis detects partial implementation +# → Proposes: "Complete auth service implementation" +# → Notes what exists vs what's missing +``` + +## Validation Checks + +After each test, verify: + +- [ ] Story file updated with "Gap Analysis" section +- [ ] Change Log includes gap analysis entry +- [ ] Tasks reflect codebase reality +- [ ] False positives caught and corrected +- [ ] Duplicate implementations prevented + +## Performance Benchmarks + +Track these metrics: + +| Workflow | Tokens | Time | Quality Score | +|----------|--------|------|---------------| +| dev-story (no gap) | Baseline | Baseline | ? | +| dev-story (with gap) | +5-10K | +10s | Higher | +| super-dev-story | +30-50K | +20-30% | Highest | + +## Rollback (If Issues Found) + +```bash +cd ~/git/ai/platform/_bmad +rm bmm +mv bmm.backup bmm + +# Report issues found +``` + +## Reporting Issues + +If you find problems: + +1. **Note the scenario** - What story, what workflow, what happened +2. **Check logs** - Any errors in Dev Agent Record +3. **Save examples** - Story files before/after gap analysis +4. **Report in Discord** - #general-dev channel +5. **Or open issue** - Use bug report template + +Include: +- Story file (before gap analysis) +- Gap analysis output +- Expected vs actual behavior +- BMAD version (`npx bmad-method@alpha --version`) + +## Success Indicators + +You'll know it's working when: + +- ✅ Story 1.2 detects Story 1.1's code and refines tasks +- ✅ Duplicate implementations prevented +- ✅ False positive completions caught +- ✅ Story accuracy improves over time +- ✅ Less rework needed during human review + +## Next Steps After Testing + +1. Gather feedback from real usage +2. Note any false positives/negatives in scanning +3. Identify improvements to gap analysis logic +4. Document any edge cases found +5. Ready to create PR if working well! + +--- + +**Testing is the critical step before contribution!** 🧪 diff --git a/src/modules/bmgd/agents/game-dev.agent.yaml b/src/modules/bmgd/agents/game-dev.agent.yaml index 43104050..b502f4e4 100644 --- a/src/modules/bmgd/agents/game-dev.agent.yaml +++ b/src/modules/bmgd/agents/game-dev.agent.yaml @@ -32,6 +32,18 @@ agent: workflow: "{project-root}/_bmad/bmgd/workflows/4-production/dev-story/workflow.yaml" description: "[DS] Execute Dev Story workflow, implementing tasks and tests" + - trigger: SDS or fuzzy match on super-dev-story + workflow: "{project-root}/_bmad/bmgd/workflows/4-production/super-dev-story/workflow.yaml" + description: "[SDS] Execute Super-Dev Story workflow (enhanced quality with post-dev validation + auto code review)" + + - trigger: GA or fuzzy match on gap-analysis + workflow: "{project-root}/_bmad/bmgd/workflows/4-production/gap-analysis/workflow.yaml" + description: "[GA] Run Gap Analysis - audit story tasks against codebase without starting development" + + - trigger: AEP or fuzzy match on autonomous-epic + workflow: "{project-root}/_bmad/bmgd/workflows/4-production/autonomous-epic/workflow.yaml" + description: "[AEP] Autonomous Epic Processing - create and develop all stories in an epic automatically" + - trigger: CR or fuzzy match on code-review workflow: "{project-root}/_bmad/bmgd/workflows/4-production/code-review/workflow.yaml" description: "[CR] Perform a thorough clean context QA code review on a story flagged Ready for Review" diff --git a/src/modules/bmgd/agents/game-scrum-master.agent.yaml b/src/modules/bmgd/agents/game-scrum-master.agent.yaml index 336515ed..b25856c5 100644 --- a/src/modules/bmgd/agents/game-scrum-master.agent.yaml +++ b/src/modules/bmgd/agents/game-scrum-master.agent.yaml @@ -40,6 +40,14 @@ agent: workflow: "{project-root}/_bmad/bmgd/workflows/4-production/create-story/workflow.yaml" description: "[CS] Create Story with direct ready-for-dev marking (Required to prepare stories for development)" + - trigger: GA or fuzzy match on gap-analysis + workflow: "{project-root}/_bmad/bmgd/workflows/4-production/gap-analysis/workflow.yaml" + description: "[GA] Run Gap Analysis - audit story tasks against codebase (useful for sprint health checks)" + + - trigger: AEP or fuzzy match on autonomous-epic + workflow: "{project-root}/_bmad/bmgd/workflows/4-production/autonomous-epic/workflow.yaml" + description: "[AEP] Autonomous Epic Processing - create and develop all stories in an epic automatically" + - trigger: VS or fuzzy match on validate-story validate-workflow: "{project-root}/_bmad/bmgd/workflows/4-production/create-story/workflow.yaml" description: "[VS] Validate Story Draft with Independent Review (Highly Recommended)" diff --git a/src/modules/bmm/agents/dev.agent.yaml b/src/modules/bmm/agents/dev.agent.yaml index fcfdda8e..c8da728d 100644 --- a/src/modules/bmm/agents/dev.agent.yaml +++ b/src/modules/bmm/agents/dev.agent.yaml @@ -39,6 +39,18 @@ agent: workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/dev-story/workflow.yaml" description: "[DS] Execute Dev Story workflow (full BMM path with sprint-status)" + - trigger: SDS or fuzzy match on super-dev-story + workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/super-dev-story/workflow.yaml" + description: "[SDS] Execute Super-Dev Story workflow (enhanced quality with post-dev validation + auto code review)" + + - trigger: GA or fuzzy match on gap-analysis + workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/gap-analysis/workflow.yaml" + description: "[GA] Run Gap Analysis - audit story tasks against codebase without starting development" + + - trigger: AE or fuzzy match on autonomous-epic + workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/autonomous-epic/workflow.yaml" + description: "[AE] Autonomous Epic Processing - create and develop all stories in an epic automatically" + - trigger: CR or fuzzy match on code-review workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/code-review/workflow.yaml" description: "[CR] Perform a thorough clean context code review (Highly Recommended, use fresh context and different LLM)" diff --git a/src/modules/bmm/agents/sm.agent.yaml b/src/modules/bmm/agents/sm.agent.yaml index 8665cd99..f671e8f6 100644 --- a/src/modules/bmm/agents/sm.agent.yaml +++ b/src/modules/bmm/agents/sm.agent.yaml @@ -36,6 +36,14 @@ agent: workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/create-story/workflow.yaml" description: "[CS] Create Story (Required to prepare stories for development)" + - trigger: GA or fuzzy match on gap-analysis + workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/gap-analysis/workflow.yaml" + description: "[GA] Run Gap Analysis - audit story tasks against codebase (useful for sprint health checks)" + + - trigger: AEP or fuzzy match on autonomous-epic + workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/autonomous-epic/workflow.yaml" + description: "[AEP] Autonomous Epic Processing - create and develop all stories in an epic automatically" + - trigger: ER or fuzzy match on epic-retrospective workflow: "{project-root}/_bmad/bmm/workflows/4-implementation/retrospective/workflow.yaml" data: "{project-root}/_bmad/_config/agent-manifest.csv"