docs: add UAT workflow implementation gaps analysis
Documents missing components needed for production-ready UAT validation: - Shell orchestration script (uat-validate.sh) - Metrics instrumentation in epic-execute.sh - UAT gate integration in epic-chain.sh - Step files following BMAD pattern - Fix context handoff integration Includes implementation priorities, code snippets, and testing plan. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Caleb
parent
57fda0915b
commit
6b1ca9333f
|
|
@ -0,0 +1,527 @@
|
|||
# UAT Workflow Implementation Gaps & Required Fixes
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The UAT validation workflow and epic chain report generator have been designed and partially implemented, but several critical components are missing to make them production-ready. This document outlines the gaps, required fixes, and context needed to complete the implementation.
|
||||
|
||||
**Current State:** Workflow definitions, templates, and agent triggers exist
|
||||
**Missing:** Shell orchestration, metrics collection, step files, and integration points
|
||||
|
||||
---
|
||||
|
||||
## Architecture Context
|
||||
|
||||
### How BMAD Workflows Achieve Context Isolation
|
||||
|
||||
Native BMAD workflows use a **shell script orchestration pattern** to isolate Claude contexts:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ BMAD CONTEXT ISOLATION PATTERN │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ Shell Script (epic-execute.sh / epic-chain.sh) │
|
||||
│ │ │
|
||||
│ ├──► claude -p "Phase 1 prompt..." ──► Fresh Context A │
|
||||
│ │ │ │
|
||||
│ │ └──► Git staging / Story file (state transfer) │
|
||||
│ │ │
|
||||
│ ├──► claude -p "Phase 2 prompt..." ──► Fresh Context B │
|
||||
│ │ │ │
|
||||
│ │ └──► Git staging / Story file (state transfer) │
|
||||
│ │ │
|
||||
│ └──► claude -p "Phase 3 prompt..." ──► Fresh Context C │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Key Files:**
|
||||
- `scripts/epic-execute.sh` - 600+ lines, handles dev→review→commit→UAT phases
|
||||
- `scripts/epic-chain.sh` - Handles multi-epic execution with handoffs
|
||||
|
||||
**State Transfer Mechanisms:**
|
||||
1. **Git staging** - Code changes pass between contexts via `git add`
|
||||
2. **Story files** - Dev Agent Record and Code Review Record sections
|
||||
3. **Handoff documents** - Cross-epic context transfer
|
||||
4. **Completion signals** - `IMPLEMENTATION COMPLETE: {id}`, `REVIEW PASSED: {id}`
|
||||
|
||||
---
|
||||
|
||||
## Gap 1: Missing Shell Orchestration Script
|
||||
|
||||
### Problem
|
||||
|
||||
The UAT validation self-healing loop requires context isolation:
|
||||
|
||||
```
|
||||
UAT Validate (Quinn) → Fix Context → Quick Dev (Barry) → Re-validate (Quinn)
|
||||
```
|
||||
|
||||
Without a shell script, these phases would run in the same Claude context, defeating the purpose of fresh-eyes validation after fixes.
|
||||
|
||||
### Required: `scripts/uat-validate.sh`
|
||||
|
||||
**Location:** `scripts/uat-validate.sh`
|
||||
|
||||
**Responsibilities:**
|
||||
1. Load UAT document for specified epic
|
||||
2. Execute automatable scenarios via shell commands
|
||||
3. Collect pass/fail results with evidence
|
||||
4. On failure: Generate fix context document
|
||||
5. Spawn fresh Claude session for quick-dev fixes
|
||||
6. Re-validate in fresh context
|
||||
7. Loop until pass or max retries
|
||||
8. Output final gate result
|
||||
|
||||
**Interface:**
|
||||
```bash
|
||||
# Usage
|
||||
./scripts/uat-validate.sh <epic_id> [options]
|
||||
|
||||
# Options
|
||||
--gate-mode=quick|full|skip # Which scenarios to run
|
||||
--max-retries=2 # Fix attempts before halt
|
||||
--skip-manual # Skip manual-only scenarios
|
||||
--verbose # Detailed output
|
||||
|
||||
# Output signals (for epic-chain.sh to parse)
|
||||
UAT_GATE_RESULT: PASS|FAIL
|
||||
UAT_FIX_ATTEMPTS: N
|
||||
UAT_SCENARIOS_PASSED: X/Y
|
||||
```
|
||||
|
||||
**Context Flow:**
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ uat-validate.sh Flow │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. Load UAT doc: docs/uat/epic-{id}-uat.md │
|
||||
│ │
|
||||
│ 2. Classify scenarios (automatable / semi-auto / manual) │
|
||||
│ │
|
||||
│ 3. Execute automatable scenarios via shell: │
|
||||
│ for scenario in automatable: │
|
||||
│ result = execute_command(scenario.command) │
|
||||
│ record_result(scenario.id, result) │
|
||||
│ │
|
||||
│ 4. Evaluate gate: │
|
||||
│ if all_passed: │
|
||||
│ echo "UAT_GATE_RESULT: PASS" │
|
||||
│ exit 0 │
|
||||
│ │
|
||||
│ 5. On failure (attempt < max_retries): │
|
||||
│ a. Generate fix context document │
|
||||
│ b. Spawn fresh Claude for quick-dev: │
|
||||
│ claude -p "Load fix context, implement fixes..." │
|
||||
│ c. Increment attempt counter │
|
||||
│ d. Go to step 3 (re-validate) │
|
||||
│ │
|
||||
│ 6. On max retries exceeded: │
|
||||
│ echo "UAT_GATE_RESULT: FAIL" │
|
||||
│ echo "UAT_MAX_RETRIES: true" │
|
||||
│ exit 2 │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Reference Implementation:** See `scripts/epic-execute.sh` lines 289-438 for how dev and review phases spawn isolated Claude sessions.
|
||||
|
||||
---
|
||||
|
||||
## Gap 2: Metrics Collection Not Instrumented
|
||||
|
||||
### Problem
|
||||
|
||||
The report generator expects metrics at `{metrics_folder}/epic-{id}-metrics.yaml`, but `epic-execute.sh` doesn't write these files.
|
||||
|
||||
### Required: Metrics Instrumentation in `epic-execute.sh`
|
||||
|
||||
**Location:** Modify `scripts/epic-execute.sh`
|
||||
|
||||
**Add at epic start (~line 520):**
|
||||
```bash
|
||||
# Initialize metrics
|
||||
EPIC_START_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
METRICS_FILE="$SPRINT_ARTIFACTS/metrics/epic-${EPIC_ID}-metrics.yaml"
|
||||
mkdir -p "$SPRINT_ARTIFACTS/metrics"
|
||||
|
||||
# Initialize metrics file
|
||||
cat > "$METRICS_FILE" << EOF
|
||||
epic_id: "$EPIC_ID"
|
||||
epic_name: "$EPIC_NAME"
|
||||
execution:
|
||||
start_time: "$EPIC_START_TIME"
|
||||
end_time: ""
|
||||
duration_seconds: 0
|
||||
stories:
|
||||
total: ${#STORIES[@]}
|
||||
completed: 0
|
||||
failed: 0
|
||||
skipped: 0
|
||||
validation:
|
||||
gate_executed: false
|
||||
gate_status: "PENDING"
|
||||
fix_attempts: 0
|
||||
issues: []
|
||||
EOF
|
||||
```
|
||||
|
||||
**Add after each story (~line 595):**
|
||||
```bash
|
||||
# Update metrics after story completion
|
||||
update_story_metrics() {
|
||||
local status="$1" # completed | failed | skipped
|
||||
local story_id="$2"
|
||||
|
||||
# Increment appropriate counter
|
||||
case "$status" in
|
||||
completed) yq -i '.stories.completed += 1' "$METRICS_FILE" ;;
|
||||
failed) yq -i '.stories.failed += 1' "$METRICS_FILE" ;;
|
||||
skipped) yq -i '.stories.skipped += 1' "$METRICS_FILE" ;;
|
||||
esac
|
||||
}
|
||||
```
|
||||
|
||||
**Add at epic end (~line 620):**
|
||||
```bash
|
||||
# Finalize metrics
|
||||
EPIC_END_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
DURATION=$(($(date +%s) - $(date -d "$EPIC_START_TIME" +%s)))
|
||||
|
||||
yq -i ".execution.end_time = \"$EPIC_END_TIME\"" "$METRICS_FILE"
|
||||
yq -i ".execution.duration_seconds = $DURATION" "$METRICS_FILE"
|
||||
```
|
||||
|
||||
**Template Reference:** `src/modules/bmm/workflows/4-implementation/epic-chain/templates/epic-metrics-template.yaml`
|
||||
|
||||
---
|
||||
|
||||
## Gap 3: UAT Gate Not Integrated Into epic-chain.sh
|
||||
|
||||
### Problem
|
||||
|
||||
`epic-chain.sh` executes epics but doesn't call UAT validation between them.
|
||||
|
||||
### Required: Add UAT Gate Phase to `epic-chain.sh`
|
||||
|
||||
**Location:** Modify `scripts/epic-chain.sh`
|
||||
|
||||
**Add configuration variables (~line 50):**
|
||||
```bash
|
||||
# UAT Gate Configuration
|
||||
UAT_GATE_ENABLED="${UAT_GATE_ENABLED:-true}"
|
||||
UAT_GATE_MODE="${UAT_GATE_MODE:-quick}"
|
||||
UAT_MAX_RETRIES="${UAT_MAX_RETRIES:-2}"
|
||||
UAT_BLOCKING="${UAT_BLOCKING:-false}"
|
||||
```
|
||||
|
||||
**Add after epic completion (~line 480, after `run_epic_execute`):**
|
||||
```bash
|
||||
# Run UAT validation if enabled
|
||||
if [ "$UAT_GATE_ENABLED" = true ]; then
|
||||
log "Running UAT validation for Epic $epic_id..."
|
||||
|
||||
uat_result=$(./scripts/uat-validate.sh "$epic_id" \
|
||||
--gate-mode="$UAT_GATE_MODE" \
|
||||
--max-retries="$UAT_MAX_RETRIES" 2>&1)
|
||||
|
||||
# Parse result
|
||||
if echo "$uat_result" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||
log "✓ UAT validation passed for Epic $epic_id"
|
||||
# Update metrics
|
||||
yq -i '.validation.gate_executed = true' "$METRICS_FILE"
|
||||
yq -i '.validation.gate_status = "PASS"' "$METRICS_FILE"
|
||||
else
|
||||
log_error "✗ UAT validation failed for Epic $epic_id"
|
||||
yq -i '.validation.gate_executed = true' "$METRICS_FILE"
|
||||
yq -i '.validation.gate_status = "FAIL"' "$METRICS_FILE"
|
||||
|
||||
if [ "$UAT_BLOCKING" = true ]; then
|
||||
log_error "UAT blocking enabled - halting chain"
|
||||
exit 1
|
||||
else
|
||||
log_warn "UAT blocking disabled - continuing to next epic"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
**Add CLI flag parsing (~line 120):**
|
||||
```bash
|
||||
--uat-gate=*)
|
||||
UAT_GATE_MODE="${1#*=}"
|
||||
;;
|
||||
--uat-blocking)
|
||||
UAT_BLOCKING=true
|
||||
;;
|
||||
--no-uat)
|
||||
UAT_GATE_ENABLED=false
|
||||
;;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gap 4: Step Files Not Following BMAD Pattern
|
||||
|
||||
### Problem
|
||||
|
||||
The UAT validation workflow has `instructions.md` but not the step-file architecture used by epic-execute:
|
||||
- `step-01-init.md`
|
||||
- `step-02-dev-story.md`
|
||||
- `step-03-code-review.md`
|
||||
- etc.
|
||||
|
||||
### Required: Create Step Files for UAT Validation
|
||||
|
||||
**Location:** `src/modules/bmm/workflows/5-validation/uat-validate/steps/`
|
||||
|
||||
**Files to create:**
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `step-01-load-uat.md` | Load UAT document, parse scenarios |
|
||||
| `step-02-classify-scenarios.md` | Categorize as automatable/semi-auto/manual |
|
||||
| `step-03-execute-scenarios.md` | Run automatable scenarios via shell |
|
||||
| `step-04-evaluate-gate.md` | Determine pass/fail, generate fix context if needed |
|
||||
| `step-05-report-results.md` | Update metrics, output signals |
|
||||
|
||||
**Step File Template (from epic-execute pattern):**
|
||||
```markdown
|
||||
# Step N: {Title}
|
||||
|
||||
## Purpose
|
||||
{What this step accomplishes}
|
||||
|
||||
## Inputs
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| ... | ... | Yes/No |
|
||||
|
||||
## Process
|
||||
|
||||
### N.1 {Sub-step}
|
||||
{Detailed instructions}
|
||||
|
||||
### N.2 {Sub-step}
|
||||
{Detailed instructions}
|
||||
|
||||
## Outputs
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| ... | ... | ... |
|
||||
|
||||
## Completion Signal
|
||||
```
|
||||
{SIGNAL_NAME}: {value}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| ... | ... |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gap 5: Fix Context Not Integrated With Handoff Pattern
|
||||
|
||||
### Problem
|
||||
|
||||
The fix context template exists but isn't written to the handoff directory or loaded by quick-dev.
|
||||
|
||||
### Required: Integrate Fix Context With Handoff Flow
|
||||
|
||||
**Fix context location should follow handoff pattern:**
|
||||
```
|
||||
docs/sprint-artifacts/
|
||||
├── handoffs/
|
||||
│ ├── epic-1-to-2-handoff.md
|
||||
│ └── epic-2-to-3-handoff.md
|
||||
├── uat-fixes/ # NEW
|
||||
│ ├── epic-1-fix-context-1.md
|
||||
│ └── epic-2-fix-context-1.md
|
||||
└── metrics/
|
||||
├── epic-1-metrics.yaml
|
||||
└── epic-2-metrics.yaml
|
||||
```
|
||||
|
||||
**In `uat-validate.sh`, write fix context:**
|
||||
```bash
|
||||
generate_fix_context() {
|
||||
local epic_id="$1"
|
||||
local attempt="$2"
|
||||
local failed_scenarios="$3"
|
||||
|
||||
local fix_dir="$SPRINT_ARTIFACTS/uat-fixes"
|
||||
mkdir -p "$fix_dir"
|
||||
|
||||
local fix_file="$fix_dir/epic-${epic_id}-fix-context-${attempt}.md"
|
||||
|
||||
# Render template with variables
|
||||
sed -e "s/{epic_id}/$epic_id/g" \
|
||||
-e "s/{attempt}/$attempt/g" \
|
||||
-e "s/{timestamp}/$(date -u +"%Y-%m-%dT%H:%M:%SZ")/g" \
|
||||
"$WORKFLOW_PATH/uat-fix-context-template.md" > "$fix_file"
|
||||
|
||||
# Append failed scenarios
|
||||
echo "" >> "$fix_file"
|
||||
echo "## Failed Scenarios" >> "$fix_file"
|
||||
echo "$failed_scenarios" >> "$fix_file"
|
||||
|
||||
echo "$fix_file"
|
||||
}
|
||||
```
|
||||
|
||||
**In quick-dev fix session, load context:**
|
||||
```bash
|
||||
run_quick_dev_fix() {
|
||||
local fix_context_file="$1"
|
||||
|
||||
local fix_prompt="You are Barry, the Quick Flow Solo Dev.
|
||||
|
||||
Load and process this fix context document:
|
||||
$fix_context_file
|
||||
|
||||
Your task:
|
||||
1. Read the failed scenarios and error details
|
||||
2. Analyze root cause for each failure
|
||||
3. Implement targeted fixes
|
||||
4. Run the failing commands to verify fixes
|
||||
5. Commit with message: fix(epic-{id}): UAT fix #{attempt}
|
||||
|
||||
Constraints:
|
||||
- Only fix the identified failures
|
||||
- Do not refactor unrelated code
|
||||
- Run tests after fixes
|
||||
|
||||
When done, output:
|
||||
FIX_COMPLETE: {number_fixed}/{total_failures}
|
||||
"
|
||||
|
||||
# Fresh Claude context for fixes
|
||||
claude --dangerously-skip-permissions -p "$fix_prompt"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Gap 6: Report Generator Missing Chain Integration
|
||||
|
||||
### Problem
|
||||
|
||||
Step 10 (generate report) exists but isn't called automatically by `epic-chain.sh`.
|
||||
|
||||
### Required: Add Report Generation to `epic-chain.sh`
|
||||
|
||||
**Add after all epics complete (~line 550):**
|
||||
```bash
|
||||
# Generate chain execution report
|
||||
if [ "$GENERATE_REPORT" = true ]; then
|
||||
log "Generating chain execution report..."
|
||||
|
||||
# Invoke report generation via Claude
|
||||
report_prompt="You are Bob, the Scrum Master.
|
||||
|
||||
Execute the chain report generation workflow:
|
||||
- Load: $INSTALLED_PATH/steps/step-10-generate-report.md
|
||||
- Metrics folder: $SPRINT_ARTIFACTS/metrics
|
||||
- Chain plan: $CHAIN_PLAN_FILE
|
||||
- Output to: $CHAIN_REPORT_FILE
|
||||
|
||||
Generate the complete execution report."
|
||||
|
||||
claude --dangerously-skip-permissions -p "$report_prompt"
|
||||
|
||||
if [ -f "$CHAIN_REPORT_FILE" ]; then
|
||||
log "✓ Report generated: $CHAIN_REPORT_FILE"
|
||||
else
|
||||
log_error "Report generation failed"
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
| Priority | Gap | Effort | Impact |
|
||||
|----------|-----|--------|--------|
|
||||
| **P0** | Create `uat-validate.sh` | High | Critical - self-healing loop won't work without it |
|
||||
| **P0** | Add metrics instrumentation to `epic-execute.sh` | Medium | Critical - report generator needs data |
|
||||
| **P1** | Integrate UAT gate into `epic-chain.sh` | Medium | High - enables automated validation |
|
||||
| **P1** | Integrate report generation into `epic-chain.sh` | Low | High - automates report creation |
|
||||
| **P2** | Create step files for UAT validation | Medium | Medium - improves maintainability |
|
||||
| **P2** | Integrate fix context with handoff pattern | Low | Medium - cleaner file organization |
|
||||
|
||||
---
|
||||
|
||||
## File Reference
|
||||
|
||||
### Existing Files (Created)
|
||||
|
||||
| File | Status | Purpose |
|
||||
|------|--------|---------|
|
||||
| `src/modules/bmm/agents/uat-validator.agent.yaml` | ✅ Complete | Quinn agent definition |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/workflow.yaml` | ✅ Complete | Workflow configuration |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/instructions.md` | ✅ Complete | Execution instructions |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/uat-fix-context-template.md` | ✅ Complete | Fix context template |
|
||||
| `src/modules/bmm/workflows/4-implementation/epic-chain/templates/chain-report-template.md` | ✅ Complete | Report template |
|
||||
| `src/modules/bmm/workflows/4-implementation/epic-chain/templates/epic-metrics-template.yaml` | ✅ Complete | Metrics schema |
|
||||
| `src/modules/bmm/workflows/4-implementation/epic-chain/steps/step-10-generate-report.md` | ✅ Complete | Report generation step |
|
||||
|
||||
### Files to Create
|
||||
|
||||
| File | Priority | Purpose |
|
||||
|------|----------|---------|
|
||||
| `scripts/uat-validate.sh` | P0 | Shell orchestration for UAT validation |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/steps/step-01-load-uat.md` | P2 | Step file |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/steps/step-02-classify-scenarios.md` | P2 | Step file |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/steps/step-03-execute-scenarios.md` | P2 | Step file |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/steps/step-04-evaluate-gate.md` | P2 | Step file |
|
||||
| `src/modules/bmm/workflows/5-validation/uat-validate/steps/step-05-report-results.md` | P2 | Step file |
|
||||
|
||||
### Files to Modify
|
||||
|
||||
| File | Priority | Changes |
|
||||
|------|----------|---------|
|
||||
| `scripts/epic-execute.sh` | P0 | Add metrics collection |
|
||||
| `scripts/epic-chain.sh` | P1 | Add UAT gate phase, report generation |
|
||||
|
||||
---
|
||||
|
||||
## Testing Plan
|
||||
|
||||
### Unit Tests
|
||||
|
||||
1. **uat-validate.sh** - Test scenario classification, command execution, signal output
|
||||
2. **Metrics collection** - Verify YAML structure matches template
|
||||
3. **Report generation** - Verify all placeholders replaced
|
||||
|
||||
### Integration Tests
|
||||
|
||||
1. **Self-healing loop** - Intentionally fail a scenario, verify fix + re-validate
|
||||
2. **Epic chain with UAT** - Run full chain, verify UAT runs after each epic
|
||||
3. **Report accuracy** - Compare report metrics to actual execution
|
||||
|
||||
### Manual Verification
|
||||
|
||||
1. Run `epic-chain 1-3` on test project
|
||||
2. Verify metrics files created
|
||||
3. Verify UAT validation runs
|
||||
4. Verify report generated with accurate data
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The UAT validation and report generation workflows are architecturally sound but missing the shell orchestration layer that makes BMAD workflows production-ready. The highest priority items are:
|
||||
|
||||
1. **Create `uat-validate.sh`** - Without this, context isolation doesn't happen
|
||||
2. **Instrument `epic-execute.sh`** - Without this, reports have no data
|
||||
|
||||
Once these are complete, the full flow will work:
|
||||
|
||||
```
|
||||
epic-chain → epic-execute → UAT validate → (fix loop if needed) → next epic → report
|
||||
```
|
||||
Loading…
Reference in New Issue