feat: implement UAT validation with self-healing fix loop
Add complete UAT validation workflow implementation: - scripts/uat-validate.sh: Core UAT validation script with scenario classification, automated execution, and self-healing fix loop - scripts/epic-execute.sh: Add metrics instrumentation for story execution tracking (init, update, finalize metrics) - scripts/epic-chain.sh: Integrate UAT gate after each epic with configurable blocking mode, and add report generation phase New workflow step documentation: - step-01-load-uat.md: UAT document loading and parsing - step-02-classify-scenarios.md: Scenario classification logic - step-03-execute-scenarios.md: Automated scenario execution - step-04-evaluate-gate.md: Gate evaluation and fix loop - step-05-report-results.md: Metrics and signal output Key features: - Gate modes: quick (automatable only), full (+ semi-auto), skip - Self-healing: spawns fresh Claude context for targeted fixes - Metrics: YAML files track execution stats and UAT results - Signals: parseable output for orchestration (UAT_GATE_RESULT, etc.) - Handoffs: include UAT status and fix context references Also fixes lint/format issues in pre-existing files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Caleb
parent
6b1ca9333f
commit
15b7bb1ae6
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,526 @@
|
|||
# User Acceptance Testing: Epic 1 - Foundation, CLI & Deployment Infrastructure
|
||||
|
||||
**Version:** 1.0
|
||||
**Date:** January 2, 2026
|
||||
**Epic:** Foundation, CLI & Deployment Infrastructure
|
||||
**Status:** Ready for Testing
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
### What Was Built
|
||||
|
||||
Epic 1 delivers the foundation of the Heimdall Customer Management system. After completing this epic, administrators can:
|
||||
|
||||
- **Initialize a new Heimdall project** using a simple command-line tool
|
||||
- **Configure connections** to Supabase (your database) and Resend (your email service)
|
||||
- **Set up the database** with all necessary tables for job processing
|
||||
- **Run a worker process** that continuously processes background jobs
|
||||
- **Deploy to Railway** (a cloud hosting platform) for production use
|
||||
- **Send test emails** to verify your email configuration is working
|
||||
|
||||
In plain terms: This is the "plumbing" that makes everything else possible. Once Epic 1 is working, you have a running system ready to handle customer management workflows.
|
||||
|
||||
---
|
||||
|
||||
## 2. Prerequisites
|
||||
|
||||
### Test Environment Requirements
|
||||
|
||||
Before starting UAT, ensure you have:
|
||||
|
||||
| Requirement | Details | How to Verify |
|
||||
|-------------|---------|---------------|
|
||||
| **Node.js 20.x** | JavaScript runtime | Run `node --version` - should show v20.x.x |
|
||||
| **npm** | Package manager | Run `npm --version` - should show 9.x or higher |
|
||||
| **Git** | Version control | Run `git --version` |
|
||||
| **Terminal access** | Command line interface | macOS Terminal, Windows PowerShell, or Linux terminal |
|
||||
|
||||
### Required Accounts
|
||||
|
||||
| Service | Purpose | Sign-up URL |
|
||||
|---------|---------|-------------|
|
||||
| **Supabase** | Database hosting | <https://supabase.com> (free tier available) |
|
||||
| **Resend** | Email delivery | <https://resend.com> (free tier: 100 emails/day) |
|
||||
| **Railway** (optional) | Cloud deployment | <https://railway.app> (for production testing) |
|
||||
|
||||
### Credentials You Will Need
|
||||
|
||||
Before testing, gather these from your service dashboards:
|
||||
|
||||
1. **From Supabase Dashboard:**
|
||||
- Project URL (e.g., `https://xxxxx.supabase.co`)
|
||||
- Anon/Public Key (starts with `eyJ...`)
|
||||
- Database Connection String (from Settings > Database > Connection string > URI)
|
||||
|
||||
2. **From Resend Dashboard:**
|
||||
- API Key (starts with `re_...`)
|
||||
- A verified sending domain (or use their default for testing)
|
||||
|
||||
3. **Your Information:**
|
||||
- Admin email address (for receiving notifications and test emails)
|
||||
- Workspace name (a label for your installation)
|
||||
|
||||
---
|
||||
|
||||
## 3. Test Scenarios
|
||||
|
||||
### Scenario 1: Project Initialization
|
||||
|
||||
**Goal:** Verify that you can create a new Heimdall project with the correct structure.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Open your terminal application
|
||||
2. Navigate to a folder where you want to create the project:
|
||||
```
|
||||
cd ~/Documents
|
||||
```
|
||||
3. Clone and set up the Heimdall project:
|
||||
```
|
||||
git clone <repository-url> heimdall-test
|
||||
cd heimdall-test
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
4. Verify the CLI is available:
|
||||
```
|
||||
npx heimdall --version
|
||||
```
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] `npm install` completes without errors
|
||||
- [ ] `npm run build` shows "5 packages built successfully" or similar
|
||||
- [ ] `npx heimdall --version` displays a version number (e.g., `1.0.0`)
|
||||
- [ ] `npx heimdall --help` shows available commands including `config`, `db`, `start`, `test-send`, `test-queue`
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record any error messages or unexpected behavior here:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 2: Configuration Setup
|
||||
|
||||
**Goal:** Create and validate your configuration file with Supabase and Resend credentials.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Generate a configuration template:
|
||||
```
|
||||
npx heimdall config init
|
||||
```
|
||||
2. Open the created file `heimdall.config.yaml` in a text editor
|
||||
3. Replace the placeholder values with your real credentials:
|
||||
- Under `workspace:` - enter your workspace name and admin email
|
||||
- Under `supabase:` - enter your Supabase URL, anon key, and database URL
|
||||
- Under `resend:` - enter your Resend API key
|
||||
4. Save the file
|
||||
5. Validate your configuration:
|
||||
```
|
||||
npx heimdall config validate
|
||||
```
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] `config init` creates a file named `heimdall.config.yaml`
|
||||
- [ ] The file contains sections for `workspace`, `supabase`, `resend`, and `ai` (optional)
|
||||
- [ ] `config validate` shows your workspace name and admin email (with secrets partially hidden)
|
||||
- [ ] Validation shows "Configuration is valid" or similar success message
|
||||
- [ ] No error messages appear about missing or invalid fields
|
||||
|
||||
**Notes for Tester:**
|
||||
_If validation fails, record the error message:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 3: Database Migration
|
||||
|
||||
**Goal:** Set up the required database tables in your Supabase instance.
|
||||
|
||||
**Prerequisites:** Scenario 2 must be completed successfully.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Run the database migration:
|
||||
```
|
||||
npx heimdall db migrate
|
||||
```
|
||||
2. Check the database status:
|
||||
```
|
||||
npx heimdall db status
|
||||
```
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] `db migrate` completes with a success message (e.g., "pg-boss initialized successfully")
|
||||
- [ ] `db status` shows:
|
||||
- Database connected: Yes
|
||||
- pg-boss schema: Exists
|
||||
- Tables: job, schedule, subscription, version (or similar)
|
||||
- [ ] Running `db migrate` a second time does NOT cause errors (idempotent)
|
||||
|
||||
**Verification in Supabase Dashboard:**
|
||||
1. Log into your Supabase project
|
||||
2. Go to the Table Editor
|
||||
3. Look for a schema named `pgboss`
|
||||
4. Verify tables exist: `job`, `schedule`, `subscription`, `version`
|
||||
|
||||
- [ ] pg-boss tables visible in Supabase Dashboard
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record database status output:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 4: Connection Validation (Detailed)
|
||||
|
||||
**Goal:** Verify both Supabase API and direct database connections work correctly.
|
||||
|
||||
**Prerequisites:** Scenario 3 must be completed successfully.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Run the full validation:
|
||||
```
|
||||
npx heimdall config validate
|
||||
```
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Message: "Supabase API connected" with response time in milliseconds
|
||||
- [ ] Message: "Operational DB connected" with response time in milliseconds
|
||||
- [ ] Message: "Resend API connected" (if Resend key is configured)
|
||||
- [ ] All connections show as successful (green checkmarks or similar)
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record connection times and any warnings:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 5: Worker Process Startup
|
||||
|
||||
**Goal:** Start the background worker that processes jobs.
|
||||
|
||||
**Prerequisites:** Scenario 3 must be completed successfully.
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Start the worker process:
|
||||
```
|
||||
npx heimdall start
|
||||
```
|
||||
2. Observe the output for approximately 30 seconds
|
||||
3. Press `Ctrl+C` to stop the worker
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Message appears: "Heimdall worker started, polling for jobs..." or similar
|
||||
- [ ] No error messages during startup
|
||||
- [ ] Worker continues running without crashing
|
||||
- [ ] When you press `Ctrl+C`, message appears: "Heimdall worker shutting down gracefully..."
|
||||
- [ ] Process exits cleanly (returns to command prompt)
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record startup messages:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 6: Job Queue Testing
|
||||
|
||||
**Goal:** Verify that jobs can be queued and processed by the worker.
|
||||
|
||||
**Prerequisites:** Scenario 5 must work (worker can start).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Open **two terminal windows** side by side
|
||||
2. In Terminal 1, start the worker:
|
||||
```
|
||||
npx heimdall start
|
||||
```
|
||||
3. Wait for the "polling for jobs" message
|
||||
4. In Terminal 2, enqueue a test job:
|
||||
```
|
||||
npx heimdall test-queue
|
||||
```
|
||||
5. Watch Terminal 1 for job processing
|
||||
6. Stop the worker in Terminal 1 with `Ctrl+C`
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Terminal 2 shows: "Test job enqueued: test-job-{some-id}"
|
||||
- [ ] Within 60 seconds, Terminal 1 shows: "Job executed: test-job-{same-id}"
|
||||
- [ ] No errors in either terminal
|
||||
- [ ] Job ID in Terminal 2 matches the ID in Terminal 1
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record time between enqueueing and execution:_
|
||||
_Record job ID:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 7: Test Email Sending
|
||||
|
||||
**Goal:** Verify that Heimdall can send emails through Resend.
|
||||
|
||||
**Prerequisites:**
|
||||
- Scenario 2 completed with valid Resend API key
|
||||
- You have access to the email inbox specified
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Send a test email to yourself:
|
||||
```
|
||||
npx heimdall test-send --to your-email@example.com
|
||||
```
|
||||
(Replace with your actual email address)
|
||||
2. Check your email inbox (including spam/junk folder)
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Command shows: "Test email sent: {resend-message-id}"
|
||||
- [ ] Email arrives in inbox within 2-5 minutes
|
||||
- [ ] Email subject: "Heimdall Test Email"
|
||||
- [ ] Email body confirms configuration is working
|
||||
|
||||
**Notes for Tester:**
|
||||
_Record Resend message ID:_
|
||||
_Time until email arrived:_
|
||||
_Did email land in spam?:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 8: Health Endpoint (Local)
|
||||
|
||||
**Goal:** Verify the worker's health endpoint responds correctly.
|
||||
|
||||
**Prerequisites:** Worker can start (Scenario 5).
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Start the worker:
|
||||
```
|
||||
npx heimdall start
|
||||
```
|
||||
2. Open a web browser
|
||||
3. Navigate to: `http://localhost:3000/health`
|
||||
4. Stop the worker with `Ctrl+C`
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Browser shows JSON response
|
||||
- [ ] Response contains: `"status": "healthy"`
|
||||
- [ ] Response contains: `"queue": "connected"`
|
||||
- [ ] Response contains: `"uptime": {some-number}`
|
||||
|
||||
**Sample Expected Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"queue": "connected",
|
||||
"uptime": 45
|
||||
}
|
||||
```
|
||||
|
||||
**Notes for Tester:**
|
||||
_Copy the actual response here:_
|
||||
|
||||
---
|
||||
|
||||
### Scenario 9: Railway Deployment (Optional)
|
||||
|
||||
**Goal:** Deploy Heimdall to Railway for production use.
|
||||
|
||||
**Prerequisites:**
|
||||
- Railway account created
|
||||
- All previous scenarios pass locally
|
||||
|
||||
**Steps:**
|
||||
|
||||
1. Log into Railway Dashboard
|
||||
2. Create a new project from the GitHub repository
|
||||
3. Add environment variables in Railway settings:
|
||||
- `DATABASE_URL` = your Supabase connection string
|
||||
- `SUPABASE_URL` = your Supabase project URL
|
||||
- `SUPABASE_ANON_KEY` = your Supabase anon key
|
||||
- `RESEND_API_KEY` = your Resend API key
|
||||
- `ADMIN_EMAIL` = your admin email
|
||||
- `WORKSPACE_NAME` = your workspace name
|
||||
4. Deploy the service
|
||||
5. Once deployed, access the health endpoint at your Railway URL + `/health`
|
||||
|
||||
**Expected Results:**
|
||||
|
||||
- [ ] Deployment completes without build errors
|
||||
- [ ] Service shows as "Running" in Railway dashboard
|
||||
- [ ] Health endpoint at `https://your-app.railway.app/health` returns:
|
||||
- `"status": "healthy"`
|
||||
- `"queue": "connected"`
|
||||
- [ ] Logs show "Heimdall worker started, polling for jobs..."
|
||||
|
||||
**Notes for Tester:**
|
||||
_Railway deployment URL:_
|
||||
_Any deployment warnings or notes:_
|
||||
|
||||
---
|
||||
|
||||
## 4. Success Criteria
|
||||
|
||||
### Minimum Requirements for Sign-off
|
||||
|
||||
All of the following must pass for Epic 1 to be accepted:
|
||||
|
||||
| # | Criteria | Scenario | Status |
|
||||
|---|----------|----------|--------|
|
||||
| 1 | Project builds successfully | Scenario 1 | [ ] Pass / [ ] Fail |
|
||||
| 2 | CLI commands are accessible | Scenario 1 | [ ] Pass / [ ] Fail |
|
||||
| 3 | Configuration file is created correctly | Scenario 2 | [ ] Pass / [ ] Fail |
|
||||
| 4 | Configuration validation works | Scenario 2 | [ ] Pass / [ ] Fail |
|
||||
| 5 | Database migration completes | Scenario 3 | [ ] Pass / [ ] Fail |
|
||||
| 6 | Database migration is idempotent | Scenario 3 | [ ] Pass / [ ] Fail |
|
||||
| 7 | Supabase API connection validated | Scenario 4 | [ ] Pass / [ ] Fail |
|
||||
| 8 | Operational DB connection validated | Scenario 4 | [ ] Pass / [ ] Fail |
|
||||
| 9 | Worker process starts and polls | Scenario 5 | [ ] Pass / [ ] Fail |
|
||||
| 10 | Worker shuts down gracefully | Scenario 5 | [ ] Pass / [ ] Fail |
|
||||
| 11 | Test jobs are processed | Scenario 6 | [ ] Pass / [ ] Fail |
|
||||
| 12 | Jobs processed within 60 seconds | Scenario 6 | [ ] Pass / [ ] Fail |
|
||||
| 13 | Test emails are sent and received | Scenario 7 | [ ] Pass / [ ] Fail |
|
||||
| 14 | Health endpoint responds correctly | Scenario 8 | [ ] Pass / [ ] Fail |
|
||||
|
||||
### Optional (Recommended)
|
||||
|
||||
| # | Criteria | Scenario | Status |
|
||||
|---|----------|----------|--------|
|
||||
| 15 | Railway deployment successful | Scenario 9 | [ ] Pass / [ ] Fail / [ ] Skipped |
|
||||
| 16 | Production health endpoint works | Scenario 9 | [ ] Pass / [ ] Fail / [ ] Skipped |
|
||||
|
||||
---
|
||||
|
||||
## 5. Known Limitations
|
||||
|
||||
The following are expected behaviors, not bugs:
|
||||
|
||||
1. **Error messages for invalid credentials** are intentionally detailed to help debugging
|
||||
2. **First database migration** may take 10-30 seconds as pg-boss creates multiple tables
|
||||
3. **Test emails** may land in spam for unverified domains
|
||||
4. **Worker polling interval** is 5 seconds, so jobs may take up to 5 seconds to start processing
|
||||
5. **Config validate** shows partial secrets (first/last 4 characters) for verification purposes
|
||||
|
||||
---
|
||||
|
||||
## 6. Troubleshooting Guide
|
||||
|
||||
### Common Issues and Solutions
|
||||
|
||||
| Symptom | Likely Cause | Solution |
|
||||
|---------|--------------|----------|
|
||||
| "Command not found: heimdall" | Build not completed | Run `npm run build` first |
|
||||
| "Cannot connect to database" | Wrong db_url | Check Supabase connection string format |
|
||||
| "Invalid API key" | Resend key incorrect | Regenerate key in Resend dashboard |
|
||||
| "ECONNREFUSED" | Database not accessible | Check Supabase project is running, check IP restrictions |
|
||||
| Worker crashes on startup | Missing configuration | Run `heimdall config validate` first |
|
||||
| Email not received | Domain not verified | Verify domain in Resend or check spam folder |
|
||||
|
||||
---
|
||||
|
||||
## 7. Sign-off Section
|
||||
|
||||
### Testing Summary
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| **Tester Name** | _________________________ |
|
||||
| **Test Date** | _________________________ |
|
||||
| **Environment** | [ ] Local / [ ] Railway / [ ] Both |
|
||||
| **Node.js Version** | _________________________ |
|
||||
| **Operating System** | _________________________ |
|
||||
|
||||
### Test Results Summary
|
||||
|
||||
| Category | Passed | Failed | Skipped |
|
||||
|----------|--------|--------|---------|
|
||||
| Required Scenarios (1-8) | ___ / 8 | ___ | ___ |
|
||||
| Optional Scenarios (9) | ___ / 1 | ___ | ___ |
|
||||
| Success Criteria (1-14) | ___ / 14 | ___ | ___ |
|
||||
|
||||
### Issues Found
|
||||
|
||||
| Issue # | Scenario | Description | Severity |
|
||||
|---------|----------|-------------|----------|
|
||||
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||
|
||||
### Final Decision
|
||||
|
||||
- [ ] **APPROVED** - All required criteria pass, ready for production
|
||||
- [ ] **APPROVED WITH CONDITIONS** - Minor issues noted, can proceed
|
||||
- [ ] **NOT APPROVED** - Blocker issues must be resolved
|
||||
|
||||
### Signatures
|
||||
|
||||
| Role | Name | Signature | Date |
|
||||
|------|------|-----------|------|
|
||||
| **Tester** | | | |
|
||||
| **Product Owner** | | | |
|
||||
| **Technical Lead** | | | |
|
||||
|
||||
---
|
||||
|
||||
## 8. Appendix
|
||||
|
||||
### A. CLI Command Reference
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `heimdall --version` | Display version number |
|
||||
| `heimdall --help` | Show available commands |
|
||||
| `heimdall config init` | Create configuration file |
|
||||
| `heimdall config validate` | Validate configuration |
|
||||
| `heimdall db migrate` | Set up database tables |
|
||||
| `heimdall db status` | Check database status |
|
||||
| `heimdall start` | Start worker process |
|
||||
| `heimdall test-queue` | Enqueue a test job |
|
||||
| `heimdall test-send --to EMAIL` | Send a test email |
|
||||
|
||||
### B. Configuration File Template
|
||||
|
||||
```yaml
|
||||
# heimdall.config.yaml
|
||||
workspace:
|
||||
name: "my-workspace"
|
||||
admin_email: "admin@example.com"
|
||||
|
||||
supabase:
|
||||
url: "https://xxxxx.supabase.co"
|
||||
anon_key: "eyJ..."
|
||||
db_url: "postgresql://postgres:password@db.xxxxx.supabase.co:5432/postgres"
|
||||
|
||||
resend:
|
||||
api_key: "re_..."
|
||||
|
||||
# Optional AI configuration
|
||||
ai:
|
||||
provider: "anthropic"
|
||||
api_key: "sk-ant-..."
|
||||
model: "claude-3-haiku-20240307"
|
||||
```
|
||||
|
||||
### C. Environment Variables for Railway
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `DATABASE_URL` | Yes | PostgreSQL connection string |
|
||||
| `SUPABASE_URL` | Yes | Supabase project URL |
|
||||
| `SUPABASE_ANON_KEY` | Yes | Supabase anonymous key |
|
||||
| `RESEND_API_KEY` | Yes | Resend API key |
|
||||
| `ADMIN_EMAIL` | Yes | Admin notification email |
|
||||
| `WORKSPACE_NAME` | No | Workspace identifier |
|
||||
| `PORT` | No | Server port (default: 3000) |
|
||||
|
||||
---
|
||||
|
||||
*Document generated: January 2, 2026*
|
||||
*Epic 1: Foundation, CLI & Deployment Infrastructure*
|
||||
|
|
@ -9,15 +9,25 @@
|
|||
# ./epic-chain.sh 36 37 38 --dry-run --verbose
|
||||
# ./epic-chain.sh 36 37 38 --analyze-only
|
||||
# ./epic-chain.sh 36 37 38 --start-from 37
|
||||
# ./epic-chain.sh 36 37 38 --uat-gate=full --uat-blocking
|
||||
#
|
||||
# Options:
|
||||
# --dry-run Show what would be executed without running
|
||||
# --analyze-only Run analysis phase only, don't execute
|
||||
# --verbose Show detailed output
|
||||
# --start-from ID Start from a specific epic (skip earlier ones)
|
||||
# --skip-done Skip epics/stories with Status: Done
|
||||
# --no-handoff Don't generate context handoffs between epics
|
||||
# --no-combined-uat Skip combined UAT generation at end
|
||||
# --dry-run Show what would be executed without running
|
||||
# --analyze-only Run analysis phase only, don't execute
|
||||
# --verbose Show detailed output
|
||||
# --start-from ID Start from a specific epic (skip earlier ones)
|
||||
# --skip-done Skip epics/stories with Status: Done
|
||||
# --no-handoff Don't generate context handoffs between epics
|
||||
# --no-combined-uat Skip combined UAT generation at end
|
||||
#
|
||||
# UAT Gate Options:
|
||||
# --uat-gate=MODE UAT validation mode: quick|full|skip (default: quick)
|
||||
# --uat-blocking Halt chain if UAT fails (default: continue)
|
||||
# --uat-retries=N Max fix attempts per epic (default: 2)
|
||||
# --no-uat Disable UAT validation gate entirely
|
||||
#
|
||||
# Report Options:
|
||||
# --no-report Skip chain execution report generation
|
||||
#
|
||||
|
||||
set -e
|
||||
|
|
@ -48,6 +58,19 @@ CYAN='\033[0;36m'
|
|||
BOLD='\033[1m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# UAT Gate Configuration
|
||||
UAT_GATE_ENABLED="${UAT_GATE_ENABLED:-true}"
|
||||
UAT_GATE_MODE="${UAT_GATE_MODE:-quick}"
|
||||
UAT_MAX_RETRIES="${UAT_MAX_RETRIES:-2}"
|
||||
UAT_BLOCKING="${UAT_BLOCKING:-false}"
|
||||
|
||||
# Metrics Configuration
|
||||
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||
|
||||
# Report Configuration
|
||||
GENERATE_REPORT="${GENERATE_REPORT:-true}"
|
||||
CHAIN_REPORT_FILE="$SPRINT_ARTIFACTS_DIR/chain-execution-report.md"
|
||||
|
||||
# =============================================================================
|
||||
# Helper Functions
|
||||
# =============================================================================
|
||||
|
|
@ -87,6 +110,80 @@ log_section() {
|
|||
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||
}
|
||||
|
||||
# Helper function to create basic report if Claude fails
|
||||
create_basic_report() {
|
||||
local end_time_iso=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
local duration_formatted="${DURATION}s"
|
||||
if [ $DURATION -gt 3600 ]; then
|
||||
duration_formatted="$((DURATION / 3600))h $((DURATION % 3600 / 60))m"
|
||||
elif [ $DURATION -gt 60 ]; then
|
||||
duration_formatted="$((DURATION / 60))m $((DURATION % 60))s"
|
||||
fi
|
||||
|
||||
cat > "$CHAIN_REPORT_FILE" << EOF
|
||||
# Epic Chain Execution Report
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Execution Method:** BMAD Epic Chain (automated AI-driven development)
|
||||
**Status:** $([ $FAILED_EPICS -eq 0 ] && echo "COMPLETE" || echo "PARTIAL")
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total Epics | ${#EPIC_IDS[@]} |
|
||||
| Completed | $COMPLETED_EPICS |
|
||||
| Failed | $FAILED_EPICS |
|
||||
| Skipped | $SKIPPED_EPICS |
|
||||
| Duration | $duration_formatted |
|
||||
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
| Epic | Status |
|
||||
|------|--------|
|
||||
EOF
|
||||
|
||||
for epic_id in "${EPIC_IDS[@]}"; do
|
||||
local status="Unknown"
|
||||
local metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||
if [ -f "$metrics_file" ]; then
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
local completed=$(yq '.stories.completed // 0' "$metrics_file")
|
||||
local failed=$(yq '.stories.failed // 0' "$metrics_file")
|
||||
if [ "$failed" -gt 0 ]; then
|
||||
status="Partial ($completed completed, $failed failed)"
|
||||
else
|
||||
status="Complete ($completed stories)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
echo "| Epic $epic_id | $status |" >> "$CHAIN_REPORT_FILE"
|
||||
done
|
||||
|
||||
cat >> "$CHAIN_REPORT_FILE" << EOF
|
||||
|
||||
---
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Artifact | Location |
|
||||
|----------|----------|
|
||||
| Chain Plan | $CHAIN_PLAN_FILE |
|
||||
| Metrics | $METRICS_DIR/ |
|
||||
| UAT Documents | $UAT_DIR/ |
|
||||
| Handoffs | $HANDOFF_DIR/ |
|
||||
| Log | $LOG_FILE |
|
||||
|
||||
---
|
||||
|
||||
*Report generated: $end_time_iso*
|
||||
*BMAD Method Epic Chain*
|
||||
EOF
|
||||
|
||||
log_success "Basic report created: $CHAIN_REPORT_FILE"
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Argument Parsing
|
||||
# =============================================================================
|
||||
|
|
@ -130,6 +227,26 @@ while [[ $# -gt 0 ]]; do
|
|||
NO_COMBINED_UAT=true
|
||||
shift
|
||||
;;
|
||||
--uat-gate=*)
|
||||
UAT_GATE_MODE="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--uat-blocking)
|
||||
UAT_BLOCKING=true
|
||||
shift
|
||||
;;
|
||||
--no-uat)
|
||||
UAT_GATE_ENABLED=false
|
||||
shift
|
||||
;;
|
||||
--uat-retries=*)
|
||||
UAT_MAX_RETRIES="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--no-report)
|
||||
GENERATE_REPORT=false
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Unknown option: $1"
|
||||
exit 1
|
||||
|
|
@ -149,15 +266,25 @@ if [ ${#EPIC_IDS[@]} -eq 0 ]; then
|
|||
echo " $0 36 37 38 --dry-run # Show what would happen"
|
||||
echo " $0 36 37 38 --analyze-only # Just analyze, don't execute"
|
||||
echo " $0 36 37 38 --start-from 37 # Resume from epic 37"
|
||||
echo " $0 36 37 38 --uat-gate=full # Run full UAT validation after each epic"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --dry-run Show execution plan without running"
|
||||
echo " --analyze-only Analyze dependencies only"
|
||||
echo " --verbose Detailed output"
|
||||
echo " --start-from ID Start from specific epic"
|
||||
echo " --skip-done Skip completed stories"
|
||||
echo " --no-handoff Skip context handoffs between epics"
|
||||
echo " --no-combined-uat Skip combined UAT at end"
|
||||
echo " --dry-run Show execution plan without running"
|
||||
echo " --analyze-only Analyze dependencies only"
|
||||
echo " --verbose Detailed output"
|
||||
echo " --start-from ID Start from specific epic"
|
||||
echo " --skip-done Skip completed stories"
|
||||
echo " --no-handoff Skip context handoffs between epics"
|
||||
echo " --no-combined-uat Skip combined UAT at end"
|
||||
echo ""
|
||||
echo "UAT Gate Options:"
|
||||
echo " --uat-gate=MODE UAT validation mode: quick|full|skip (default: quick)"
|
||||
echo " --uat-blocking Halt chain if UAT fails (default: continue)"
|
||||
echo " --uat-retries=N Max fix attempts per epic (default: 2)"
|
||||
echo " --no-uat Disable UAT validation gate entirely"
|
||||
echo ""
|
||||
echo "Report Options:"
|
||||
echo " --no-report Skip chain execution report generation"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
|
@ -395,6 +522,60 @@ for current_idx in "${!EXECUTION_ORDER[@]}"; do
|
|||
else
|
||||
if $exec_cmd; then
|
||||
log_success "Epic $epic_id completed"
|
||||
|
||||
# Run UAT validation if enabled
|
||||
if [ "$UAT_GATE_ENABLED" = true ]; then
|
||||
log_section "UAT Validation Gate: Epic $epic_id"
|
||||
|
||||
uat_cmd="$SCRIPT_DIR/uat-validate.sh $epic_id --gate-mode=$UAT_GATE_MODE --max-retries=$UAT_MAX_RETRIES"
|
||||
|
||||
if [ "$VERBOSE" = true ]; then
|
||||
uat_cmd="$uat_cmd --verbose"
|
||||
fi
|
||||
|
||||
log "Running: $uat_cmd"
|
||||
|
||||
# Capture UAT result
|
||||
uat_output=""
|
||||
uat_exit_code=0
|
||||
uat_output=$($uat_cmd 2>&1) || uat_exit_code=$?
|
||||
|
||||
echo "$uat_output" >> "$LOG_FILE"
|
||||
|
||||
# Parse result signals
|
||||
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||
log_success "UAT validation passed for Epic $epic_id"
|
||||
|
||||
# Update metrics file if it exists
|
||||
epic_metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||
if [ -f "$epic_metrics_file" ] && command -v yq >/dev/null 2>&1; then
|
||||
yq -i '.validation.gate_executed = true' "$epic_metrics_file"
|
||||
yq -i '.validation.gate_status = "PASS"' "$epic_metrics_file"
|
||||
fi
|
||||
else
|
||||
log_error "UAT validation failed for Epic $epic_id"
|
||||
|
||||
# Update metrics file if it exists
|
||||
epic_metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||
if [ -f "$epic_metrics_file" ] && command -v yq >/dev/null 2>&1; then
|
||||
yq -i '.validation.gate_executed = true' "$epic_metrics_file"
|
||||
yq -i '.validation.gate_status = "FAIL"' "$epic_metrics_file"
|
||||
fi
|
||||
|
||||
# Extract fix attempts from output
|
||||
fix_attempts=$(echo "$uat_output" | grep -oE "UAT_FIX_ATTEMPTS: [0-9]+" | grep -oE "[0-9]+" || echo "0")
|
||||
[ "$VERBOSE" = true ] && log "Fix attempts: $fix_attempts"
|
||||
|
||||
if [ "$UAT_BLOCKING" = true ]; then
|
||||
log_error "UAT blocking enabled - halting chain"
|
||||
((FAILED_EPICS++))
|
||||
break
|
||||
else
|
||||
log_warn "UAT blocking disabled - continuing to next epic"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
((COMPLETED_EPICS++))
|
||||
|
||||
# Generate handoff for next epic
|
||||
|
|
@ -408,6 +589,26 @@ for current_idx in "${!EXECUTION_ORDER[@]}"; do
|
|||
log "Generating context handoff: Epic $epic_id → Epic $next_epic"
|
||||
|
||||
story_count=${EPIC_STORIES_LIST[$current_idx]}
|
||||
|
||||
# Determine UAT validation status for handoff
|
||||
uat_status="Not executed"
|
||||
uat_fix_info=""
|
||||
if [ "$UAT_GATE_ENABLED" = true ]; then
|
||||
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||
uat_status="PASS"
|
||||
local fix_count=$(echo "$uat_output" | grep -oE "UAT_FIX_ATTEMPTS: [0-9]+" | grep -oE "[0-9]+" || echo "0")
|
||||
if [ "$fix_count" -gt 0 ]; then
|
||||
uat_status="PASS (after $fix_count fix attempts)"
|
||||
uat_fix_info="Self-healing fixes were applied. Review fix contexts at:
|
||||
\`docs/sprint-artifacts/uat-fixes/epic-${epic_id}-fix-context-*.md\`"
|
||||
fi
|
||||
else
|
||||
uat_status="FAIL (non-blocking)"
|
||||
uat_fix_info="UAT validation failed but chain continued (non-blocking mode).
|
||||
Review failures at: \`docs/sprint-artifacts/uat-fixes/epic-${epic_id}-fix-context-*.md\`"
|
||||
fi
|
||||
fi
|
||||
|
||||
cat > "$handoff_file" << EOF
|
||||
# Epic $epic_id → Epic $next_epic Handoff
|
||||
|
||||
|
|
@ -418,6 +619,11 @@ $(date '+%Y-%m-%d %H:%M:%S')
|
|||
|
||||
Epic $epic_id has been completed. Key context for Epic $next_epic:
|
||||
|
||||
### Implementation Status
|
||||
- **Stories:** Completed via epic-execute workflow
|
||||
- **UAT Validation:** $uat_status
|
||||
- **Metrics:** \`$METRICS_DIR/epic-${epic_id}-metrics.yaml\`
|
||||
|
||||
### Patterns Established
|
||||
- Review code changes in Epic $epic_id for established patterns
|
||||
- Check \`docs/stories/${epic_id}-*\` for implementation details
|
||||
|
|
@ -425,9 +631,17 @@ Epic $epic_id has been completed. Key context for Epic $next_epic:
|
|||
### Files Modified
|
||||
$(git diff --name-only HEAD~${story_count} HEAD 2>/dev/null | head -20 || echo "Unable to determine - check git log")
|
||||
|
||||
### UAT Document
|
||||
- Location: \`docs/uat/epic-${epic_id}-uat.md\`
|
||||
- Contains test scenarios for regression testing
|
||||
|
||||
$([ -n "$uat_fix_info" ] && echo "### Fix Context
|
||||
$uat_fix_info")
|
||||
|
||||
### Notes for Next Epic
|
||||
- Continue following patterns established in this epic
|
||||
- Reference UAT document at \`docs/uat/epic-${epic_id}-uat.md\` for context
|
||||
- Ensure changes don't break Epic $epic_id functionality
|
||||
- Reference UAT document for integration points
|
||||
|
||||
EOF
|
||||
log_success "Handoff saved to: $handoff_file"
|
||||
|
|
@ -508,6 +722,108 @@ EOF
|
|||
log_success "Combined UAT saved to: $combined_uat_file"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# Phase 7: Generate Chain Execution Report
|
||||
# =============================================================================
|
||||
|
||||
if [ "$GENERATE_REPORT" = true ] && [ "$DRY_RUN" = false ]; then
|
||||
log_section "Generating Chain Execution Report"
|
||||
|
||||
# Check if metrics files exist
|
||||
metrics_found=0
|
||||
for epic_id in "${EPIC_IDS[@]}"; do
|
||||
if [ -f "$METRICS_DIR/epic-${epic_id}-metrics.yaml" ]; then
|
||||
((metrics_found++))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $metrics_found -eq 0 ]; then
|
||||
log_warn "No metrics files found - skipping report generation"
|
||||
else
|
||||
log "Found $metrics_found metrics files"
|
||||
|
||||
# Determine workflow path (installed vs source)
|
||||
WORKFLOW_PATH=""
|
||||
if [ -d "$BMAD_DIR/bmm/workflows/4-implementation/epic-chain" ]; then
|
||||
WORKFLOW_PATH="$BMAD_DIR/bmm/workflows/4-implementation/epic-chain"
|
||||
elif [ -d "$PROJECT_ROOT/src/modules/bmm/workflows/4-implementation/epic-chain" ]; then
|
||||
WORKFLOW_PATH="$PROJECT_ROOT/src/modules/bmm/workflows/4-implementation/epic-chain"
|
||||
fi
|
||||
|
||||
# Build report generation prompt
|
||||
report_prompt="You are Bob, the Scrum Master, generating a chain execution report.
|
||||
|
||||
## Your Task
|
||||
|
||||
Generate a comprehensive chain execution report for the completed epic chain.
|
||||
|
||||
## Configuration
|
||||
|
||||
- Chain Plan: $CHAIN_PLAN_FILE
|
||||
- Metrics Folder: $METRICS_DIR
|
||||
- Output File: $CHAIN_REPORT_FILE
|
||||
- Stories Location: $STORIES_DIR
|
||||
- UAT Location: $UAT_DIR
|
||||
- Epics Location: $EPICS_DIR
|
||||
- Handoffs Location: $HANDOFF_DIR
|
||||
|
||||
## Epics in Chain
|
||||
|
||||
${EPIC_IDS[*]}
|
||||
|
||||
## Process
|
||||
|
||||
1. Read the chain plan file to understand the epic sequence
|
||||
2. For each epic, load the metrics file from: $METRICS_DIR/epic-{id}-metrics.yaml
|
||||
3. Aggregate metrics across all epics:
|
||||
- Total duration
|
||||
- Story counts (total, completed, failed, skipped)
|
||||
- UAT gate results
|
||||
- Issues encountered
|
||||
4. Generate the report following the template structure
|
||||
|
||||
## Report Structure
|
||||
|
||||
Generate a markdown report with these sections:
|
||||
- Executive Summary (status, counts, duration)
|
||||
- Timeline (epic-by-epic execution details)
|
||||
- What Was Built (brief per-epic summary)
|
||||
- Issues Encountered (aggregated from metrics)
|
||||
- UAT Validation Summary (gate results, fix attempts)
|
||||
- Artifacts Generated (list generated files)
|
||||
- Conclusion
|
||||
|
||||
## Output
|
||||
|
||||
Write the report to: $CHAIN_REPORT_FILE
|
||||
|
||||
When complete, output exactly:
|
||||
REPORT_GENERATED: $CHAIN_REPORT_FILE"
|
||||
|
||||
log "Invoking report generator..."
|
||||
|
||||
# Execute report generation
|
||||
report_result=$(claude --dangerously-skip-permissions -p "$report_prompt" 2>&1) || true
|
||||
|
||||
echo "$report_result" >> "$LOG_FILE"
|
||||
|
||||
if echo "$report_result" | grep -q "REPORT_GENERATED"; then
|
||||
log_success "Report generated: $CHAIN_REPORT_FILE"
|
||||
|
||||
# Stage report file
|
||||
git add "$CHAIN_REPORT_FILE" 2>/dev/null || true
|
||||
else
|
||||
log_warn "Report generation may not have completed cleanly"
|
||||
|
||||
# If Claude didn't generate it, create a basic report
|
||||
if [ ! -f "$CHAIN_REPORT_FILE" ]; then
|
||||
log "Creating basic report from metrics..."
|
||||
create_basic_report
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# Summary
|
||||
# =============================================================================
|
||||
|
|
@ -527,6 +843,10 @@ echo " Artifacts:"
|
|||
echo " - Chain Plan: $CHAIN_PLAN_FILE"
|
||||
echo " - Handoffs: $HANDOFF_DIR/"
|
||||
echo " - UAT Documents: $UAT_DIR/"
|
||||
echo " - Metrics: $METRICS_DIR/"
|
||||
if [ -f "$CHAIN_REPORT_FILE" ]; then
|
||||
echo " - Report: $CHAIN_REPORT_FILE"
|
||||
fi
|
||||
echo " - Log: $LOG_FILE"
|
||||
echo ""
|
||||
|
||||
|
|
@ -537,5 +857,12 @@ fi
|
|||
|
||||
log_success "All epics completed successfully"
|
||||
echo ""
|
||||
echo "Next step: Review UAT documents and run manual testing"
|
||||
if [ -f "$CHAIN_REPORT_FILE" ]; then
|
||||
echo "Next steps:"
|
||||
echo " 1. Review execution report: $CHAIN_REPORT_FILE"
|
||||
echo " 2. Run UAT validation for each epic"
|
||||
echo " 3. Execute manual test scenarios"
|
||||
else
|
||||
echo "Next step: Review UAT documents and run manual testing"
|
||||
fi
|
||||
echo ""
|
||||
|
|
|
|||
|
|
@ -63,6 +63,121 @@ log_warn() {
|
|||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Metrics Functions
|
||||
# =============================================================================
|
||||
|
||||
METRICS_DIR=""
|
||||
METRICS_FILE=""
|
||||
|
||||
init_metrics() {
|
||||
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||
METRICS_FILE="$METRICS_DIR/epic-${EPIC_ID}-metrics.yaml"
|
||||
mkdir -p "$METRICS_DIR"
|
||||
|
||||
local start_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
cat > "$METRICS_FILE" << EOF
|
||||
epic_id: "$EPIC_ID"
|
||||
execution:
|
||||
start_time: "$start_time"
|
||||
end_time: ""
|
||||
duration_seconds: 0
|
||||
stories:
|
||||
total: 0
|
||||
completed: 0
|
||||
failed: 0
|
||||
skipped: 0
|
||||
validation:
|
||||
gate_executed: false
|
||||
gate_status: "PENDING"
|
||||
fix_attempts: 0
|
||||
issues: []
|
||||
EOF
|
||||
|
||||
log "Metrics initialized: $METRICS_FILE"
|
||||
}
|
||||
|
||||
update_story_metrics() {
|
||||
local status="$1" # completed|failed|skipped
|
||||
|
||||
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
# Check if yq is available for YAML manipulation
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
case "$status" in
|
||||
completed) yq -i '.stories.completed += 1' "$METRICS_FILE" ;;
|
||||
failed) yq -i '.stories.failed += 1' "$METRICS_FILE" ;;
|
||||
skipped) yq -i '.stories.skipped += 1' "$METRICS_FILE" ;;
|
||||
esac
|
||||
else
|
||||
# Fallback: log warning (metrics will be finalized at end)
|
||||
[ "$VERBOSE" = true ] && log_warn "yq not found - metrics update deferred"
|
||||
fi
|
||||
}
|
||||
|
||||
add_metrics_issue() {
|
||||
local story_id="$1"
|
||||
local issue_type="$2"
|
||||
local message="$3"
|
||||
|
||||
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
yq -i ".issues += [{\"story\": \"$story_id\", \"type\": \"$issue_type\", \"message\": \"$message\", \"timestamp\": \"$timestamp\"}]" "$METRICS_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
finalize_metrics() {
|
||||
local total_stories="$1"
|
||||
local completed="$2"
|
||||
local failed="$3"
|
||||
local skipped="$4"
|
||||
local duration="$5"
|
||||
|
||||
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
local end_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
yq -i ".execution.end_time = \"$end_time\"" "$METRICS_FILE"
|
||||
yq -i ".execution.duration_seconds = $duration" "$METRICS_FILE"
|
||||
yq -i ".stories.total = $total_stories" "$METRICS_FILE"
|
||||
yq -i ".stories.completed = $completed" "$METRICS_FILE"
|
||||
yq -i ".stories.failed = $failed" "$METRICS_FILE"
|
||||
yq -i ".stories.skipped = $skipped" "$METRICS_FILE"
|
||||
else
|
||||
# Fallback: rewrite the file with final values
|
||||
cat > "$METRICS_FILE" << EOF
|
||||
epic_id: "$EPIC_ID"
|
||||
execution:
|
||||
start_time: "$EPIC_START_TIME"
|
||||
end_time: "$end_time"
|
||||
duration_seconds: $duration
|
||||
stories:
|
||||
total: $total_stories
|
||||
completed: $completed
|
||||
failed: $failed
|
||||
skipped: $skipped
|
||||
validation:
|
||||
gate_executed: false
|
||||
gate_status: "PENDING"
|
||||
fix_attempts: 0
|
||||
issues: []
|
||||
EOF
|
||||
fi
|
||||
|
||||
log "Metrics finalized: $METRICS_FILE"
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Argument Parsing
|
||||
# =============================================================================
|
||||
|
|
@ -142,6 +257,11 @@ log "Project root: $PROJECT_ROOT"
|
|||
mkdir -p "$UAT_DIR"
|
||||
mkdir -p "$SPRINTS_DIR"
|
||||
|
||||
# Initialize metrics collection
|
||||
EPIC_START_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
EPIC_START_SECONDS=$(date +%s)
|
||||
init_metrics
|
||||
|
||||
# Find epic file (supports both epic-39-*.md and epic-039-*.md formats)
|
||||
EPIC_FILE=""
|
||||
# Pad epic ID with leading zero for 3-digit format (e.g., 40 -> 040)
|
||||
|
|
@ -554,6 +674,7 @@ for story_file in "${STORIES[@]}"; do
|
|||
else
|
||||
log_warn "Skipping $story_id (waiting for $START_FROM)"
|
||||
((SKIPPED++))
|
||||
update_story_metrics "skipped"
|
||||
continue
|
||||
fi
|
||||
fi
|
||||
|
|
@ -563,6 +684,7 @@ for story_file in "${STORIES[@]}"; do
|
|||
if grep -q "^Status:.*Done" "$story_file" 2>/dev/null; then
|
||||
log_warn "Skipping $story_id (Status: Done)"
|
||||
((SKIPPED++))
|
||||
update_story_metrics "skipped"
|
||||
continue
|
||||
fi
|
||||
fi
|
||||
|
|
@ -576,22 +698,27 @@ for story_file in "${STORIES[@]}"; do
|
|||
if ! execute_dev_phase "$story_file"; then
|
||||
log_error "Dev phase failed for $story_id"
|
||||
((FAILED++))
|
||||
update_story_metrics "failed"
|
||||
add_metrics_issue "$story_id" "dev_phase_failed" "Development phase did not complete"
|
||||
continue
|
||||
fi
|
||||
|
||||
|
||||
# REVIEW PHASE (Context 2 - Fresh)
|
||||
if [ "$SKIP_REVIEW" = false ]; then
|
||||
if ! execute_review_phase "$story_file"; then
|
||||
log_error "Review phase failed for $story_id"
|
||||
((FAILED++))
|
||||
update_story_metrics "failed"
|
||||
add_metrics_issue "$story_id" "review_failed" "Code review phase failed"
|
||||
continue
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# COMMIT
|
||||
commit_story "$story_id"
|
||||
|
||||
|
||||
((COMPLETED++))
|
||||
update_story_metrics "completed"
|
||||
log_success "Story complete: $story_id ($COMPLETED/${#STORIES[@]})"
|
||||
done
|
||||
|
||||
|
|
@ -613,6 +740,9 @@ generate_uat
|
|||
END_TIME=$(date +%s)
|
||||
DURATION=$((END_TIME - START_TIME))
|
||||
|
||||
# Finalize metrics with final counts
|
||||
finalize_metrics "${#STORIES[@]}" "$COMPLETED" "$FAILED" "$SKIPPED" "$DURATION"
|
||||
|
||||
echo ""
|
||||
log "=========================================="
|
||||
log "EPIC EXECUTION COMPLETE"
|
||||
|
|
@ -628,6 +758,7 @@ echo ""
|
|||
echo " Deliverables:"
|
||||
echo " - Stories: $STORIES_DIR/"
|
||||
echo " - UAT: $UAT_DIR/epic-${EPIC_ID}-uat.md"
|
||||
echo " - Metrics: $METRICS_FILE"
|
||||
echo " - Log: $LOG_FILE"
|
||||
echo ""
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,827 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# BMAD UAT Validate - Automated UAT Scenario Execution with Self-Healing Fix Loop
|
||||
#
|
||||
# Usage: ./uat-validate.sh <epic-id> [options]
|
||||
#
|
||||
# Options:
|
||||
# --gate-mode=MODE Validation mode: quick|full|skip (default: quick)
|
||||
# --max-retries=N Max fix attempts before halt (default: 2)
|
||||
# --skip-manual Skip manual-only scenarios (default: skip)
|
||||
# --verbose Show detailed output
|
||||
# --dry-run Show what would be executed without running
|
||||
# --timeout=SECONDS Timeout per scenario (default: 30)
|
||||
#
|
||||
# Exit Codes:
|
||||
# 0 - UAT PASS (all automatable scenarios passed)
|
||||
# 1 - UAT FAIL (fixable, retries remain or self-heal succeeded)
|
||||
# 2 - UAT FAIL (max retries exceeded)
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
# =============================================================================
|
||||
# Section 1: Configuration
|
||||
# =============================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
BMAD_DIR="$PROJECT_ROOT/.bmad"
|
||||
|
||||
UAT_DIR="$PROJECT_ROOT/docs/uat"
|
||||
SPRINT_ARTIFACTS_DIR="$PROJECT_ROOT/docs/sprint-artifacts"
|
||||
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||
FIX_DIR="$SPRINT_ARTIFACTS_DIR/uat-fixes"
|
||||
STORIES_DIR="$PROJECT_ROOT/docs/stories"
|
||||
|
||||
LOG_FILE="/tmp/bmad-uat-validate-$$.log"
|
||||
|
||||
# Default configuration
|
||||
UAT_GATE_MODE="quick"
|
||||
MAX_RETRIES=2
|
||||
SKIP_MANUAL=true
|
||||
VERBOSE=false
|
||||
DRY_RUN=false
|
||||
TIMEOUT_SECONDS=30
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
CYAN='\033[0;36m'
|
||||
BOLD='\033[1m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# =============================================================================
|
||||
# Section 2: Helper Functions
|
||||
# =============================================================================
|
||||
|
||||
log() {
|
||||
echo -e "${BLUE}[UAT]${NC} $1"
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo -e "${GREEN}[PASS]${NC} $1"
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [PASS] $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo -e "${RED}[FAIL]${NC} $1"
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [FAIL] $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
log_warn() {
|
||||
echo -e "${YELLOW}[!]${NC} $1"
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
|
||||
}
|
||||
|
||||
log_section() {
|
||||
echo ""
|
||||
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||
echo -e "${BOLD} $1${NC}"
|
||||
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||
}
|
||||
|
||||
log_header() {
|
||||
echo ""
|
||||
echo -e "${CYAN}${BOLD}═══════════════════════════════════════════════════════════${NC}"
|
||||
echo -e "${CYAN}${BOLD} $1${NC}"
|
||||
echo -e "${CYAN}${BOLD}═══════════════════════════════════════════════════════════${NC}"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 3: Argument Parsing
|
||||
# =============================================================================
|
||||
|
||||
EPIC_ID=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--gate-mode=*)
|
||||
UAT_GATE_MODE="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--max-retries=*)
|
||||
MAX_RETRIES="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--skip-manual)
|
||||
SKIP_MANUAL=true
|
||||
shift
|
||||
;;
|
||||
--include-manual)
|
||||
SKIP_MANUAL=false
|
||||
shift
|
||||
;;
|
||||
--verbose)
|
||||
VERBOSE=true
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
shift
|
||||
;;
|
||||
--timeout=*)
|
||||
TIMEOUT_SECONDS="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
-*)
|
||||
echo "Unknown option: $1"
|
||||
exit 1
|
||||
;;
|
||||
*)
|
||||
EPIC_ID="$1"
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "$EPIC_ID" ]; then
|
||||
echo "Usage: $0 <epic-id> [options]"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --gate-mode=MODE Validation mode: quick|full|skip (default: quick)"
|
||||
echo " --max-retries=N Max fix attempts before halt (default: 2)"
|
||||
echo " --skip-manual Skip manual-only scenarios (default)"
|
||||
echo " --include-manual Include manual scenarios in checklist"
|
||||
echo " --verbose Detailed output"
|
||||
echo " --dry-run Show what would be executed"
|
||||
echo " --timeout=SECONDS Timeout per scenario (default: 30)"
|
||||
echo ""
|
||||
echo "Exit Codes:"
|
||||
echo " 0 - UAT PASS"
|
||||
echo " 1 - UAT FAIL (fixable)"
|
||||
echo " 2 - UAT FAIL (max retries exceeded)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate gate mode
|
||||
if [[ ! "$UAT_GATE_MODE" =~ ^(quick|full|skip)$ ]]; then
|
||||
echo "Invalid gate mode: $UAT_GATE_MODE"
|
||||
echo "Valid modes: quick, full, skip"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# Section 4: UAT Document Loading
|
||||
# =============================================================================
|
||||
|
||||
load_uat_document() {
|
||||
local epic_id="$1"
|
||||
|
||||
# Find UAT document (try multiple patterns)
|
||||
UAT_FILE=""
|
||||
for pattern in "epic-${epic_id}-uat.md" "epic-0${epic_id}-uat.md" "${epic_id}-uat.md"; do
|
||||
found=$(find "$UAT_DIR" -name "$pattern" 2>/dev/null | head -1)
|
||||
if [ -n "$found" ]; then
|
||||
UAT_FILE="$found"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -z "$UAT_FILE" ] || [ ! -f "$UAT_FILE" ]; then
|
||||
log_error "UAT document not found for Epic $epic_id"
|
||||
log_error "Searched in: $UAT_DIR"
|
||||
log_error "Expected: epic-${epic_id}-uat.md"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log "Found UAT document: $UAT_FILE"
|
||||
|
||||
# Validate structure - check for scenarios section
|
||||
if ! grep -qE "^##.*[Ss]cenario|^##.*[Tt]est|^##.*[Cc]riteria" "$UAT_FILE"; then
|
||||
log_warn "UAT document may not have standard scenario sections"
|
||||
fi
|
||||
|
||||
# Count scenario blocks (lines starting with ### or numbered items under Test Scenarios)
|
||||
SCENARIO_COUNT=$(grep -cE "^###|^[0-9]+\." "$UAT_FILE" 2>/dev/null || echo "0")
|
||||
log "Found approximately $SCENARIO_COUNT scenario entries"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 5: Scenario Classification
|
||||
# =============================================================================
|
||||
|
||||
# Arrays to store classified scenarios
|
||||
declare -a AUTOMATABLE_SCENARIOS
|
||||
declare -a SEMI_AUTO_SCENARIOS
|
||||
declare -a MANUAL_SCENARIOS
|
||||
|
||||
classify_scenarios() {
|
||||
local uat_file="$1"
|
||||
|
||||
# Reset arrays
|
||||
AUTOMATABLE_SCENARIOS=()
|
||||
SEMI_AUTO_SCENARIOS=()
|
||||
MANUAL_SCENARIOS=()
|
||||
|
||||
# Read the UAT file and extract scenario blocks
|
||||
local current_scenario=""
|
||||
local current_name=""
|
||||
local in_scenario=false
|
||||
local scenario_num=0
|
||||
|
||||
while IFS= read -r line; do
|
||||
# Detect scenario headers (### or numbered items)
|
||||
if [[ "$line" =~ ^###[[:space:]]*(.*) ]] || [[ "$line" =~ ^([0-9]+)\.[[:space:]]+(.*) ]]; then
|
||||
# Save previous scenario if exists
|
||||
if [ -n "$current_scenario" ]; then
|
||||
classify_single_scenario "$scenario_num" "$current_name" "$current_scenario"
|
||||
fi
|
||||
|
||||
# Start new scenario
|
||||
((scenario_num++))
|
||||
if [[ "$line" =~ ^###[[:space:]]*(.*) ]]; then
|
||||
current_name="${BASH_REMATCH[1]}"
|
||||
else
|
||||
current_name="${BASH_REMATCH[2]}"
|
||||
fi
|
||||
current_scenario="$line"
|
||||
in_scenario=true
|
||||
elif [ "$in_scenario" = true ]; then
|
||||
# Continue accumulating scenario content
|
||||
current_scenario+=$'\n'"$line"
|
||||
fi
|
||||
done < "$uat_file"
|
||||
|
||||
# Handle last scenario
|
||||
if [ -n "$current_scenario" ]; then
|
||||
classify_single_scenario "$scenario_num" "$current_name" "$current_scenario"
|
||||
fi
|
||||
|
||||
log "Classification complete:"
|
||||
log " Automatable: ${#AUTOMATABLE_SCENARIOS[@]}"
|
||||
log " Semi-auto: ${#SEMI_AUTO_SCENARIOS[@]}"
|
||||
log " Manual: ${#MANUAL_SCENARIOS[@]}"
|
||||
}
|
||||
|
||||
classify_single_scenario() {
|
||||
local id="$1"
|
||||
local name="$2"
|
||||
local content="$3"
|
||||
|
||||
# Check for automatable indicators
|
||||
if echo "$content" | grep -qiE 'npx|npm run|yarn|node |curl |wget |pytest|jest|vitest|--version|/health|/api/|exit code|returns [0-9]|\.sh |bash '; then
|
||||
# Extract command from code block if present
|
||||
local cmd=""
|
||||
cmd=$(echo "$content" | grep -oE '`[^`]+`' | head -1 | tr -d '`')
|
||||
if [ -z "$cmd" ]; then
|
||||
cmd=$(echo "$content" | grep -oE 'npx [a-zA-Z0-9_-]+.*|npm run [a-zA-Z0-9_:-]+.*|curl [^[:space:]]+.*' | head -1)
|
||||
fi
|
||||
AUTOMATABLE_SCENARIOS+=("$id|$name|$cmd")
|
||||
[ "$VERBOSE" = true ] && log " [AUTO] Scenario $id: $name"
|
||||
|
||||
# Check for semi-automated indicators
|
||||
elif echo "$content" | grep -qiE 'test-send|email|inbox|check your|verify.*manually|setup.*first|start.*server'; then
|
||||
SEMI_AUTO_SCENARIOS+=("$id|$name|")
|
||||
[ "$VERBOSE" = true ] && log " [SEMI] Scenario $id: $name"
|
||||
|
||||
# Everything else is manual
|
||||
else
|
||||
MANUAL_SCENARIOS+=("$id|$name|")
|
||||
[ "$VERBOSE" = true ] && log " [MANUAL] Scenario $id: $name"
|
||||
fi
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 6: Scenario Execution
|
||||
# =============================================================================
|
||||
|
||||
# Arrays to store results
|
||||
declare -a PASSED_SCENARIOS
|
||||
declare -a FAILED_SCENARIOS
|
||||
declare -a FAILED_DETAILS
|
||||
|
||||
execute_scenarios() {
|
||||
local gate_mode="$1"
|
||||
|
||||
# Reset results
|
||||
PASSED_SCENARIOS=()
|
||||
FAILED_SCENARIOS=()
|
||||
FAILED_DETAILS=()
|
||||
|
||||
# Skip mode - pass automatically
|
||||
if [ "$gate_mode" = "skip" ]; then
|
||||
log "Gate mode: skip - bypassing scenario execution"
|
||||
echo "UAT_GATE_RESULT: PASS"
|
||||
echo "UAT_SCENARIOS_PASSED: 0/0 (skipped)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Select scenarios based on gate mode
|
||||
local scenarios_to_run=()
|
||||
if [ "$gate_mode" = "quick" ]; then
|
||||
scenarios_to_run=("${AUTOMATABLE_SCENARIOS[@]}")
|
||||
elif [ "$gate_mode" = "full" ]; then
|
||||
scenarios_to_run=("${AUTOMATABLE_SCENARIOS[@]}" "${SEMI_AUTO_SCENARIOS[@]}")
|
||||
fi
|
||||
|
||||
if [ ${#scenarios_to_run[@]} -eq 0 ]; then
|
||||
log_warn "No automatable scenarios found - gate passes by default"
|
||||
echo "UAT_GATE_RESULT: PASS"
|
||||
echo "UAT_SCENARIOS_PASSED: 0/0 (none automatable)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_section "Executing ${#scenarios_to_run[@]} Scenarios"
|
||||
|
||||
for scenario_entry in "${scenarios_to_run[@]}"; do
|
||||
IFS='|' read -r scenario_id scenario_name scenario_cmd <<< "$scenario_entry"
|
||||
|
||||
execute_single_scenario "$scenario_id" "$scenario_name" "$scenario_cmd"
|
||||
done
|
||||
|
||||
# Report results
|
||||
local total=${#scenarios_to_run[@]}
|
||||
local passed=${#PASSED_SCENARIOS[@]}
|
||||
local failed=${#FAILED_SCENARIOS[@]}
|
||||
|
||||
echo ""
|
||||
log "Results: $passed/$total passed"
|
||||
|
||||
if [ $failed -eq 0 ]; then
|
||||
return 0
|
||||
else
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
execute_single_scenario() {
|
||||
local scenario_id="$1"
|
||||
local scenario_name="$2"
|
||||
local scenario_cmd="$3"
|
||||
|
||||
echo ""
|
||||
log "Scenario $scenario_id: $scenario_name"
|
||||
|
||||
# If no command extracted, try to infer from name
|
||||
if [ -z "$scenario_cmd" ]; then
|
||||
log_warn " No command detected - marking as manual verification needed"
|
||||
FAILED_SCENARIOS+=("$scenario_id")
|
||||
FAILED_DETAILS+=("$scenario_id|$scenario_name|No automatable command found|manual|1")
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [ "$VERBOSE" = true ]; then
|
||||
log " Command: $scenario_cmd"
|
||||
fi
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo " [DRY RUN] Would execute: $scenario_cmd"
|
||||
PASSED_SCENARIOS+=("$scenario_id")
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Execute with timeout
|
||||
local start_time=$(date +%s%N)
|
||||
local output=""
|
||||
local exit_code=0
|
||||
local stderr_file="/tmp/uat-stderr-$$.txt"
|
||||
|
||||
# Run command with timeout
|
||||
set +e
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
output=$(timeout "$TIMEOUT_SECONDS" bash -c "$scenario_cmd" 2>"$stderr_file")
|
||||
exit_code=$?
|
||||
# timeout returns 124 on timeout
|
||||
if [ $exit_code -eq 124 ]; then
|
||||
exit_code=124
|
||||
fi
|
||||
else
|
||||
# macOS fallback using perl
|
||||
output=$(perl -e 'alarm shift @ARGV; exec @ARGV' "$TIMEOUT_SECONDS" bash -c "$scenario_cmd" 2>"$stderr_file")
|
||||
exit_code=$?
|
||||
fi
|
||||
set -e
|
||||
|
||||
local end_time=$(date +%s%N)
|
||||
local duration_ms=$(( (end_time - start_time) / 1000000 ))
|
||||
|
||||
local stderr=""
|
||||
[ -f "$stderr_file" ] && stderr=$(cat "$stderr_file")
|
||||
rm -f "$stderr_file"
|
||||
|
||||
# Evaluate result
|
||||
if [ $exit_code -eq 0 ]; then
|
||||
log_success " Scenario $scenario_id: PASS (${duration_ms}ms)"
|
||||
PASSED_SCENARIOS+=("$scenario_id")
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Scenario $scenario_id PASS: $scenario_cmd" >> "$LOG_FILE"
|
||||
elif [ $exit_code -eq 124 ]; then
|
||||
log_error " Scenario $scenario_id: FAIL (timeout after ${TIMEOUT_SECONDS}s)"
|
||||
FAILED_SCENARIOS+=("$scenario_id")
|
||||
FAILED_DETAILS+=("$scenario_id|$scenario_name|$scenario_cmd|timeout|$exit_code|$output|$stderr")
|
||||
else
|
||||
log_error " Scenario $scenario_id: FAIL (exit code $exit_code)"
|
||||
if [ -n "$stderr" ] && [ "$VERBOSE" = true ]; then
|
||||
echo " Error: $stderr"
|
||||
fi
|
||||
FAILED_SCENARIOS+=("$scenario_id")
|
||||
FAILED_DETAILS+=("$scenario_id|$scenario_name|$scenario_cmd|error|$exit_code|$output|$stderr")
|
||||
fi
|
||||
|
||||
return $exit_code
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 7: Gate Evaluation
|
||||
# =============================================================================
|
||||
|
||||
evaluate_gate() {
|
||||
local total=${#AUTOMATABLE_SCENARIOS[@]}
|
||||
local passed=${#PASSED_SCENARIOS[@]}
|
||||
local failed=${#FAILED_SCENARIOS[@]}
|
||||
|
||||
log_section "Gate Evaluation"
|
||||
|
||||
if [ $failed -eq 0 ]; then
|
||||
log_success "All automatable scenarios passed"
|
||||
return 0
|
||||
else
|
||||
log_error "$failed scenario(s) failed"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 8: Self-Healing Loop
|
||||
# =============================================================================
|
||||
|
||||
generate_fix_context() {
|
||||
local epic_id="$1"
|
||||
local attempt="$2"
|
||||
|
||||
mkdir -p "$FIX_DIR"
|
||||
|
||||
local fix_file="$FIX_DIR/epic-${epic_id}-fix-context-${attempt}.md"
|
||||
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
# Find template
|
||||
local template="$PROJECT_ROOT/src/modules/bmm/workflows/5-validation/uat-validate/uat-fix-context-template.md"
|
||||
|
||||
if [ -f "$template" ]; then
|
||||
# Render template with basic variable substitution
|
||||
sed -e "s/{epic_id}/$epic_id/g" \
|
||||
-e "s/{attempt}/$attempt/g" \
|
||||
-e "s/{timestamp}/$timestamp/g" \
|
||||
-e "s/{max_retries}/$MAX_RETRIES/g" \
|
||||
-e "s/{next_attempt}/$((attempt + 1))/g" \
|
||||
-e "s/{failure_count}/${#FAILED_SCENARIOS[@]}/g" \
|
||||
-e "s|{uat_doc_path}|$UAT_FILE|g" \
|
||||
"$template" > "$fix_file"
|
||||
else
|
||||
# Create minimal fix context without template
|
||||
cat > "$fix_file" << EOF
|
||||
# UAT Fix Context - Epic $epic_id (Attempt $attempt)
|
||||
|
||||
**Generated:** $timestamp
|
||||
**Epic:** $epic_id
|
||||
**Gate Result:** FAIL (${#PASSED_SCENARIOS[@]}/${#AUTOMATABLE_SCENARIOS[@]} scenarios passed)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
This document contains the context needed to fix UAT failures for Epic $epic_id.
|
||||
|
||||
**Failures to fix:** ${#FAILED_SCENARIOS[@]}
|
||||
**Fix attempt:** $attempt of $MAX_RETRIES
|
||||
|
||||
---
|
||||
|
||||
EOF
|
||||
fi
|
||||
|
||||
# Append failed scenarios details
|
||||
echo "" >> "$fix_file"
|
||||
echo "## Failed Scenarios" >> "$fix_file"
|
||||
echo "" >> "$fix_file"
|
||||
|
||||
for detail in "${FAILED_DETAILS[@]}"; do
|
||||
IFS='|' read -r scenario_id scenario_name cmd error_type exit_code output stderr <<< "$detail"
|
||||
|
||||
cat >> "$fix_file" << EOF
|
||||
### Scenario $scenario_id: $scenario_name
|
||||
|
||||
**Command Executed:**
|
||||
\`\`\`bash
|
||||
$cmd
|
||||
\`\`\`
|
||||
|
||||
**Error Type:** $error_type
|
||||
**Exit Code:** $exit_code
|
||||
|
||||
**Output:**
|
||||
\`\`\`
|
||||
$output
|
||||
\`\`\`
|
||||
|
||||
**Error Output:**
|
||||
\`\`\`
|
||||
$stderr
|
||||
\`\`\`
|
||||
|
||||
---
|
||||
|
||||
EOF
|
||||
done
|
||||
|
||||
# Add context references section
|
||||
cat >> "$fix_file" << EOF
|
||||
|
||||
## Context References
|
||||
|
||||
The following files provide additional context for fixing these failures:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| \`$UAT_FILE\` | Full UAT document with all scenarios |
|
||||
| \`$STORIES_DIR/${epic_id}-*\` | Story files with acceptance criteria |
|
||||
| \`$METRICS_DIR/epic-${epic_id}-metrics.yaml\` | Execution metrics |
|
||||
|
||||
## Fix Instructions
|
||||
|
||||
Address the failures above in priority order. For each fix:
|
||||
|
||||
1. **Analyze** - Understand why the scenario failed
|
||||
2. **Locate** - Find the relevant code files
|
||||
3. **Fix** - Implement the minimum change to resolve the failure
|
||||
4. **Verify** - Run the scenario command locally to confirm fix
|
||||
5. **Commit** - Use message format: \`fix(epic-$epic_id): {description}\`
|
||||
|
||||
### Constraints
|
||||
|
||||
- Only fix the identified failures - do not refactor unrelated code
|
||||
- Run the specific failing commands to verify each fix
|
||||
- Run project tests after all fixes: \`npm test\`
|
||||
- If a fix requires changes that would break other scenarios, document the tradeoff
|
||||
|
||||
## After Fixing
|
||||
|
||||
Once all fixes are committed, the UAT validation will automatically re-run.
|
||||
|
||||
- **If all pass:** Epic continues to next phase
|
||||
- **If failures remain:** Another fix context will be generated (attempt $((attempt + 1)))
|
||||
- **If max retries exceeded:** Chain halts for human intervention
|
||||
|
||||
---
|
||||
|
||||
*Generated by UAT Validate Workflow*
|
||||
*BMAD Method - Epic Chain Self-Healing*
|
||||
*Fix Context: epic-${epic_id}-fix-context-${attempt}.md*
|
||||
EOF
|
||||
|
||||
log "Fix context generated: $fix_file"
|
||||
echo "$fix_file"
|
||||
}
|
||||
|
||||
run_quick_dev_fix() {
|
||||
local fix_context_file="$1"
|
||||
local epic_id="$2"
|
||||
local attempt="$3"
|
||||
|
||||
log "Spawning quick-dev fix session (attempt $attempt/$MAX_RETRIES)"
|
||||
|
||||
local fix_prompt="You are Barry, the Quick Flow Solo Dev.
|
||||
|
||||
Load and process this fix context document:
|
||||
$fix_context_file
|
||||
|
||||
Your task:
|
||||
1. Read the failed scenarios and error details from the fix context
|
||||
2. Analyze root cause for each failure
|
||||
3. Implement targeted fixes
|
||||
4. Run the failing commands to verify fixes
|
||||
5. Stage changes: git add -A
|
||||
6. Commit with message: fix(epic-${epic_id}): UAT fix #${attempt}
|
||||
|
||||
Constraints:
|
||||
- Only fix the identified failures
|
||||
- Do not refactor unrelated code
|
||||
- Run tests after fixes
|
||||
|
||||
When done, output exactly:
|
||||
FIX_COMPLETE: {number_fixed}/${#FAILED_SCENARIOS[@]}"
|
||||
|
||||
if [ "$DRY_RUN" = true ]; then
|
||||
echo "[DRY RUN] Would spawn Claude for fixes with prompt:"
|
||||
echo " Fix context: $fix_context_file"
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Execute in isolated context
|
||||
local result
|
||||
result=$(claude --dangerously-skip-permissions -p "$fix_prompt" 2>&1) || true
|
||||
|
||||
echo "$result" >> "$LOG_FILE"
|
||||
|
||||
if echo "$result" | grep -q "FIX_COMPLETE"; then
|
||||
log_success "Quick-dev fix session completed"
|
||||
return 0
|
||||
else
|
||||
log_warn "Quick-dev fix session may not have completed cleanly"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
self_healing_loop() {
|
||||
local epic_id="$1"
|
||||
local attempt=0
|
||||
|
||||
while [ $attempt -lt $MAX_RETRIES ]; do
|
||||
((attempt++))
|
||||
|
||||
log_section "Self-Healing Fix Loop (Attempt $attempt/$MAX_RETRIES)"
|
||||
|
||||
# Generate fix context
|
||||
local fix_file
|
||||
fix_file=$(generate_fix_context "$epic_id" "$attempt")
|
||||
|
||||
# Run quick-dev fix
|
||||
if ! run_quick_dev_fix "$fix_file" "$epic_id" "$attempt"; then
|
||||
log_warn "Fix attempt $attempt may have issues"
|
||||
fi
|
||||
|
||||
# Re-run validation
|
||||
log "Re-validating after fix attempt $attempt..."
|
||||
|
||||
# Reset and re-execute
|
||||
PASSED_SCENARIOS=()
|
||||
FAILED_SCENARIOS=()
|
||||
FAILED_DETAILS=()
|
||||
|
||||
if execute_scenarios "$UAT_GATE_MODE"; then
|
||||
log_success "UAT passed after fix attempt $attempt"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log_warn "UAT still failing after attempt $attempt"
|
||||
done
|
||||
|
||||
log_error "Max retries ($MAX_RETRIES) exceeded"
|
||||
return 2
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Section 9: Output Signals and Metrics
|
||||
# =============================================================================
|
||||
|
||||
update_metrics() {
|
||||
local epic_id="$1"
|
||||
local gate_status="$2"
|
||||
local fix_attempts="$3"
|
||||
|
||||
mkdir -p "$METRICS_DIR"
|
||||
|
||||
local metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
# Check if yq is available for YAML manipulation
|
||||
if command -v yq >/dev/null 2>&1; then
|
||||
if [ -f "$metrics_file" ]; then
|
||||
yq -i ".validation.gate_executed = true" "$metrics_file"
|
||||
yq -i ".validation.gate_status = \"$gate_status\"" "$metrics_file"
|
||||
yq -i ".validation.fix_attempts = $fix_attempts" "$metrics_file"
|
||||
yq -i ".validation.scenarios_passed = ${#PASSED_SCENARIOS[@]}" "$metrics_file"
|
||||
yq -i ".validation.scenarios_failed = ${#FAILED_SCENARIOS[@]}" "$metrics_file"
|
||||
yq -i ".validation.timestamp = \"$timestamp\"" "$metrics_file"
|
||||
else
|
||||
# Create new metrics file
|
||||
cat > "$metrics_file" << EOF
|
||||
epic_id: "$epic_id"
|
||||
validation:
|
||||
gate_executed: true
|
||||
gate_status: "$gate_status"
|
||||
fix_attempts: $fix_attempts
|
||||
scenarios_passed: ${#PASSED_SCENARIOS[@]}
|
||||
scenarios_failed: ${#FAILED_SCENARIOS[@]}
|
||||
timestamp: "$timestamp"
|
||||
EOF
|
||||
fi
|
||||
else
|
||||
# Fallback: append to file or create new
|
||||
if [ ! -f "$metrics_file" ]; then
|
||||
cat > "$metrics_file" << EOF
|
||||
epic_id: "$epic_id"
|
||||
validation:
|
||||
gate_executed: true
|
||||
gate_status: "$gate_status"
|
||||
fix_attempts: $fix_attempts
|
||||
scenarios_passed: ${#PASSED_SCENARIOS[@]}
|
||||
scenarios_failed: ${#FAILED_SCENARIOS[@]}
|
||||
timestamp: "$timestamp"
|
||||
EOF
|
||||
else
|
||||
# Simple append for validation section
|
||||
log_warn "yq not found - metrics update may be incomplete"
|
||||
fi
|
||||
fi
|
||||
|
||||
log "Metrics updated: $metrics_file"
|
||||
}
|
||||
|
||||
output_signals() {
|
||||
local gate_status="$1"
|
||||
local fix_attempts="$2"
|
||||
|
||||
local total=${#AUTOMATABLE_SCENARIOS[@]}
|
||||
local passed=${#PASSED_SCENARIOS[@]}
|
||||
|
||||
echo ""
|
||||
echo "UAT_GATE_RESULT: $gate_status"
|
||||
echo "UAT_FIX_ATTEMPTS: $fix_attempts"
|
||||
echo "UAT_SCENARIOS_PASSED: $passed/$total"
|
||||
}
|
||||
|
||||
print_summary() {
|
||||
local gate_status="$1"
|
||||
local fix_attempts="$2"
|
||||
|
||||
log_header "UAT VALIDATION COMPLETE"
|
||||
|
||||
echo " Epic: $EPIC_ID"
|
||||
echo " Gate Mode: $UAT_GATE_MODE"
|
||||
echo " Gate Result: $gate_status"
|
||||
echo ""
|
||||
echo " Scenarios:"
|
||||
echo " Automatable: ${#AUTOMATABLE_SCENARIOS[@]}"
|
||||
echo " Semi-automated: ${#SEMI_AUTO_SCENARIOS[@]}"
|
||||
echo " Manual: ${#MANUAL_SCENARIOS[@]}"
|
||||
echo ""
|
||||
echo " Results:"
|
||||
echo " Passed: ${#PASSED_SCENARIOS[@]}"
|
||||
echo " Failed: ${#FAILED_SCENARIOS[@]}"
|
||||
echo " Fix Attempts: $fix_attempts"
|
||||
echo ""
|
||||
echo " Artifacts:"
|
||||
echo " Log: $LOG_FILE"
|
||||
echo " UAT Document: $UAT_FILE"
|
||||
if [ ${#FAILED_SCENARIOS[@]} -gt 0 ] && [ -d "$FIX_DIR" ]; then
|
||||
echo " Fix Contexts: $FIX_DIR/"
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
# Main Execution
|
||||
# =============================================================================
|
||||
|
||||
log_header "UAT VALIDATION: Epic $EPIC_ID"
|
||||
log "Gate mode: $UAT_GATE_MODE"
|
||||
log "Max retries: $MAX_RETRIES"
|
||||
log "Timeout: ${TIMEOUT_SECONDS}s"
|
||||
|
||||
# Ensure directories exist
|
||||
mkdir -p "$METRICS_DIR"
|
||||
mkdir -p "$FIX_DIR"
|
||||
|
||||
# Step 1: Load UAT document
|
||||
log_section "Loading UAT Document"
|
||||
if ! load_uat_document "$EPIC_ID"; then
|
||||
echo "UAT_GATE_RESULT: FAIL"
|
||||
echo "UAT_FIX_ATTEMPTS: 0"
|
||||
echo "UAT_SCENARIOS_PASSED: 0/0"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Step 2: Classify scenarios
|
||||
log_section "Classifying Scenarios"
|
||||
classify_scenarios "$UAT_FILE"
|
||||
|
||||
# Step 3: Execute scenarios
|
||||
if ! execute_scenarios "$UAT_GATE_MODE"; then
|
||||
# Gate failed - check if we should try self-healing
|
||||
if [ "$DRY_RUN" = false ] && [ $MAX_RETRIES -gt 0 ]; then
|
||||
if ! self_healing_loop "$EPIC_ID"; then
|
||||
# Max retries exceeded
|
||||
update_metrics "$EPIC_ID" "FAIL" "$MAX_RETRIES"
|
||||
output_signals "FAIL" "$MAX_RETRIES"
|
||||
print_summary "FAIL" "$MAX_RETRIES"
|
||||
exit 2
|
||||
fi
|
||||
else
|
||||
# No self-healing or dry-run
|
||||
update_metrics "$EPIC_ID" "FAIL" "0"
|
||||
output_signals "FAIL" "0"
|
||||
print_summary "FAIL" "0"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Step 4: Gate passed
|
||||
FINAL_ATTEMPTS=0
|
||||
if [ ${#FAILED_SCENARIOS[@]} -gt 0 ]; then
|
||||
# Passed after retries
|
||||
FINAL_ATTEMPTS=$((MAX_RETRIES - $(ls -1 "$FIX_DIR"/epic-${EPIC_ID}-fix-context-*.md 2>/dev/null | wc -l) + 1))
|
||||
fi
|
||||
|
||||
update_metrics "$EPIC_ID" "PASS" "$FINAL_ATTEMPTS"
|
||||
output_signals "PASS" "$FINAL_ATTEMPTS"
|
||||
print_summary "PASS" "$FINAL_ATTEMPTS"
|
||||
|
||||
log_success "UAT validation passed for Epic $EPIC_ID"
|
||||
exit 0
|
||||
|
|
@ -9,10 +9,10 @@ epic_file: "{epic_file_path}"
|
|||
|
||||
# Execution timing
|
||||
execution:
|
||||
start_time: "{start_timestamp}" # ISO 8601 format
|
||||
end_time: "{end_timestamp}" # ISO 8601 format
|
||||
duration_seconds: 0 # Calculated from start/end
|
||||
duration_formatted: "" # Human readable, e.g., "1.5 hours"
|
||||
start_time: "{start_timestamp}" # ISO 8601 format
|
||||
end_time: "{end_timestamp}" # ISO 8601 format
|
||||
duration_seconds: 0 # Calculated from start/end
|
||||
duration_formatted: "" # Human readable, e.g., "1.5 hours"
|
||||
|
||||
# Story counts
|
||||
stories:
|
||||
|
|
@ -44,13 +44,13 @@ uat:
|
|||
# Validation gate results
|
||||
validation:
|
||||
gate_executed: false
|
||||
gate_mode: "quick" # quick | full | skip
|
||||
gate_mode: "quick" # quick | full | skip
|
||||
timestamp: ""
|
||||
results:
|
||||
passed: 0
|
||||
failed: 0
|
||||
skipped: 0
|
||||
gate_status: "PENDING" # PASS | FAIL | PENDING | SKIPPED
|
||||
gate_status: "PENDING" # PASS | FAIL | PENDING | SKIPPED
|
||||
blocking_issues: []
|
||||
|
||||
# Self-healing fix loop tracking
|
||||
|
|
@ -76,19 +76,19 @@ issues: []
|
|||
|
||||
# Dependencies (from chain plan)
|
||||
dependencies:
|
||||
requires: [] # Epic IDs this epic depends on
|
||||
enables: [] # Epic IDs that depend on this epic
|
||||
requires: [] # Epic IDs this epic depends on
|
||||
enables: [] # Epic IDs that depend on this epic
|
||||
|
||||
# Git information
|
||||
git:
|
||||
commits: 0 # Number of commits for this epic
|
||||
branch: "" # Branch used (if feature branch)
|
||||
first_commit: "" # SHA of first commit
|
||||
last_commit: "" # SHA of last commit
|
||||
commits: 0 # Number of commits for this epic
|
||||
branch: "" # Branch used (if feature branch)
|
||||
first_commit: "" # SHA of first commit
|
||||
last_commit: "" # SHA of last commit
|
||||
|
||||
# Estimated token usage
|
||||
tokens:
|
||||
estimated_calls: 0 # stories * 2 (dev + review)
|
||||
estimated_input: 0 # estimated_calls * 8000
|
||||
estimated_output: 0 # estimated_calls * 4000
|
||||
estimated_total: 0 # input + output
|
||||
estimated_calls: 0 # stories * 2 (dev + review)
|
||||
estimated_input: 0 # estimated_calls * 8000
|
||||
estimated_output: 0 # estimated_calls * 4000
|
||||
estimated_total: 0 # input + output
|
||||
|
|
|
|||
|
|
@ -7,13 +7,13 @@
|
|||
execution:
|
||||
# Automatically commit after each story completes
|
||||
auto_commit: true
|
||||
|
||||
|
||||
# Run tests before transitioning to review phase
|
||||
run_tests_before_review: true
|
||||
|
||||
|
||||
# Maximum retries for failed phases
|
||||
max_retries: 2
|
||||
|
||||
|
||||
# Timeout per phase in seconds (0 = no timeout)
|
||||
phase_timeout: 0
|
||||
|
||||
|
|
@ -24,21 +24,21 @@ review:
|
|||
# standard - Block on acceptance criteria, security, major quality
|
||||
# strict - Block on any quality issue, no auto-fixing
|
||||
mode: standard
|
||||
|
||||
|
||||
# Allow reviewer to auto-fix issues
|
||||
auto_fix_enabled: true
|
||||
|
||||
|
||||
# Require all tests pass before approval
|
||||
require_passing_tests: true
|
||||
|
||||
|
||||
# Issue fix policy thresholds
|
||||
fix_policy:
|
||||
# Always fix HIGH severity issues
|
||||
always_fix_high: true
|
||||
|
||||
|
||||
# Fix MEDIUM severity if total issues exceed this threshold
|
||||
medium_fix_threshold: 5
|
||||
|
||||
|
||||
# Never auto-fix LOW severity (just document)
|
||||
fix_low: false
|
||||
|
||||
|
|
@ -46,13 +46,13 @@ review:
|
|||
context:
|
||||
# Clear context between phases (recommended)
|
||||
isolate_phases: true
|
||||
|
||||
|
||||
# Include architecture doc in dev context
|
||||
include_architecture: true
|
||||
|
||||
|
||||
# Include PRD in dev context
|
||||
include_prd: false
|
||||
|
||||
|
||||
# Max lines of code context to include
|
||||
max_code_context: 500
|
||||
|
||||
|
|
@ -60,16 +60,16 @@ context:
|
|||
uat:
|
||||
# Generate UAT after all stories complete
|
||||
enabled: true
|
||||
|
||||
|
||||
# UAT output location (relative to project root)
|
||||
output_dir: docs/uat
|
||||
|
||||
|
||||
# Include edge case scenarios
|
||||
include_edge_cases: true
|
||||
|
||||
|
||||
# Minimum scenarios to generate
|
||||
min_scenarios: 3
|
||||
|
||||
|
||||
# Maximum scenarios to generate
|
||||
max_scenarios: 10
|
||||
|
||||
|
|
@ -77,13 +77,13 @@ uat:
|
|||
git:
|
||||
# Commit message prefix
|
||||
commit_prefix: "feat"
|
||||
|
||||
|
||||
# Include epic ID in commit message
|
||||
include_epic_in_commit: true
|
||||
|
||||
|
||||
# Push after each commit
|
||||
auto_push: false
|
||||
|
||||
|
||||
# Branch naming pattern (use {epic_id} and {story_id} as placeholders)
|
||||
branch_pattern: "feature/epic-{epic_id}"
|
||||
|
||||
|
|
@ -91,10 +91,10 @@ git:
|
|||
parallel:
|
||||
# Enable parallel story execution
|
||||
enabled: false
|
||||
|
||||
|
||||
# Maximum concurrent stories
|
||||
max_concurrent: 3
|
||||
|
||||
|
||||
# Only parallelize independent stories (no dependencies)
|
||||
respect_dependencies: true
|
||||
|
||||
|
|
@ -102,13 +102,13 @@ parallel:
|
|||
logging:
|
||||
# Log level: debug | info | warn | error
|
||||
level: info
|
||||
|
||||
|
||||
# Save execution log
|
||||
save_log: true
|
||||
|
||||
|
||||
# Log output location
|
||||
log_dir: docs/sprints
|
||||
|
||||
|
||||
# Include Claude responses in log (verbose)
|
||||
log_responses: false
|
||||
|
||||
|
|
@ -116,9 +116,9 @@ logging:
|
|||
notifications:
|
||||
# Notify on completion
|
||||
on_complete: false
|
||||
|
||||
|
||||
# Notify on failure
|
||||
on_failure: false
|
||||
|
||||
|
||||
# Notification method: slack | email | none
|
||||
method: none
|
||||
|
|
|
|||
|
|
@ -124,8 +124,8 @@ This epic is **successful** when a user can:
|
|||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Scenarios Tested | __ / __ |
|
||||
| Scenarios Passed | __ / __ |
|
||||
| Scenarios Tested | \_\_ / \_\_ |
|
||||
| Scenarios Passed | \_\_ / \_\_ |
|
||||
| Critical Issues | |
|
||||
| Major Issues | |
|
||||
| Minor Issues | |
|
||||
|
|
|
|||
|
|
@ -0,0 +1,102 @@
|
|||
# Step 1: Load UAT Document
|
||||
|
||||
## Purpose
|
||||
|
||||
Load and validate the UAT document for the specified epic, extracting all test scenarios for classification and execution.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| epic_id | CLI argument | Yes |
|
||||
| uat_dir | Configuration | Yes (default: `docs/uat`) |
|
||||
|
||||
## Process
|
||||
|
||||
### 1.1 Locate UAT Document
|
||||
|
||||
Search for UAT document using these patterns in order:
|
||||
1. `{uat_dir}/epic-{epic_id}-uat.md`
|
||||
2. `{uat_dir}/epic-0{epic_id}-uat.md` (zero-padded)
|
||||
3. `{uat_dir}/{epic_id}-uat.md`
|
||||
|
||||
**If not found:** Exit with error code 1 and message:
|
||||
```
|
||||
UAT document not found for Epic {epic_id}
|
||||
Searched in: {uat_dir}
|
||||
Expected: epic-{epic_id}-uat.md
|
||||
```
|
||||
|
||||
### 1.2 Validate Document Structure
|
||||
|
||||
Confirm document contains at least one of these sections:
|
||||
- `## Test Scenarios`
|
||||
- `## Acceptance Criteria`
|
||||
- `## Scenarios`
|
||||
- `## Success Criteria`
|
||||
|
||||
**Warning if missing:** Log warning but continue (scenarios may be inline)
|
||||
|
||||
### 1.3 Parse Scenarios
|
||||
|
||||
Extract scenario blocks by detecting:
|
||||
- Headers starting with `###` (individual scenario titles)
|
||||
- Numbered items `1.`, `2.`, etc. under scenario sections
|
||||
- Checkbox items `- [ ]` in criteria sections
|
||||
|
||||
For each scenario, extract:
|
||||
- **Scenario ID**: Numeric index or explicit ID
|
||||
- **Scenario Name**: Title text
|
||||
- **Steps**: Given/When/Then or numbered steps
|
||||
- **Verification Command**: Code block or CLI reference (if present)
|
||||
- **Expected Result**: Success criteria text
|
||||
|
||||
### 1.4 Build Scenario List
|
||||
|
||||
Create structured list of scenarios:
|
||||
|
||||
```yaml
|
||||
scenarios:
|
||||
- id: 1
|
||||
name: "Project Initialization"
|
||||
steps:
|
||||
- "Run npx heimdall init"
|
||||
- "Verify config file created"
|
||||
verification_command: "npx heimdall --version"
|
||||
expected_result: "displays a version number"
|
||||
raw_content: |
|
||||
### 1. Project Initialization
|
||||
...
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| scenario_list | Memory/State | Array of parsed scenario objects |
|
||||
| scenario_count | Console | Total number of scenarios found |
|
||||
| uat_file_path | State | Path to loaded UAT document |
|
||||
|
||||
## Completion Signal
|
||||
|
||||
```
|
||||
UAT_LOADED: {scenario_count} scenarios from {uat_file_path}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| File not found | Exit 1 with clear error message |
|
||||
| Empty file | Exit 1 with "UAT document is empty" |
|
||||
| No scenarios detected | Log warning, return empty list (gate passes by default) |
|
||||
| Parse error | Log warning for specific section, continue with partial results |
|
||||
|
||||
## Example Output
|
||||
|
||||
```
|
||||
[UAT] Loading UAT document for Epic 1
|
||||
[UAT] Found: docs/uat/epic-1-uat.md
|
||||
[UAT] Parsed 9 scenarios
|
||||
UAT_LOADED: 9 scenarios from docs/uat/epic-1-uat.md
|
||||
```
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
# Step 2: Classify Scenarios
|
||||
|
||||
## Purpose
|
||||
|
||||
Categorize each scenario by its executability level to determine which can be automated, which need partial automation, and which require manual verification.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| scenario_list | Step 1 | Yes |
|
||||
|
||||
## Process
|
||||
|
||||
### 2.1 Classification Categories
|
||||
|
||||
| Classification | Description | Gate Behavior |
|
||||
|----------------|-------------|---------------|
|
||||
| **Automatable** | Can be fully executed via shell command | Execute and verify |
|
||||
| **Semi-automated** | Requires setup, then automated verification | Execute with warning |
|
||||
| **Manual** | Requires human interaction or visual verification | Skip, add to checklist |
|
||||
|
||||
### 2.2 Detect Automatable Scenarios
|
||||
|
||||
Check scenario content for these indicators (case-insensitive):
|
||||
|
||||
**CLI/Command indicators:**
|
||||
- `npx`, `npm run`, `yarn`, `pnpm`
|
||||
- `node `, `python `, `ruby `
|
||||
- `curl `, `wget `, `http`
|
||||
- `bash `, `sh `, `./`
|
||||
|
||||
**Test framework indicators:**
|
||||
- `pytest`, `jest`, `vitest`, `mocha`
|
||||
- `npm test`, `yarn test`
|
||||
|
||||
**Verification indicators:**
|
||||
- `--version`, `--help`
|
||||
- `/health`, `/api/`, `/status`
|
||||
- `exit code`, `returns 0`, `returns 1`
|
||||
- `outputs`, `prints`, `displays`
|
||||
|
||||
**Database/Config indicators:**
|
||||
- `db migrate`, `db status`
|
||||
- `config validate`, `config check`
|
||||
|
||||
### 2.3 Detect Semi-Automated Scenarios
|
||||
|
||||
Scenarios with commands that require prior setup:
|
||||
|
||||
**Setup-required indicators:**
|
||||
- "Start the server first"
|
||||
- "Ensure database is running"
|
||||
- "In a separate terminal"
|
||||
- "After deploying"
|
||||
|
||||
**Partial automation indicators:**
|
||||
- `test-send`, `send test`
|
||||
- "check your email/inbox"
|
||||
- "verify in browser"
|
||||
- Manual setup + automated verification
|
||||
|
||||
### 2.4 Classify as Manual
|
||||
|
||||
Scenarios without detectable automation path:
|
||||
|
||||
**Manual-only indicators:**
|
||||
- "Railway dashboard", "Vercel dashboard"
|
||||
- "Open browser", "Navigate to"
|
||||
- "Visual inspection", "Visually verify"
|
||||
- "Two terminals", "Side by side"
|
||||
- "User should see", "Observe that"
|
||||
- No code blocks or CLI references
|
||||
|
||||
### 2.5 Extract Commands
|
||||
|
||||
For automatable/semi-automated scenarios, extract the verification command:
|
||||
|
||||
1. Look for inline code: `` `command here` ``
|
||||
2. Look for code blocks: ```bash ... ```
|
||||
3. Look for CLI patterns: `npx ...`, `npm run ...`, `curl ...`
|
||||
4. Look for expected patterns after "Run:" or "Execute:"
|
||||
|
||||
### 2.6 Build Classification Result
|
||||
|
||||
```yaml
|
||||
classification:
|
||||
total: 9
|
||||
automatable: 6
|
||||
semi_automated: 2
|
||||
manual: 1
|
||||
|
||||
automatable_scenarios:
|
||||
- id: 1
|
||||
name: "Project Initialization"
|
||||
command: "npx heimdall --version"
|
||||
expected: "displays a version number"
|
||||
|
||||
- id: 3
|
||||
name: "Database Migration"
|
||||
command: "npx heimdall db migrate"
|
||||
expected: "success message"
|
||||
|
||||
semi_automated_scenarios:
|
||||
- id: 7
|
||||
name: "Email Notification"
|
||||
command: "curl -X POST localhost:3000/test-send"
|
||||
expected: "email received"
|
||||
note: "Requires manual inbox verification"
|
||||
|
||||
manual_scenarios:
|
||||
- id: 9
|
||||
name: "Dashboard Visual Check"
|
||||
note: "Requires browser inspection"
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| automatable | Array | Scenarios to execute automatically |
|
||||
| semi_automated | Array | Scenarios needing setup + execution |
|
||||
| manual | Array | Scenarios requiring human verification |
|
||||
| classification_summary | Console | Counts per category |
|
||||
|
||||
## Completion Signal
|
||||
|
||||
```
|
||||
SCENARIOS_CLASSIFIED: {automatable}/{semi_automated}/{manual}
|
||||
```
|
||||
|
||||
## Example Output
|
||||
|
||||
```
|
||||
[UAT] Classifying 9 scenarios...
|
||||
[AUTO] Scenario 1: Project Initialization
|
||||
[AUTO] Scenario 2: Configuration Setup
|
||||
[AUTO] Scenario 3: Database Migration
|
||||
[AUTO] Scenario 4: Connection Validation
|
||||
[AUTO] Scenario 5: Worker Process Startup
|
||||
[AUTO] Scenario 6: Job Queue Testing
|
||||
[SEMI] Scenario 7: Email Notification
|
||||
[SEMI] Scenario 8: Webhook Delivery
|
||||
[MANUAL] Scenario 9: Dashboard Visual Check
|
||||
|
||||
SCENARIOS_CLASSIFIED: 6/2/1
|
||||
```
|
||||
|
||||
## Classification Confidence
|
||||
|
||||
For edge cases, use this priority:
|
||||
1. If command is clearly present → Automatable
|
||||
2. If command present but requires setup → Semi-automated
|
||||
3. If no command detected → Manual
|
||||
|
||||
When in doubt, classify as semi-automated (attempts execution, flags for review).
|
||||
|
|
@ -0,0 +1,202 @@
|
|||
# Step 3: Execute Scenarios
|
||||
|
||||
## Purpose
|
||||
|
||||
Run automatable scenarios via shell commands, capture results, and determine pass/fail status for each.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| automatable | Step 2 | Yes |
|
||||
| semi_automated | Step 2 | For full gate mode |
|
||||
| gate_mode | CLI | Yes (quick/full/skip) |
|
||||
| timeout | Config | No (default: 30s) |
|
||||
|
||||
## Process
|
||||
|
||||
### 3.1 Filter by Gate Mode
|
||||
|
||||
| Mode | Scenarios Executed |
|
||||
|------|-------------------|
|
||||
| `quick` | Automatable only |
|
||||
| `full` | Automatable + Semi-automated |
|
||||
| `skip` | None (gate passes automatically) |
|
||||
|
||||
**Skip mode behavior:**
|
||||
```
|
||||
[UAT] Gate mode: skip - bypassing scenario execution
|
||||
UAT_GATE_RESULT: PASS
|
||||
UAT_SCENARIOS_PASSED: 0/0 (skipped)
|
||||
```
|
||||
|
||||
### 3.2 Handle Empty Scenario List
|
||||
|
||||
If no scenarios to execute:
|
||||
```
|
||||
[UAT] No automatable scenarios found - gate passes by default
|
||||
UAT_GATE_RESULT: PASS
|
||||
UAT_SCENARIOS_PASSED: 0/0 (none automatable)
|
||||
```
|
||||
|
||||
### 3.3 Execute Each Scenario
|
||||
|
||||
For each scenario in the execution list:
|
||||
|
||||
#### 3.3.1 Extract Command
|
||||
|
||||
```bash
|
||||
# From scenario object
|
||||
command="${scenario.command}"
|
||||
|
||||
# If no command extracted, mark as failed
|
||||
if [ -z "$command" ]; then
|
||||
status="FAIL"
|
||||
error="No automatable command found"
|
||||
fi
|
||||
```
|
||||
|
||||
#### 3.3.2 Execute with Timeout
|
||||
|
||||
```bash
|
||||
# Using GNU timeout
|
||||
output=$(timeout $TIMEOUT_SECONDS bash -c "$command" 2>&1)
|
||||
exit_code=$?
|
||||
|
||||
# macOS fallback (perl-based timeout)
|
||||
output=$(perl -e 'alarm shift @ARGV; exec @ARGV' $TIMEOUT_SECONDS bash -c "$command" 2>&1)
|
||||
exit_code=$?
|
||||
```
|
||||
|
||||
#### 3.3.3 Capture Results
|
||||
|
||||
Record for each execution:
|
||||
- **stdout**: Command output
|
||||
- **stderr**: Error output (captured via 2>&1)
|
||||
- **exit_code**: Process exit code
|
||||
- **duration_ms**: Execution time in milliseconds
|
||||
- **status**: PASS or FAIL
|
||||
|
||||
### 3.4 Result Evaluation Rules
|
||||
|
||||
| Condition | Result | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Exit code 0 | PASS | Command succeeded |
|
||||
| Exit code 124 | FAIL (timeout) | Exceeded timeout limit |
|
||||
| Exit code 127 | FAIL (not found) | Command not found |
|
||||
| Exit code non-zero | FAIL (error) | Command failed |
|
||||
|
||||
### 3.5 Expected Output Matching (Optional)
|
||||
|
||||
If scenario has `expected_result`, apply flexible matching:
|
||||
|
||||
**Contains match** (default):
|
||||
```bash
|
||||
if echo "$output" | grep -qi "$expected"; then
|
||||
match="true"
|
||||
fi
|
||||
```
|
||||
|
||||
**Exit code only match:**
|
||||
```bash
|
||||
# If expected contains "exit 0" or "returns 0"
|
||||
if [ $exit_code -eq 0 ]; then
|
||||
match="true"
|
||||
fi
|
||||
```
|
||||
|
||||
**Note:** Output matching is advisory. Primary pass/fail is based on exit code.
|
||||
|
||||
### 3.6 Record Execution Results
|
||||
|
||||
```yaml
|
||||
execution_results:
|
||||
- scenario_id: 1
|
||||
name: "Project Initialization"
|
||||
command: "npx heimdall --version"
|
||||
status: "PASS"
|
||||
exit_code: 0
|
||||
output: "1.0.0"
|
||||
duration_ms: 1250
|
||||
|
||||
- scenario_id: 3
|
||||
name: "Database Migration"
|
||||
command: "npx heimdall db migrate"
|
||||
status: "FAIL"
|
||||
exit_code: 1
|
||||
output: ""
|
||||
stderr: "Error: relation 'events' does not exist"
|
||||
duration_ms: 3400
|
||||
```
|
||||
|
||||
### 3.7 Progress Output
|
||||
|
||||
During execution, output progress:
|
||||
|
||||
```
|
||||
[UAT] Executing 6 scenarios...
|
||||
|
||||
Scenario 1: Project Initialization
|
||||
Command: npx heimdall --version
|
||||
[PASS] (1250ms)
|
||||
|
||||
Scenario 2: Configuration Setup
|
||||
Command: npx heimdall config validate
|
||||
[PASS] (890ms)
|
||||
|
||||
Scenario 3: Database Migration
|
||||
Command: npx heimdall db migrate
|
||||
[FAIL] (3400ms)
|
||||
Error: relation 'events' does not exist
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| results | Array | Per-scenario execution results |
|
||||
| passed_count | State | Number of passed scenarios |
|
||||
| failed_count | State | Number of failed scenarios |
|
||||
| failed_details | Array | Details for failed scenarios |
|
||||
|
||||
## Completion Signal
|
||||
|
||||
```
|
||||
SCENARIOS_EXECUTED: {passed}/{total}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| Command not found | Record as FAIL with "command not found" message |
|
||||
| Timeout exceeded | Record as FAIL with "timeout after Xs" message |
|
||||
| Permission denied | Record as FAIL with permission error |
|
||||
| Shell syntax error | Record as FAIL with syntax error details |
|
||||
|
||||
## Example Complete Output
|
||||
|
||||
```
|
||||
[UAT] Executing 6 scenarios (gate mode: quick)
|
||||
|
||||
Scenario 1: Project Initialization
|
||||
[PASS] npx heimdall --version (1250ms)
|
||||
|
||||
Scenario 2: Configuration Setup
|
||||
[PASS] npx heimdall config validate (890ms)
|
||||
|
||||
Scenario 3: Database Migration
|
||||
[FAIL] npx heimdall db migrate (3400ms)
|
||||
Error: relation 'events' does not exist
|
||||
|
||||
Scenario 4: Connection Validation
|
||||
[PASS] npx heimdall db status (450ms)
|
||||
|
||||
Scenario 5: Worker Process Startup
|
||||
[PASS] npx heimdall worker --check (2100ms)
|
||||
|
||||
Scenario 6: Job Queue Testing
|
||||
[PASS] npx heimdall queue test (1800ms)
|
||||
|
||||
SCENARIOS_EXECUTED: 5/6
|
||||
```
|
||||
|
|
@ -0,0 +1,237 @@
|
|||
# Step 4: Evaluate Gate
|
||||
|
||||
## Purpose
|
||||
|
||||
Determine overall pass/fail status based on execution results. If failed and self-healing is enabled, generate fix context and trigger the quick-dev fix loop.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| results | Step 3 | Yes |
|
||||
| passed_count | Step 3 | Yes |
|
||||
| failed_count | Step 3 | Yes |
|
||||
| failed_details | Step 3 | Yes |
|
||||
| max_retries | CLI | Yes (default: 2) |
|
||||
| current_attempt | State | Yes (starts at 0) |
|
||||
| self_heal_enabled | Config | Yes (default: true) |
|
||||
|
||||
## Process
|
||||
|
||||
### 4.1 Check All Results
|
||||
|
||||
```
|
||||
if failed_count == 0:
|
||||
gate_status = PASS
|
||||
→ Skip to Step 5 (Report Results)
|
||||
else:
|
||||
gate_status = FAIL
|
||||
→ Continue to 4.2
|
||||
```
|
||||
|
||||
### 4.2 Evaluate Retry Eligibility
|
||||
|
||||
```
|
||||
if current_attempt >= max_retries:
|
||||
→ Max retries exceeded, halt
|
||||
exit_code = 2
|
||||
|
||||
if self_heal_enabled == false:
|
||||
→ Self-healing disabled, report failure
|
||||
exit_code = 1
|
||||
|
||||
→ Continue to 4.3 (Generate Fix Context)
|
||||
```
|
||||
|
||||
### 4.3 Generate Fix Context Document
|
||||
|
||||
Create fix context document at:
|
||||
```
|
||||
{sprint_artifacts}/uat-fixes/epic-{epic_id}-fix-context-{attempt}.md
|
||||
```
|
||||
|
||||
#### 4.3.1 Load Template
|
||||
|
||||
Load from: `{workflow_path}/uat-fix-context-template.md`
|
||||
|
||||
If template not found, use inline default structure.
|
||||
|
||||
#### 4.3.2 Populate Template Variables
|
||||
|
||||
| Variable | Value |
|
||||
|----------|-------|
|
||||
| `{epic_id}` | Epic ID being validated |
|
||||
| `{attempt}` | Current fix attempt number |
|
||||
| `{timestamp}` | ISO 8601 timestamp |
|
||||
| `{max_retries}` | Maximum retry limit |
|
||||
| `{next_attempt}` | attempt + 1 |
|
||||
| `{failure_count}` | Number of failed scenarios |
|
||||
| `{passed}` | Number of passed scenarios |
|
||||
| `{total}` | Total scenarios executed |
|
||||
|
||||
#### 4.3.3 Append Failed Scenario Details
|
||||
|
||||
For each failed scenario:
|
||||
|
||||
```markdown
|
||||
### Scenario {id}: {name}
|
||||
|
||||
**Command Executed:**
|
||||
```bash
|
||||
{command}
|
||||
```
|
||||
|
||||
**Expected Result:**
|
||||
{expected_result}
|
||||
|
||||
**Actual Result:**
|
||||
```
|
||||
{actual_output}
|
||||
```
|
||||
|
||||
**Error Output:**
|
||||
```
|
||||
{stderr}
|
||||
```
|
||||
|
||||
**Exit Code:** {exit_code}
|
||||
|
||||
**Related Story:** {story_id} (if determinable)
|
||||
|
||||
**Root Cause Hint:**
|
||||
{analyze_error_for_hints}
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
#### 4.3.4 Add Fix Instructions
|
||||
|
||||
```markdown
|
||||
## Fix Instructions
|
||||
|
||||
Address the failures above in priority order. For each fix:
|
||||
|
||||
1. **Analyze** - Understand why the scenario failed
|
||||
2. **Locate** - Find the relevant code files
|
||||
3. **Fix** - Implement the minimum change to resolve
|
||||
4. **Verify** - Run the scenario command locally
|
||||
5. **Commit** - Use: `fix(epic-{epic_id}): {description}`
|
||||
|
||||
### Constraints
|
||||
|
||||
- Only fix identified failures
|
||||
- Do not refactor unrelated code
|
||||
- Run tests after fixes: `npm test`
|
||||
```
|
||||
|
||||
### 4.4 Trigger Quick-Dev Fix
|
||||
|
||||
Spawn a fresh Claude session for fixes:
|
||||
|
||||
```bash
|
||||
fix_prompt="You are Barry, the Quick Flow Solo Dev.
|
||||
|
||||
Load and process this fix context document:
|
||||
{fix_context_file}
|
||||
|
||||
Your task:
|
||||
1. Read the failed scenarios and error details
|
||||
2. Analyze root cause for each failure
|
||||
3. Implement targeted fixes
|
||||
4. Run the failing commands to verify fixes
|
||||
5. Stage changes: git add -A
|
||||
6. Commit: fix(epic-{epic_id}): UAT fix #{attempt}
|
||||
|
||||
Constraints:
|
||||
- Only fix identified failures
|
||||
- Do not refactor unrelated code
|
||||
- Run tests after fixes
|
||||
|
||||
When done, output:
|
||||
FIX_COMPLETE: {fixed_count}/{failure_count}"
|
||||
|
||||
claude --dangerously-skip-permissions -p "$fix_prompt"
|
||||
```
|
||||
|
||||
### 4.5 Increment and Re-validate
|
||||
|
||||
After fix session completes:
|
||||
|
||||
1. Increment `current_attempt`
|
||||
2. Return to Step 3 (Execute Scenarios)
|
||||
3. Re-run validation with fresh results
|
||||
|
||||
```
|
||||
current_attempt += 1
|
||||
→ Go to Step 3: Execute Scenarios
|
||||
→ Then return to Step 4: Evaluate Gate
|
||||
```
|
||||
|
||||
### 4.6 Gate Decision Flowchart
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Execute Results │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────┐
|
||||
│All Pass?│──Yes──▶ GATE_PASS (exit 0)
|
||||
└────┬────┘
|
||||
│No
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Retries Left?│──No──▶ GATE_FAIL (exit 2)
|
||||
└──────┬───────┘ Max retries exceeded
|
||||
│Yes
|
||||
▼
|
||||
┌──────────────┐
|
||||
│Self-Heal On? │──No──▶ GATE_FAIL (exit 1)
|
||||
└──────┬───────┘ Fixable but disabled
|
||||
│Yes
|
||||
▼
|
||||
┌──────────────┐
|
||||
│Generate Fix │
|
||||
│Context Doc │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│Spawn Quick- │
|
||||
│Dev Fix │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│Re-validate │──────▶ (Back to Execute)
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| gate_status | State | PASS or FAIL |
|
||||
| fix_context_file | Path | Generated fix context (if failed) |
|
||||
| fix_attempt | State | Current attempt number |
|
||||
|
||||
## Completion Signal
|
||||
|
||||
```
|
||||
GATE_EVALUATED: PASS
|
||||
```
|
||||
or
|
||||
```
|
||||
GATE_EVALUATED: FAIL
|
||||
FIX_CONTEXT: {fix_context_file}
|
||||
FIX_ATTEMPT: {attempt}/{max_retries}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| Template not found | Use inline default template |
|
||||
| Fix context write fails | Log error, continue without self-healing |
|
||||
| Claude session fails | Log output, count as failed attempt |
|
||||
| Fix didn't resolve issue | Increment attempt, retry if allowed |
|
||||
|
|
@ -0,0 +1,235 @@
|
|||
# Step 5: Report Results
|
||||
|
||||
## Purpose
|
||||
|
||||
Update metrics files with validation results and output parseable signals for orchestration scripts (epic-chain.sh) to consume.
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Source | Required |
|
||||
|-------|--------|----------|
|
||||
| gate_status | Step 4 | Yes |
|
||||
| results | Step 3 | Yes |
|
||||
| passed_count | Step 3 | Yes |
|
||||
| failed_count | Step 3 | Yes |
|
||||
| fix_attempts | State | Yes |
|
||||
| epic_id | CLI | Yes |
|
||||
| metrics_dir | Config | Yes |
|
||||
|
||||
## Process
|
||||
|
||||
### 5.1 Update Metrics File
|
||||
|
||||
Update or create metrics file at:
|
||||
```
|
||||
{metrics_dir}/epic-{epic_id}-metrics.yaml
|
||||
```
|
||||
|
||||
#### 5.1.1 Using yq (if available)
|
||||
|
||||
```bash
|
||||
yq -i '.validation.gate_executed = true' "$metrics_file"
|
||||
yq -i '.validation.gate_status = "PASS"' "$metrics_file"
|
||||
yq -i '.validation.gate_mode = "quick"' "$metrics_file"
|
||||
yq -i '.validation.fix_attempts = 0' "$metrics_file"
|
||||
yq -i '.validation.scenarios_passed = 5' "$metrics_file"
|
||||
yq -i '.validation.scenarios_failed = 1' "$metrics_file"
|
||||
yq -i '.validation.timestamp = "2026-01-05T17:30:00Z"' "$metrics_file"
|
||||
```
|
||||
|
||||
#### 5.1.2 Fallback (no yq)
|
||||
|
||||
Create or append validation section:
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
gate_executed: true
|
||||
gate_mode: "quick"
|
||||
timestamp: "2026-01-05T17:30:00Z"
|
||||
results:
|
||||
passed: 5
|
||||
failed: 1
|
||||
skipped: 3
|
||||
gate_status: "PASS"
|
||||
fix_attempts: 0
|
||||
```
|
||||
|
||||
#### 5.1.3 Record Blocking Issues (if failed)
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
# ... other fields ...
|
||||
blocking_issues:
|
||||
- scenario_id: 3
|
||||
name: "Database Migration"
|
||||
error: "relation 'events' does not exist"
|
||||
command: "npx heimdall db migrate"
|
||||
```
|
||||
|
||||
#### 5.1.4 Record Fix History (if self-healing occurred)
|
||||
|
||||
```yaml
|
||||
validation:
|
||||
# ... other fields ...
|
||||
fix_history:
|
||||
- attempt: 1
|
||||
timestamp: "2026-01-05T17:35:00Z"
|
||||
failed_scenarios: ["scenario-3"]
|
||||
fix_context: "docs/sprint-artifacts/uat-fixes/epic-1-fix-context-1.md"
|
||||
result: "partial"
|
||||
- attempt: 2
|
||||
timestamp: "2026-01-05T17:42:00Z"
|
||||
failed_scenarios: []
|
||||
result: "success"
|
||||
```
|
||||
|
||||
### 5.2 Output Parseable Signals
|
||||
|
||||
Print to stdout in format that epic-chain.sh can parse:
|
||||
|
||||
```bash
|
||||
# Always output these signals
|
||||
echo "UAT_GATE_RESULT: $gate_status"
|
||||
echo "UAT_FIX_ATTEMPTS: $fix_attempts"
|
||||
echo "UAT_SCENARIOS_PASSED: $passed_count/$total_count"
|
||||
```
|
||||
|
||||
**Additional signals on failure:**
|
||||
|
||||
```bash
|
||||
echo "UAT_FIX_REQUIRED: true"
|
||||
echo "UAT_FIX_CONTEXT: $fix_context_file"
|
||||
```
|
||||
|
||||
**Additional signals on max retries:**
|
||||
|
||||
```bash
|
||||
echo "UAT_MAX_RETRIES: true"
|
||||
echo "UAT_HALT_CHAIN: true"
|
||||
```
|
||||
|
||||
### 5.3 Set Exit Code
|
||||
|
||||
| Status | Exit Code | Meaning |
|
||||
|--------|-----------|---------|
|
||||
| PASS | 0 | All scenarios passed |
|
||||
| FAIL (fixable) | 1 | Failed but retries remain or self-heal disabled |
|
||||
| FAIL (max retries) | 2 | Max retries exceeded, chain should halt |
|
||||
|
||||
### 5.4 Print Human-Readable Summary
|
||||
|
||||
```
|
||||
═══════════════════════════════════════════════════════════
|
||||
UAT VALIDATION COMPLETE
|
||||
═══════════════════════════════════════════════════════════
|
||||
|
||||
Epic: 1
|
||||
Gate Mode: quick
|
||||
Gate Result: PASS
|
||||
|
||||
Scenarios:
|
||||
Automatable: 6
|
||||
Semi-automated: 2
|
||||
Manual: 1
|
||||
|
||||
Results:
|
||||
Passed: 6
|
||||
Failed: 0
|
||||
Fix Attempts: 0
|
||||
|
||||
Artifacts:
|
||||
Log: /tmp/bmad-uat-validate-12345.log
|
||||
UAT Document: docs/uat/epic-1-uat.md
|
||||
Metrics: docs/sprint-artifacts/metrics/epic-1-metrics.yaml
|
||||
|
||||
═══════════════════════════════════════════════════════════
|
||||
```
|
||||
|
||||
### 5.5 Failure Summary (if applicable)
|
||||
|
||||
```
|
||||
═══════════════════════════════════════════════════════════
|
||||
UAT VALIDATION FAILED
|
||||
═══════════════════════════════════════════════════════════
|
||||
|
||||
Epic: 1
|
||||
Gate Mode: quick
|
||||
Gate Result: FAIL
|
||||
|
||||
Scenarios:
|
||||
Automatable: 6
|
||||
Semi-automated: 2
|
||||
Manual: 1
|
||||
|
||||
Results:
|
||||
Passed: 5
|
||||
Failed: 1
|
||||
Fix Attempts: 2
|
||||
|
||||
Blocking Issues:
|
||||
- Scenario 3: Database Migration
|
||||
Error: relation 'events' does not exist
|
||||
|
||||
Fix Context:
|
||||
docs/sprint-artifacts/uat-fixes/epic-1-fix-context-2.md
|
||||
|
||||
Action Required:
|
||||
Manual intervention needed - max retries exceeded
|
||||
|
||||
═══════════════════════════════════════════════════════════
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Location | Description |
|
||||
|--------|----------|-------------|
|
||||
| Updated metrics | YAML file | Validation results persisted |
|
||||
| Signals | stdout | Parseable output for orchestration |
|
||||
| Summary | stdout | Human-readable summary |
|
||||
| Exit code | Process | 0=pass, 1=fail-fixable, 2=fail-max-retries |
|
||||
|
||||
## Completion Signal
|
||||
|
||||
```
|
||||
RESULTS_REPORTED: {metrics_file_path}
|
||||
```
|
||||
|
||||
## Signal Reference
|
||||
|
||||
### For Orchestration Scripts
|
||||
|
||||
| Signal | Values | Description |
|
||||
|--------|--------|-------------|
|
||||
| `UAT_GATE_RESULT` | PASS, FAIL | Overall gate status |
|
||||
| `UAT_FIX_ATTEMPTS` | 0-N | Number of fix attempts made |
|
||||
| `UAT_SCENARIOS_PASSED` | X/Y | Passed/Total ratio |
|
||||
| `UAT_FIX_REQUIRED` | true | Indicates fixes were needed |
|
||||
| `UAT_FIX_CONTEXT` | path | Location of fix context doc |
|
||||
| `UAT_MAX_RETRIES` | true | Max retries exceeded |
|
||||
| `UAT_HALT_CHAIN` | true | Chain should stop |
|
||||
|
||||
### Parsing in Shell
|
||||
|
||||
```bash
|
||||
# In epic-chain.sh
|
||||
uat_output=$("$SCRIPT_DIR/uat-validate.sh" "$epic_id" 2>&1)
|
||||
|
||||
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||
log_success "UAT passed"
|
||||
else
|
||||
log_error "UAT failed"
|
||||
|
||||
if echo "$uat_output" | grep -q "UAT_HALT_CHAIN: true"; then
|
||||
log_error "Halting chain - max retries exceeded"
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| Metrics file write fails | Log warning, continue with signal output |
|
||||
| yq not available | Use fallback append method |
|
||||
| Invalid metrics file | Create new file with validation section |
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# UAT Fix Context - Epic {epic_id} (Attempt {attempt})
|
||||
|
||||
**Generated:** {timestamp}
|
||||
**Epic:** {epic_name}
|
||||
**Epic:** {epic_id}
|
||||
**Gate Result:** FAIL ({passed}/{total} scenarios passed)
|
||||
|
||||
---
|
||||
|
|
@ -12,101 +12,16 @@ This document contains the context needed to fix UAT failures for Epic {epic_id}
|
|||
|
||||
**Failures to fix:** {failure_count}
|
||||
**Fix attempt:** {attempt} of {max_retries}
|
||||
**UAT Document:** `{uat_doc_path}`
|
||||
|
||||
---
|
||||
|
||||
## Failed Scenarios
|
||||
## Quick Start
|
||||
|
||||
{#each failed_scenario}
|
||||
### Scenario {scenario_id}: {scenario_name}
|
||||
|
||||
**Command Executed:**
|
||||
```bash
|
||||
{command}
|
||||
```
|
||||
|
||||
**Expected Result:**
|
||||
{expected_result}
|
||||
|
||||
**Actual Result:**
|
||||
```
|
||||
{actual_output}
|
||||
```
|
||||
|
||||
**Error Output:**
|
||||
```
|
||||
{stderr}
|
||||
```
|
||||
|
||||
**Exit Code:** {exit_code}
|
||||
|
||||
**Related Story:** {story_id}
|
||||
|
||||
**Acceptance Criteria:**
|
||||
{#each acceptance_criteria}
|
||||
- {criterion}
|
||||
{/each}
|
||||
|
||||
**Root Cause Hint:**
|
||||
{root_cause_hint}
|
||||
|
||||
**Files Likely Involved:**
|
||||
{#each likely_files}
|
||||
- `{file_path}`
|
||||
{/each}
|
||||
1. Read each failed scenario below
|
||||
2. Identify root cause from error output
|
||||
3. Locate and fix the relevant code
|
||||
4. Verify fix: run the failing command
|
||||
5. Commit: `fix(epic-{epic_id}): {description}`
|
||||
|
||||
---
|
||||
{/each}
|
||||
|
||||
## Fix Instructions
|
||||
|
||||
Address the failures above in priority order. For each fix:
|
||||
|
||||
1. **Analyze** - Understand why the scenario failed
|
||||
2. **Locate** - Find the relevant code files
|
||||
3. **Fix** - Implement the minimum change to resolve the failure
|
||||
4. **Verify** - Run the scenario command locally to confirm fix
|
||||
5. **Commit** - Use message format: `fix(epic-{epic_id}): {description}`
|
||||
|
||||
### Priority Order
|
||||
|
||||
{#each failed_scenario}
|
||||
{priority}. **Scenario {scenario_id}**: {one_line_description}
|
||||
{/each}
|
||||
|
||||
---
|
||||
|
||||
## Constraints
|
||||
|
||||
- Only fix the identified failures - do not refactor unrelated code
|
||||
- Run the specific failing commands to verify each fix
|
||||
- Run project tests after all fixes: `npm test`
|
||||
- Commit with conventional format: `fix(epic-{epic_id}): {description}`
|
||||
- If a fix requires changes that would break other scenarios, document the tradeoff
|
||||
|
||||
---
|
||||
|
||||
## Context Files
|
||||
|
||||
The following files may provide additional context:
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `{uat_doc_path}` | Full UAT document with all scenarios |
|
||||
| `{story_files}` | Story files with complete acceptance criteria |
|
||||
| `{architecture_doc}` | System architecture reference |
|
||||
|
||||
---
|
||||
|
||||
## After Fixing
|
||||
|
||||
Once all fixes are committed, the UAT validation will automatically re-run.
|
||||
|
||||
- **If all pass:** Epic continues to next phase
|
||||
- **If failures remain:** Another fix context will be generated (attempt {next_attempt})
|
||||
- **If max retries exceeded:** Chain halts for human intervention
|
||||
|
||||
---
|
||||
|
||||
*Generated by UAT Validate Workflow*
|
||||
*BMAD Method - Epic Chain Self-Healing*
|
||||
|
|
|
|||
Loading…
Reference in New Issue