feat: implement UAT validation with self-healing fix loop
Add complete UAT validation workflow implementation: - scripts/uat-validate.sh: Core UAT validation script with scenario classification, automated execution, and self-healing fix loop - scripts/epic-execute.sh: Add metrics instrumentation for story execution tracking (init, update, finalize metrics) - scripts/epic-chain.sh: Integrate UAT gate after each epic with configurable blocking mode, and add report generation phase New workflow step documentation: - step-01-load-uat.md: UAT document loading and parsing - step-02-classify-scenarios.md: Scenario classification logic - step-03-execute-scenarios.md: Automated scenario execution - step-04-evaluate-gate.md: Gate evaluation and fix loop - step-05-report-results.md: Metrics and signal output Key features: - Gate modes: quick (automatable only), full (+ semi-auto), skip - Self-healing: spawns fresh Claude context for targeted fixes - Metrics: YAML files track execution stats and UAT results - Signals: parseable output for orchestration (UAT_GATE_RESULT, etc.) - Handoffs: include UAT status and fix context references Also fixes lint/format issues in pre-existing files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Caleb
parent
6b1ca9333f
commit
15b7bb1ae6
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,526 @@
|
||||||
|
# User Acceptance Testing: Epic 1 - Foundation, CLI & Deployment Infrastructure
|
||||||
|
|
||||||
|
**Version:** 1.0
|
||||||
|
**Date:** January 2, 2026
|
||||||
|
**Epic:** Foundation, CLI & Deployment Infrastructure
|
||||||
|
**Status:** Ready for Testing
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Overview
|
||||||
|
|
||||||
|
### What Was Built
|
||||||
|
|
||||||
|
Epic 1 delivers the foundation of the Heimdall Customer Management system. After completing this epic, administrators can:
|
||||||
|
|
||||||
|
- **Initialize a new Heimdall project** using a simple command-line tool
|
||||||
|
- **Configure connections** to Supabase (your database) and Resend (your email service)
|
||||||
|
- **Set up the database** with all necessary tables for job processing
|
||||||
|
- **Run a worker process** that continuously processes background jobs
|
||||||
|
- **Deploy to Railway** (a cloud hosting platform) for production use
|
||||||
|
- **Send test emails** to verify your email configuration is working
|
||||||
|
|
||||||
|
In plain terms: This is the "plumbing" that makes everything else possible. Once Epic 1 is working, you have a running system ready to handle customer management workflows.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Prerequisites
|
||||||
|
|
||||||
|
### Test Environment Requirements
|
||||||
|
|
||||||
|
Before starting UAT, ensure you have:
|
||||||
|
|
||||||
|
| Requirement | Details | How to Verify |
|
||||||
|
|-------------|---------|---------------|
|
||||||
|
| **Node.js 20.x** | JavaScript runtime | Run `node --version` - should show v20.x.x |
|
||||||
|
| **npm** | Package manager | Run `npm --version` - should show 9.x or higher |
|
||||||
|
| **Git** | Version control | Run `git --version` |
|
||||||
|
| **Terminal access** | Command line interface | macOS Terminal, Windows PowerShell, or Linux terminal |
|
||||||
|
|
||||||
|
### Required Accounts
|
||||||
|
|
||||||
|
| Service | Purpose | Sign-up URL |
|
||||||
|
|---------|---------|-------------|
|
||||||
|
| **Supabase** | Database hosting | <https://supabase.com> (free tier available) |
|
||||||
|
| **Resend** | Email delivery | <https://resend.com> (free tier: 100 emails/day) |
|
||||||
|
| **Railway** (optional) | Cloud deployment | <https://railway.app> (for production testing) |
|
||||||
|
|
||||||
|
### Credentials You Will Need
|
||||||
|
|
||||||
|
Before testing, gather these from your service dashboards:
|
||||||
|
|
||||||
|
1. **From Supabase Dashboard:**
|
||||||
|
- Project URL (e.g., `https://xxxxx.supabase.co`)
|
||||||
|
- Anon/Public Key (starts with `eyJ...`)
|
||||||
|
- Database Connection String (from Settings > Database > Connection string > URI)
|
||||||
|
|
||||||
|
2. **From Resend Dashboard:**
|
||||||
|
- API Key (starts with `re_...`)
|
||||||
|
- A verified sending domain (or use their default for testing)
|
||||||
|
|
||||||
|
3. **Your Information:**
|
||||||
|
- Admin email address (for receiving notifications and test emails)
|
||||||
|
- Workspace name (a label for your installation)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Test Scenarios
|
||||||
|
|
||||||
|
### Scenario 1: Project Initialization
|
||||||
|
|
||||||
|
**Goal:** Verify that you can create a new Heimdall project with the correct structure.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Open your terminal application
|
||||||
|
2. Navigate to a folder where you want to create the project:
|
||||||
|
```
|
||||||
|
cd ~/Documents
|
||||||
|
```
|
||||||
|
3. Clone and set up the Heimdall project:
|
||||||
|
```
|
||||||
|
git clone <repository-url> heimdall-test
|
||||||
|
cd heimdall-test
|
||||||
|
npm install
|
||||||
|
npm run build
|
||||||
|
```
|
||||||
|
4. Verify the CLI is available:
|
||||||
|
```
|
||||||
|
npx heimdall --version
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] `npm install` completes without errors
|
||||||
|
- [ ] `npm run build` shows "5 packages built successfully" or similar
|
||||||
|
- [ ] `npx heimdall --version` displays a version number (e.g., `1.0.0`)
|
||||||
|
- [ ] `npx heimdall --help` shows available commands including `config`, `db`, `start`, `test-send`, `test-queue`
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record any error messages or unexpected behavior here:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 2: Configuration Setup
|
||||||
|
|
||||||
|
**Goal:** Create and validate your configuration file with Supabase and Resend credentials.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Generate a configuration template:
|
||||||
|
```
|
||||||
|
npx heimdall config init
|
||||||
|
```
|
||||||
|
2. Open the created file `heimdall.config.yaml` in a text editor
|
||||||
|
3. Replace the placeholder values with your real credentials:
|
||||||
|
- Under `workspace:` - enter your workspace name and admin email
|
||||||
|
- Under `supabase:` - enter your Supabase URL, anon key, and database URL
|
||||||
|
- Under `resend:` - enter your Resend API key
|
||||||
|
4. Save the file
|
||||||
|
5. Validate your configuration:
|
||||||
|
```
|
||||||
|
npx heimdall config validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] `config init` creates a file named `heimdall.config.yaml`
|
||||||
|
- [ ] The file contains sections for `workspace`, `supabase`, `resend`, and `ai` (optional)
|
||||||
|
- [ ] `config validate` shows your workspace name and admin email (with secrets partially hidden)
|
||||||
|
- [ ] Validation shows "Configuration is valid" or similar success message
|
||||||
|
- [ ] No error messages appear about missing or invalid fields
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_If validation fails, record the error message:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 3: Database Migration
|
||||||
|
|
||||||
|
**Goal:** Set up the required database tables in your Supabase instance.
|
||||||
|
|
||||||
|
**Prerequisites:** Scenario 2 must be completed successfully.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Run the database migration:
|
||||||
|
```
|
||||||
|
npx heimdall db migrate
|
||||||
|
```
|
||||||
|
2. Check the database status:
|
||||||
|
```
|
||||||
|
npx heimdall db status
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] `db migrate` completes with a success message (e.g., "pg-boss initialized successfully")
|
||||||
|
- [ ] `db status` shows:
|
||||||
|
- Database connected: Yes
|
||||||
|
- pg-boss schema: Exists
|
||||||
|
- Tables: job, schedule, subscription, version (or similar)
|
||||||
|
- [ ] Running `db migrate` a second time does NOT cause errors (idempotent)
|
||||||
|
|
||||||
|
**Verification in Supabase Dashboard:**
|
||||||
|
1. Log into your Supabase project
|
||||||
|
2. Go to the Table Editor
|
||||||
|
3. Look for a schema named `pgboss`
|
||||||
|
4. Verify tables exist: `job`, `schedule`, `subscription`, `version`
|
||||||
|
|
||||||
|
- [ ] pg-boss tables visible in Supabase Dashboard
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record database status output:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 4: Connection Validation (Detailed)
|
||||||
|
|
||||||
|
**Goal:** Verify both Supabase API and direct database connections work correctly.
|
||||||
|
|
||||||
|
**Prerequisites:** Scenario 3 must be completed successfully.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Run the full validation:
|
||||||
|
```
|
||||||
|
npx heimdall config validate
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Message: "Supabase API connected" with response time in milliseconds
|
||||||
|
- [ ] Message: "Operational DB connected" with response time in milliseconds
|
||||||
|
- [ ] Message: "Resend API connected" (if Resend key is configured)
|
||||||
|
- [ ] All connections show as successful (green checkmarks or similar)
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record connection times and any warnings:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 5: Worker Process Startup
|
||||||
|
|
||||||
|
**Goal:** Start the background worker that processes jobs.
|
||||||
|
|
||||||
|
**Prerequisites:** Scenario 3 must be completed successfully.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Start the worker process:
|
||||||
|
```
|
||||||
|
npx heimdall start
|
||||||
|
```
|
||||||
|
2. Observe the output for approximately 30 seconds
|
||||||
|
3. Press `Ctrl+C` to stop the worker
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Message appears: "Heimdall worker started, polling for jobs..." or similar
|
||||||
|
- [ ] No error messages during startup
|
||||||
|
- [ ] Worker continues running without crashing
|
||||||
|
- [ ] When you press `Ctrl+C`, message appears: "Heimdall worker shutting down gracefully..."
|
||||||
|
- [ ] Process exits cleanly (returns to command prompt)
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record startup messages:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 6: Job Queue Testing
|
||||||
|
|
||||||
|
**Goal:** Verify that jobs can be queued and processed by the worker.
|
||||||
|
|
||||||
|
**Prerequisites:** Scenario 5 must work (worker can start).
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Open **two terminal windows** side by side
|
||||||
|
2. In Terminal 1, start the worker:
|
||||||
|
```
|
||||||
|
npx heimdall start
|
||||||
|
```
|
||||||
|
3. Wait for the "polling for jobs" message
|
||||||
|
4. In Terminal 2, enqueue a test job:
|
||||||
|
```
|
||||||
|
npx heimdall test-queue
|
||||||
|
```
|
||||||
|
5. Watch Terminal 1 for job processing
|
||||||
|
6. Stop the worker in Terminal 1 with `Ctrl+C`
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Terminal 2 shows: "Test job enqueued: test-job-{some-id}"
|
||||||
|
- [ ] Within 60 seconds, Terminal 1 shows: "Job executed: test-job-{same-id}"
|
||||||
|
- [ ] No errors in either terminal
|
||||||
|
- [ ] Job ID in Terminal 2 matches the ID in Terminal 1
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record time between enqueueing and execution:_
|
||||||
|
_Record job ID:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 7: Test Email Sending
|
||||||
|
|
||||||
|
**Goal:** Verify that Heimdall can send emails through Resend.
|
||||||
|
|
||||||
|
**Prerequisites:**
|
||||||
|
- Scenario 2 completed with valid Resend API key
|
||||||
|
- You have access to the email inbox specified
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Send a test email to yourself:
|
||||||
|
```
|
||||||
|
npx heimdall test-send --to your-email@example.com
|
||||||
|
```
|
||||||
|
(Replace with your actual email address)
|
||||||
|
2. Check your email inbox (including spam/junk folder)
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Command shows: "Test email sent: {resend-message-id}"
|
||||||
|
- [ ] Email arrives in inbox within 2-5 minutes
|
||||||
|
- [ ] Email subject: "Heimdall Test Email"
|
||||||
|
- [ ] Email body confirms configuration is working
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Record Resend message ID:_
|
||||||
|
_Time until email arrived:_
|
||||||
|
_Did email land in spam?:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 8: Health Endpoint (Local)
|
||||||
|
|
||||||
|
**Goal:** Verify the worker's health endpoint responds correctly.
|
||||||
|
|
||||||
|
**Prerequisites:** Worker can start (Scenario 5).
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Start the worker:
|
||||||
|
```
|
||||||
|
npx heimdall start
|
||||||
|
```
|
||||||
|
2. Open a web browser
|
||||||
|
3. Navigate to: `http://localhost:3000/health`
|
||||||
|
4. Stop the worker with `Ctrl+C`
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Browser shows JSON response
|
||||||
|
- [ ] Response contains: `"status": "healthy"`
|
||||||
|
- [ ] Response contains: `"queue": "connected"`
|
||||||
|
- [ ] Response contains: `"uptime": {some-number}`
|
||||||
|
|
||||||
|
**Sample Expected Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "healthy",
|
||||||
|
"queue": "connected",
|
||||||
|
"uptime": 45
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Copy the actual response here:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 9: Railway Deployment (Optional)
|
||||||
|
|
||||||
|
**Goal:** Deploy Heimdall to Railway for production use.
|
||||||
|
|
||||||
|
**Prerequisites:**
|
||||||
|
- Railway account created
|
||||||
|
- All previous scenarios pass locally
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. Log into Railway Dashboard
|
||||||
|
2. Create a new project from the GitHub repository
|
||||||
|
3. Add environment variables in Railway settings:
|
||||||
|
- `DATABASE_URL` = your Supabase connection string
|
||||||
|
- `SUPABASE_URL` = your Supabase project URL
|
||||||
|
- `SUPABASE_ANON_KEY` = your Supabase anon key
|
||||||
|
- `RESEND_API_KEY` = your Resend API key
|
||||||
|
- `ADMIN_EMAIL` = your admin email
|
||||||
|
- `WORKSPACE_NAME` = your workspace name
|
||||||
|
4. Deploy the service
|
||||||
|
5. Once deployed, access the health endpoint at your Railway URL + `/health`
|
||||||
|
|
||||||
|
**Expected Results:**
|
||||||
|
|
||||||
|
- [ ] Deployment completes without build errors
|
||||||
|
- [ ] Service shows as "Running" in Railway dashboard
|
||||||
|
- [ ] Health endpoint at `https://your-app.railway.app/health` returns:
|
||||||
|
- `"status": "healthy"`
|
||||||
|
- `"queue": "connected"`
|
||||||
|
- [ ] Logs show "Heimdall worker started, polling for jobs..."
|
||||||
|
|
||||||
|
**Notes for Tester:**
|
||||||
|
_Railway deployment URL:_
|
||||||
|
_Any deployment warnings or notes:_
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Success Criteria
|
||||||
|
|
||||||
|
### Minimum Requirements for Sign-off
|
||||||
|
|
||||||
|
All of the following must pass for Epic 1 to be accepted:
|
||||||
|
|
||||||
|
| # | Criteria | Scenario | Status |
|
||||||
|
|---|----------|----------|--------|
|
||||||
|
| 1 | Project builds successfully | Scenario 1 | [ ] Pass / [ ] Fail |
|
||||||
|
| 2 | CLI commands are accessible | Scenario 1 | [ ] Pass / [ ] Fail |
|
||||||
|
| 3 | Configuration file is created correctly | Scenario 2 | [ ] Pass / [ ] Fail |
|
||||||
|
| 4 | Configuration validation works | Scenario 2 | [ ] Pass / [ ] Fail |
|
||||||
|
| 5 | Database migration completes | Scenario 3 | [ ] Pass / [ ] Fail |
|
||||||
|
| 6 | Database migration is idempotent | Scenario 3 | [ ] Pass / [ ] Fail |
|
||||||
|
| 7 | Supabase API connection validated | Scenario 4 | [ ] Pass / [ ] Fail |
|
||||||
|
| 8 | Operational DB connection validated | Scenario 4 | [ ] Pass / [ ] Fail |
|
||||||
|
| 9 | Worker process starts and polls | Scenario 5 | [ ] Pass / [ ] Fail |
|
||||||
|
| 10 | Worker shuts down gracefully | Scenario 5 | [ ] Pass / [ ] Fail |
|
||||||
|
| 11 | Test jobs are processed | Scenario 6 | [ ] Pass / [ ] Fail |
|
||||||
|
| 12 | Jobs processed within 60 seconds | Scenario 6 | [ ] Pass / [ ] Fail |
|
||||||
|
| 13 | Test emails are sent and received | Scenario 7 | [ ] Pass / [ ] Fail |
|
||||||
|
| 14 | Health endpoint responds correctly | Scenario 8 | [ ] Pass / [ ] Fail |
|
||||||
|
|
||||||
|
### Optional (Recommended)
|
||||||
|
|
||||||
|
| # | Criteria | Scenario | Status |
|
||||||
|
|---|----------|----------|--------|
|
||||||
|
| 15 | Railway deployment successful | Scenario 9 | [ ] Pass / [ ] Fail / [ ] Skipped |
|
||||||
|
| 16 | Production health endpoint works | Scenario 9 | [ ] Pass / [ ] Fail / [ ] Skipped |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Known Limitations
|
||||||
|
|
||||||
|
The following are expected behaviors, not bugs:
|
||||||
|
|
||||||
|
1. **Error messages for invalid credentials** are intentionally detailed to help debugging
|
||||||
|
2. **First database migration** may take 10-30 seconds as pg-boss creates multiple tables
|
||||||
|
3. **Test emails** may land in spam for unverified domains
|
||||||
|
4. **Worker polling interval** is 5 seconds, so jobs may take up to 5 seconds to start processing
|
||||||
|
5. **Config validate** shows partial secrets (first/last 4 characters) for verification purposes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Troubleshooting Guide
|
||||||
|
|
||||||
|
### Common Issues and Solutions
|
||||||
|
|
||||||
|
| Symptom | Likely Cause | Solution |
|
||||||
|
|---------|--------------|----------|
|
||||||
|
| "Command not found: heimdall" | Build not completed | Run `npm run build` first |
|
||||||
|
| "Cannot connect to database" | Wrong db_url | Check Supabase connection string format |
|
||||||
|
| "Invalid API key" | Resend key incorrect | Regenerate key in Resend dashboard |
|
||||||
|
| "ECONNREFUSED" | Database not accessible | Check Supabase project is running, check IP restrictions |
|
||||||
|
| Worker crashes on startup | Missing configuration | Run `heimdall config validate` first |
|
||||||
|
| Email not received | Domain not verified | Verify domain in Resend or check spam folder |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Sign-off Section
|
||||||
|
|
||||||
|
### Testing Summary
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|------|-------|
|
||||||
|
| **Tester Name** | _________________________ |
|
||||||
|
| **Test Date** | _________________________ |
|
||||||
|
| **Environment** | [ ] Local / [ ] Railway / [ ] Both |
|
||||||
|
| **Node.js Version** | _________________________ |
|
||||||
|
| **Operating System** | _________________________ |
|
||||||
|
|
||||||
|
### Test Results Summary
|
||||||
|
|
||||||
|
| Category | Passed | Failed | Skipped |
|
||||||
|
|----------|--------|--------|---------|
|
||||||
|
| Required Scenarios (1-8) | ___ / 8 | ___ | ___ |
|
||||||
|
| Optional Scenarios (9) | ___ / 1 | ___ | ___ |
|
||||||
|
| Success Criteria (1-14) | ___ / 14 | ___ | ___ |
|
||||||
|
|
||||||
|
### Issues Found
|
||||||
|
|
||||||
|
| Issue # | Scenario | Description | Severity |
|
||||||
|
|---------|----------|-------------|----------|
|
||||||
|
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||||
|
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||||
|
| | | | [ ] Blocker / [ ] Major / [ ] Minor |
|
||||||
|
|
||||||
|
### Final Decision
|
||||||
|
|
||||||
|
- [ ] **APPROVED** - All required criteria pass, ready for production
|
||||||
|
- [ ] **APPROVED WITH CONDITIONS** - Minor issues noted, can proceed
|
||||||
|
- [ ] **NOT APPROVED** - Blocker issues must be resolved
|
||||||
|
|
||||||
|
### Signatures
|
||||||
|
|
||||||
|
| Role | Name | Signature | Date |
|
||||||
|
|------|------|-----------|------|
|
||||||
|
| **Tester** | | | |
|
||||||
|
| **Product Owner** | | | |
|
||||||
|
| **Technical Lead** | | | |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Appendix
|
||||||
|
|
||||||
|
### A. CLI Command Reference
|
||||||
|
|
||||||
|
| Command | Purpose |
|
||||||
|
|---------|---------|
|
||||||
|
| `heimdall --version` | Display version number |
|
||||||
|
| `heimdall --help` | Show available commands |
|
||||||
|
| `heimdall config init` | Create configuration file |
|
||||||
|
| `heimdall config validate` | Validate configuration |
|
||||||
|
| `heimdall db migrate` | Set up database tables |
|
||||||
|
| `heimdall db status` | Check database status |
|
||||||
|
| `heimdall start` | Start worker process |
|
||||||
|
| `heimdall test-queue` | Enqueue a test job |
|
||||||
|
| `heimdall test-send --to EMAIL` | Send a test email |
|
||||||
|
|
||||||
|
### B. Configuration File Template
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# heimdall.config.yaml
|
||||||
|
workspace:
|
||||||
|
name: "my-workspace"
|
||||||
|
admin_email: "admin@example.com"
|
||||||
|
|
||||||
|
supabase:
|
||||||
|
url: "https://xxxxx.supabase.co"
|
||||||
|
anon_key: "eyJ..."
|
||||||
|
db_url: "postgresql://postgres:password@db.xxxxx.supabase.co:5432/postgres"
|
||||||
|
|
||||||
|
resend:
|
||||||
|
api_key: "re_..."
|
||||||
|
|
||||||
|
# Optional AI configuration
|
||||||
|
ai:
|
||||||
|
provider: "anthropic"
|
||||||
|
api_key: "sk-ant-..."
|
||||||
|
model: "claude-3-haiku-20240307"
|
||||||
|
```
|
||||||
|
|
||||||
|
### C. Environment Variables for Railway
|
||||||
|
|
||||||
|
| Variable | Required | Description |
|
||||||
|
|----------|----------|-------------|
|
||||||
|
| `DATABASE_URL` | Yes | PostgreSQL connection string |
|
||||||
|
| `SUPABASE_URL` | Yes | Supabase project URL |
|
||||||
|
| `SUPABASE_ANON_KEY` | Yes | Supabase anonymous key |
|
||||||
|
| `RESEND_API_KEY` | Yes | Resend API key |
|
||||||
|
| `ADMIN_EMAIL` | Yes | Admin notification email |
|
||||||
|
| `WORKSPACE_NAME` | No | Workspace identifier |
|
||||||
|
| `PORT` | No | Server port (default: 3000) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Document generated: January 2, 2026*
|
||||||
|
*Epic 1: Foundation, CLI & Deployment Infrastructure*
|
||||||
|
|
@ -9,15 +9,25 @@
|
||||||
# ./epic-chain.sh 36 37 38 --dry-run --verbose
|
# ./epic-chain.sh 36 37 38 --dry-run --verbose
|
||||||
# ./epic-chain.sh 36 37 38 --analyze-only
|
# ./epic-chain.sh 36 37 38 --analyze-only
|
||||||
# ./epic-chain.sh 36 37 38 --start-from 37
|
# ./epic-chain.sh 36 37 38 --start-from 37
|
||||||
|
# ./epic-chain.sh 36 37 38 --uat-gate=full --uat-blocking
|
||||||
#
|
#
|
||||||
# Options:
|
# Options:
|
||||||
# --dry-run Show what would be executed without running
|
# --dry-run Show what would be executed without running
|
||||||
# --analyze-only Run analysis phase only, don't execute
|
# --analyze-only Run analysis phase only, don't execute
|
||||||
# --verbose Show detailed output
|
# --verbose Show detailed output
|
||||||
# --start-from ID Start from a specific epic (skip earlier ones)
|
# --start-from ID Start from a specific epic (skip earlier ones)
|
||||||
# --skip-done Skip epics/stories with Status: Done
|
# --skip-done Skip epics/stories with Status: Done
|
||||||
# --no-handoff Don't generate context handoffs between epics
|
# --no-handoff Don't generate context handoffs between epics
|
||||||
# --no-combined-uat Skip combined UAT generation at end
|
# --no-combined-uat Skip combined UAT generation at end
|
||||||
|
#
|
||||||
|
# UAT Gate Options:
|
||||||
|
# --uat-gate=MODE UAT validation mode: quick|full|skip (default: quick)
|
||||||
|
# --uat-blocking Halt chain if UAT fails (default: continue)
|
||||||
|
# --uat-retries=N Max fix attempts per epic (default: 2)
|
||||||
|
# --no-uat Disable UAT validation gate entirely
|
||||||
|
#
|
||||||
|
# Report Options:
|
||||||
|
# --no-report Skip chain execution report generation
|
||||||
#
|
#
|
||||||
|
|
||||||
set -e
|
set -e
|
||||||
|
|
@ -48,6 +58,19 @@ CYAN='\033[0;36m'
|
||||||
BOLD='\033[1m'
|
BOLD='\033[1m'
|
||||||
NC='\033[0m' # No Color
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# UAT Gate Configuration
|
||||||
|
UAT_GATE_ENABLED="${UAT_GATE_ENABLED:-true}"
|
||||||
|
UAT_GATE_MODE="${UAT_GATE_MODE:-quick}"
|
||||||
|
UAT_MAX_RETRIES="${UAT_MAX_RETRIES:-2}"
|
||||||
|
UAT_BLOCKING="${UAT_BLOCKING:-false}"
|
||||||
|
|
||||||
|
# Metrics Configuration
|
||||||
|
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||||
|
|
||||||
|
# Report Configuration
|
||||||
|
GENERATE_REPORT="${GENERATE_REPORT:-true}"
|
||||||
|
CHAIN_REPORT_FILE="$SPRINT_ARTIFACTS_DIR/chain-execution-report.md"
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Helper Functions
|
# Helper Functions
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
@ -87,6 +110,80 @@ log_section() {
|
||||||
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Helper function to create basic report if Claude fails
|
||||||
|
create_basic_report() {
|
||||||
|
local end_time_iso=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
local duration_formatted="${DURATION}s"
|
||||||
|
if [ $DURATION -gt 3600 ]; then
|
||||||
|
duration_formatted="$((DURATION / 3600))h $((DURATION % 3600 / 60))m"
|
||||||
|
elif [ $DURATION -gt 60 ]; then
|
||||||
|
duration_formatted="$((DURATION / 60))m $((DURATION % 60))s"
|
||||||
|
fi
|
||||||
|
|
||||||
|
cat > "$CHAIN_REPORT_FILE" << EOF
|
||||||
|
# Epic Chain Execution Report
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
**Execution Method:** BMAD Epic Chain (automated AI-driven development)
|
||||||
|
**Status:** $([ $FAILED_EPICS -eq 0 ] && echo "COMPLETE" || echo "PARTIAL")
|
||||||
|
|
||||||
|
| Metric | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| Total Epics | ${#EPIC_IDS[@]} |
|
||||||
|
| Completed | $COMPLETED_EPICS |
|
||||||
|
| Failed | $FAILED_EPICS |
|
||||||
|
| Skipped | $SKIPPED_EPICS |
|
||||||
|
| Duration | $duration_formatted |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Timeline
|
||||||
|
|
||||||
|
| Epic | Status |
|
||||||
|
|------|--------|
|
||||||
|
EOF
|
||||||
|
|
||||||
|
for epic_id in "${EPIC_IDS[@]}"; do
|
||||||
|
local status="Unknown"
|
||||||
|
local metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||||
|
if [ -f "$metrics_file" ]; then
|
||||||
|
if command -v yq >/dev/null 2>&1; then
|
||||||
|
local completed=$(yq '.stories.completed // 0' "$metrics_file")
|
||||||
|
local failed=$(yq '.stories.failed // 0' "$metrics_file")
|
||||||
|
if [ "$failed" -gt 0 ]; then
|
||||||
|
status="Partial ($completed completed, $failed failed)"
|
||||||
|
else
|
||||||
|
status="Complete ($completed stories)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
echo "| Epic $epic_id | $status |" >> "$CHAIN_REPORT_FILE"
|
||||||
|
done
|
||||||
|
|
||||||
|
cat >> "$CHAIN_REPORT_FILE" << EOF
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
| Artifact | Location |
|
||||||
|
|----------|----------|
|
||||||
|
| Chain Plan | $CHAIN_PLAN_FILE |
|
||||||
|
| Metrics | $METRICS_DIR/ |
|
||||||
|
| UAT Documents | $UAT_DIR/ |
|
||||||
|
| Handoffs | $HANDOFF_DIR/ |
|
||||||
|
| Log | $LOG_FILE |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Report generated: $end_time_iso*
|
||||||
|
*BMAD Method Epic Chain*
|
||||||
|
EOF
|
||||||
|
|
||||||
|
log_success "Basic report created: $CHAIN_REPORT_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Argument Parsing
|
# Argument Parsing
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
@ -130,6 +227,26 @@ while [[ $# -gt 0 ]]; do
|
||||||
NO_COMBINED_UAT=true
|
NO_COMBINED_UAT=true
|
||||||
shift
|
shift
|
||||||
;;
|
;;
|
||||||
|
--uat-gate=*)
|
||||||
|
UAT_GATE_MODE="${1#*=}"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--uat-blocking)
|
||||||
|
UAT_BLOCKING=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--no-uat)
|
||||||
|
UAT_GATE_ENABLED=false
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--uat-retries=*)
|
||||||
|
UAT_MAX_RETRIES="${1#*=}"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--no-report)
|
||||||
|
GENERATE_REPORT=false
|
||||||
|
shift
|
||||||
|
;;
|
||||||
-*)
|
-*)
|
||||||
echo "Unknown option: $1"
|
echo "Unknown option: $1"
|
||||||
exit 1
|
exit 1
|
||||||
|
|
@ -149,15 +266,25 @@ if [ ${#EPIC_IDS[@]} -eq 0 ]; then
|
||||||
echo " $0 36 37 38 --dry-run # Show what would happen"
|
echo " $0 36 37 38 --dry-run # Show what would happen"
|
||||||
echo " $0 36 37 38 --analyze-only # Just analyze, don't execute"
|
echo " $0 36 37 38 --analyze-only # Just analyze, don't execute"
|
||||||
echo " $0 36 37 38 --start-from 37 # Resume from epic 37"
|
echo " $0 36 37 38 --start-from 37 # Resume from epic 37"
|
||||||
|
echo " $0 36 37 38 --uat-gate=full # Run full UAT validation after each epic"
|
||||||
echo ""
|
echo ""
|
||||||
echo "Options:"
|
echo "Options:"
|
||||||
echo " --dry-run Show execution plan without running"
|
echo " --dry-run Show execution plan without running"
|
||||||
echo " --analyze-only Analyze dependencies only"
|
echo " --analyze-only Analyze dependencies only"
|
||||||
echo " --verbose Detailed output"
|
echo " --verbose Detailed output"
|
||||||
echo " --start-from ID Start from specific epic"
|
echo " --start-from ID Start from specific epic"
|
||||||
echo " --skip-done Skip completed stories"
|
echo " --skip-done Skip completed stories"
|
||||||
echo " --no-handoff Skip context handoffs between epics"
|
echo " --no-handoff Skip context handoffs between epics"
|
||||||
echo " --no-combined-uat Skip combined UAT at end"
|
echo " --no-combined-uat Skip combined UAT at end"
|
||||||
|
echo ""
|
||||||
|
echo "UAT Gate Options:"
|
||||||
|
echo " --uat-gate=MODE UAT validation mode: quick|full|skip (default: quick)"
|
||||||
|
echo " --uat-blocking Halt chain if UAT fails (default: continue)"
|
||||||
|
echo " --uat-retries=N Max fix attempts per epic (default: 2)"
|
||||||
|
echo " --no-uat Disable UAT validation gate entirely"
|
||||||
|
echo ""
|
||||||
|
echo "Report Options:"
|
||||||
|
echo " --no-report Skip chain execution report generation"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
|
@ -395,6 +522,60 @@ for current_idx in "${!EXECUTION_ORDER[@]}"; do
|
||||||
else
|
else
|
||||||
if $exec_cmd; then
|
if $exec_cmd; then
|
||||||
log_success "Epic $epic_id completed"
|
log_success "Epic $epic_id completed"
|
||||||
|
|
||||||
|
# Run UAT validation if enabled
|
||||||
|
if [ "$UAT_GATE_ENABLED" = true ]; then
|
||||||
|
log_section "UAT Validation Gate: Epic $epic_id"
|
||||||
|
|
||||||
|
uat_cmd="$SCRIPT_DIR/uat-validate.sh $epic_id --gate-mode=$UAT_GATE_MODE --max-retries=$UAT_MAX_RETRIES"
|
||||||
|
|
||||||
|
if [ "$VERBOSE" = true ]; then
|
||||||
|
uat_cmd="$uat_cmd --verbose"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Running: $uat_cmd"
|
||||||
|
|
||||||
|
# Capture UAT result
|
||||||
|
uat_output=""
|
||||||
|
uat_exit_code=0
|
||||||
|
uat_output=$($uat_cmd 2>&1) || uat_exit_code=$?
|
||||||
|
|
||||||
|
echo "$uat_output" >> "$LOG_FILE"
|
||||||
|
|
||||||
|
# Parse result signals
|
||||||
|
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||||
|
log_success "UAT validation passed for Epic $epic_id"
|
||||||
|
|
||||||
|
# Update metrics file if it exists
|
||||||
|
epic_metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||||
|
if [ -f "$epic_metrics_file" ] && command -v yq >/dev/null 2>&1; then
|
||||||
|
yq -i '.validation.gate_executed = true' "$epic_metrics_file"
|
||||||
|
yq -i '.validation.gate_status = "PASS"' "$epic_metrics_file"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log_error "UAT validation failed for Epic $epic_id"
|
||||||
|
|
||||||
|
# Update metrics file if it exists
|
||||||
|
epic_metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||||
|
if [ -f "$epic_metrics_file" ] && command -v yq >/dev/null 2>&1; then
|
||||||
|
yq -i '.validation.gate_executed = true' "$epic_metrics_file"
|
||||||
|
yq -i '.validation.gate_status = "FAIL"' "$epic_metrics_file"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Extract fix attempts from output
|
||||||
|
fix_attempts=$(echo "$uat_output" | grep -oE "UAT_FIX_ATTEMPTS: [0-9]+" | grep -oE "[0-9]+" || echo "0")
|
||||||
|
[ "$VERBOSE" = true ] && log "Fix attempts: $fix_attempts"
|
||||||
|
|
||||||
|
if [ "$UAT_BLOCKING" = true ]; then
|
||||||
|
log_error "UAT blocking enabled - halting chain"
|
||||||
|
((FAILED_EPICS++))
|
||||||
|
break
|
||||||
|
else
|
||||||
|
log_warn "UAT blocking disabled - continuing to next epic"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
((COMPLETED_EPICS++))
|
((COMPLETED_EPICS++))
|
||||||
|
|
||||||
# Generate handoff for next epic
|
# Generate handoff for next epic
|
||||||
|
|
@ -408,6 +589,26 @@ for current_idx in "${!EXECUTION_ORDER[@]}"; do
|
||||||
log "Generating context handoff: Epic $epic_id → Epic $next_epic"
|
log "Generating context handoff: Epic $epic_id → Epic $next_epic"
|
||||||
|
|
||||||
story_count=${EPIC_STORIES_LIST[$current_idx]}
|
story_count=${EPIC_STORIES_LIST[$current_idx]}
|
||||||
|
|
||||||
|
# Determine UAT validation status for handoff
|
||||||
|
uat_status="Not executed"
|
||||||
|
uat_fix_info=""
|
||||||
|
if [ "$UAT_GATE_ENABLED" = true ]; then
|
||||||
|
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||||
|
uat_status="PASS"
|
||||||
|
local fix_count=$(echo "$uat_output" | grep -oE "UAT_FIX_ATTEMPTS: [0-9]+" | grep -oE "[0-9]+" || echo "0")
|
||||||
|
if [ "$fix_count" -gt 0 ]; then
|
||||||
|
uat_status="PASS (after $fix_count fix attempts)"
|
||||||
|
uat_fix_info="Self-healing fixes were applied. Review fix contexts at:
|
||||||
|
\`docs/sprint-artifacts/uat-fixes/epic-${epic_id}-fix-context-*.md\`"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
uat_status="FAIL (non-blocking)"
|
||||||
|
uat_fix_info="UAT validation failed but chain continued (non-blocking mode).
|
||||||
|
Review failures at: \`docs/sprint-artifacts/uat-fixes/epic-${epic_id}-fix-context-*.md\`"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
cat > "$handoff_file" << EOF
|
cat > "$handoff_file" << EOF
|
||||||
# Epic $epic_id → Epic $next_epic Handoff
|
# Epic $epic_id → Epic $next_epic Handoff
|
||||||
|
|
||||||
|
|
@ -418,6 +619,11 @@ $(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
|
||||||
Epic $epic_id has been completed. Key context for Epic $next_epic:
|
Epic $epic_id has been completed. Key context for Epic $next_epic:
|
||||||
|
|
||||||
|
### Implementation Status
|
||||||
|
- **Stories:** Completed via epic-execute workflow
|
||||||
|
- **UAT Validation:** $uat_status
|
||||||
|
- **Metrics:** \`$METRICS_DIR/epic-${epic_id}-metrics.yaml\`
|
||||||
|
|
||||||
### Patterns Established
|
### Patterns Established
|
||||||
- Review code changes in Epic $epic_id for established patterns
|
- Review code changes in Epic $epic_id for established patterns
|
||||||
- Check \`docs/stories/${epic_id}-*\` for implementation details
|
- Check \`docs/stories/${epic_id}-*\` for implementation details
|
||||||
|
|
@ -425,9 +631,17 @@ Epic $epic_id has been completed. Key context for Epic $next_epic:
|
||||||
### Files Modified
|
### Files Modified
|
||||||
$(git diff --name-only HEAD~${story_count} HEAD 2>/dev/null | head -20 || echo "Unable to determine - check git log")
|
$(git diff --name-only HEAD~${story_count} HEAD 2>/dev/null | head -20 || echo "Unable to determine - check git log")
|
||||||
|
|
||||||
|
### UAT Document
|
||||||
|
- Location: \`docs/uat/epic-${epic_id}-uat.md\`
|
||||||
|
- Contains test scenarios for regression testing
|
||||||
|
|
||||||
|
$([ -n "$uat_fix_info" ] && echo "### Fix Context
|
||||||
|
$uat_fix_info")
|
||||||
|
|
||||||
### Notes for Next Epic
|
### Notes for Next Epic
|
||||||
- Continue following patterns established in this epic
|
- Continue following patterns established in this epic
|
||||||
- Reference UAT document at \`docs/uat/epic-${epic_id}-uat.md\` for context
|
- Ensure changes don't break Epic $epic_id functionality
|
||||||
|
- Reference UAT document for integration points
|
||||||
|
|
||||||
EOF
|
EOF
|
||||||
log_success "Handoff saved to: $handoff_file"
|
log_success "Handoff saved to: $handoff_file"
|
||||||
|
|
@ -508,6 +722,108 @@ EOF
|
||||||
log_success "Combined UAT saved to: $combined_uat_file"
|
log_success "Combined UAT saved to: $combined_uat_file"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Phase 7: Generate Chain Execution Report
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
if [ "$GENERATE_REPORT" = true ] && [ "$DRY_RUN" = false ]; then
|
||||||
|
log_section "Generating Chain Execution Report"
|
||||||
|
|
||||||
|
# Check if metrics files exist
|
||||||
|
metrics_found=0
|
||||||
|
for epic_id in "${EPIC_IDS[@]}"; do
|
||||||
|
if [ -f "$METRICS_DIR/epic-${epic_id}-metrics.yaml" ]; then
|
||||||
|
((metrics_found++))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ $metrics_found -eq 0 ]; then
|
||||||
|
log_warn "No metrics files found - skipping report generation"
|
||||||
|
else
|
||||||
|
log "Found $metrics_found metrics files"
|
||||||
|
|
||||||
|
# Determine workflow path (installed vs source)
|
||||||
|
WORKFLOW_PATH=""
|
||||||
|
if [ -d "$BMAD_DIR/bmm/workflows/4-implementation/epic-chain" ]; then
|
||||||
|
WORKFLOW_PATH="$BMAD_DIR/bmm/workflows/4-implementation/epic-chain"
|
||||||
|
elif [ -d "$PROJECT_ROOT/src/modules/bmm/workflows/4-implementation/epic-chain" ]; then
|
||||||
|
WORKFLOW_PATH="$PROJECT_ROOT/src/modules/bmm/workflows/4-implementation/epic-chain"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build report generation prompt
|
||||||
|
report_prompt="You are Bob, the Scrum Master, generating a chain execution report.
|
||||||
|
|
||||||
|
## Your Task
|
||||||
|
|
||||||
|
Generate a comprehensive chain execution report for the completed epic chain.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
- Chain Plan: $CHAIN_PLAN_FILE
|
||||||
|
- Metrics Folder: $METRICS_DIR
|
||||||
|
- Output File: $CHAIN_REPORT_FILE
|
||||||
|
- Stories Location: $STORIES_DIR
|
||||||
|
- UAT Location: $UAT_DIR
|
||||||
|
- Epics Location: $EPICS_DIR
|
||||||
|
- Handoffs Location: $HANDOFF_DIR
|
||||||
|
|
||||||
|
## Epics in Chain
|
||||||
|
|
||||||
|
${EPIC_IDS[*]}
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
1. Read the chain plan file to understand the epic sequence
|
||||||
|
2. For each epic, load the metrics file from: $METRICS_DIR/epic-{id}-metrics.yaml
|
||||||
|
3. Aggregate metrics across all epics:
|
||||||
|
- Total duration
|
||||||
|
- Story counts (total, completed, failed, skipped)
|
||||||
|
- UAT gate results
|
||||||
|
- Issues encountered
|
||||||
|
4. Generate the report following the template structure
|
||||||
|
|
||||||
|
## Report Structure
|
||||||
|
|
||||||
|
Generate a markdown report with these sections:
|
||||||
|
- Executive Summary (status, counts, duration)
|
||||||
|
- Timeline (epic-by-epic execution details)
|
||||||
|
- What Was Built (brief per-epic summary)
|
||||||
|
- Issues Encountered (aggregated from metrics)
|
||||||
|
- UAT Validation Summary (gate results, fix attempts)
|
||||||
|
- Artifacts Generated (list generated files)
|
||||||
|
- Conclusion
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Write the report to: $CHAIN_REPORT_FILE
|
||||||
|
|
||||||
|
When complete, output exactly:
|
||||||
|
REPORT_GENERATED: $CHAIN_REPORT_FILE"
|
||||||
|
|
||||||
|
log "Invoking report generator..."
|
||||||
|
|
||||||
|
# Execute report generation
|
||||||
|
report_result=$(claude --dangerously-skip-permissions -p "$report_prompt" 2>&1) || true
|
||||||
|
|
||||||
|
echo "$report_result" >> "$LOG_FILE"
|
||||||
|
|
||||||
|
if echo "$report_result" | grep -q "REPORT_GENERATED"; then
|
||||||
|
log_success "Report generated: $CHAIN_REPORT_FILE"
|
||||||
|
|
||||||
|
# Stage report file
|
||||||
|
git add "$CHAIN_REPORT_FILE" 2>/dev/null || true
|
||||||
|
else
|
||||||
|
log_warn "Report generation may not have completed cleanly"
|
||||||
|
|
||||||
|
# If Claude didn't generate it, create a basic report
|
||||||
|
if [ ! -f "$CHAIN_REPORT_FILE" ]; then
|
||||||
|
log "Creating basic report from metrics..."
|
||||||
|
create_basic_report
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Summary
|
# Summary
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
@ -527,6 +843,10 @@ echo " Artifacts:"
|
||||||
echo " - Chain Plan: $CHAIN_PLAN_FILE"
|
echo " - Chain Plan: $CHAIN_PLAN_FILE"
|
||||||
echo " - Handoffs: $HANDOFF_DIR/"
|
echo " - Handoffs: $HANDOFF_DIR/"
|
||||||
echo " - UAT Documents: $UAT_DIR/"
|
echo " - UAT Documents: $UAT_DIR/"
|
||||||
|
echo " - Metrics: $METRICS_DIR/"
|
||||||
|
if [ -f "$CHAIN_REPORT_FILE" ]; then
|
||||||
|
echo " - Report: $CHAIN_REPORT_FILE"
|
||||||
|
fi
|
||||||
echo " - Log: $LOG_FILE"
|
echo " - Log: $LOG_FILE"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
|
@ -537,5 +857,12 @@ fi
|
||||||
|
|
||||||
log_success "All epics completed successfully"
|
log_success "All epics completed successfully"
|
||||||
echo ""
|
echo ""
|
||||||
echo "Next step: Review UAT documents and run manual testing"
|
if [ -f "$CHAIN_REPORT_FILE" ]; then
|
||||||
|
echo "Next steps:"
|
||||||
|
echo " 1. Review execution report: $CHAIN_REPORT_FILE"
|
||||||
|
echo " 2. Run UAT validation for each epic"
|
||||||
|
echo " 3. Execute manual test scenarios"
|
||||||
|
else
|
||||||
|
echo "Next step: Review UAT documents and run manual testing"
|
||||||
|
fi
|
||||||
echo ""
|
echo ""
|
||||||
|
|
|
||||||
|
|
@ -63,6 +63,121 @@ log_warn() {
|
||||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Metrics Functions
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
METRICS_DIR=""
|
||||||
|
METRICS_FILE=""
|
||||||
|
|
||||||
|
init_metrics() {
|
||||||
|
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||||
|
METRICS_FILE="$METRICS_DIR/epic-${EPIC_ID}-metrics.yaml"
|
||||||
|
mkdir -p "$METRICS_DIR"
|
||||||
|
|
||||||
|
local start_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
cat > "$METRICS_FILE" << EOF
|
||||||
|
epic_id: "$EPIC_ID"
|
||||||
|
execution:
|
||||||
|
start_time: "$start_time"
|
||||||
|
end_time: ""
|
||||||
|
duration_seconds: 0
|
||||||
|
stories:
|
||||||
|
total: 0
|
||||||
|
completed: 0
|
||||||
|
failed: 0
|
||||||
|
skipped: 0
|
||||||
|
validation:
|
||||||
|
gate_executed: false
|
||||||
|
gate_status: "PENDING"
|
||||||
|
fix_attempts: 0
|
||||||
|
issues: []
|
||||||
|
EOF
|
||||||
|
|
||||||
|
log "Metrics initialized: $METRICS_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
update_story_metrics() {
|
||||||
|
local status="$1" # completed|failed|skipped
|
||||||
|
|
||||||
|
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if yq is available for YAML manipulation
|
||||||
|
if command -v yq >/dev/null 2>&1; then
|
||||||
|
case "$status" in
|
||||||
|
completed) yq -i '.stories.completed += 1' "$METRICS_FILE" ;;
|
||||||
|
failed) yq -i '.stories.failed += 1' "$METRICS_FILE" ;;
|
||||||
|
skipped) yq -i '.stories.skipped += 1' "$METRICS_FILE" ;;
|
||||||
|
esac
|
||||||
|
else
|
||||||
|
# Fallback: log warning (metrics will be finalized at end)
|
||||||
|
[ "$VERBOSE" = true ] && log_warn "yq not found - metrics update deferred"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
add_metrics_issue() {
|
||||||
|
local story_id="$1"
|
||||||
|
local issue_type="$2"
|
||||||
|
local message="$3"
|
||||||
|
|
||||||
|
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
if command -v yq >/dev/null 2>&1; then
|
||||||
|
yq -i ".issues += [{\"story\": \"$story_id\", \"type\": \"$issue_type\", \"message\": \"$message\", \"timestamp\": \"$timestamp\"}]" "$METRICS_FILE"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
finalize_metrics() {
|
||||||
|
local total_stories="$1"
|
||||||
|
local completed="$2"
|
||||||
|
local failed="$3"
|
||||||
|
local skipped="$4"
|
||||||
|
local duration="$5"
|
||||||
|
|
||||||
|
if [ -z "$METRICS_FILE" ] || [ ! -f "$METRICS_FILE" ]; then
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
local end_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
if command -v yq >/dev/null 2>&1; then
|
||||||
|
yq -i ".execution.end_time = \"$end_time\"" "$METRICS_FILE"
|
||||||
|
yq -i ".execution.duration_seconds = $duration" "$METRICS_FILE"
|
||||||
|
yq -i ".stories.total = $total_stories" "$METRICS_FILE"
|
||||||
|
yq -i ".stories.completed = $completed" "$METRICS_FILE"
|
||||||
|
yq -i ".stories.failed = $failed" "$METRICS_FILE"
|
||||||
|
yq -i ".stories.skipped = $skipped" "$METRICS_FILE"
|
||||||
|
else
|
||||||
|
# Fallback: rewrite the file with final values
|
||||||
|
cat > "$METRICS_FILE" << EOF
|
||||||
|
epic_id: "$EPIC_ID"
|
||||||
|
execution:
|
||||||
|
start_time: "$EPIC_START_TIME"
|
||||||
|
end_time: "$end_time"
|
||||||
|
duration_seconds: $duration
|
||||||
|
stories:
|
||||||
|
total: $total_stories
|
||||||
|
completed: $completed
|
||||||
|
failed: $failed
|
||||||
|
skipped: $skipped
|
||||||
|
validation:
|
||||||
|
gate_executed: false
|
||||||
|
gate_status: "PENDING"
|
||||||
|
fix_attempts: 0
|
||||||
|
issues: []
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Metrics finalized: $METRICS_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Argument Parsing
|
# Argument Parsing
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
@ -142,6 +257,11 @@ log "Project root: $PROJECT_ROOT"
|
||||||
mkdir -p "$UAT_DIR"
|
mkdir -p "$UAT_DIR"
|
||||||
mkdir -p "$SPRINTS_DIR"
|
mkdir -p "$SPRINTS_DIR"
|
||||||
|
|
||||||
|
# Initialize metrics collection
|
||||||
|
EPIC_START_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
EPIC_START_SECONDS=$(date +%s)
|
||||||
|
init_metrics
|
||||||
|
|
||||||
# Find epic file (supports both epic-39-*.md and epic-039-*.md formats)
|
# Find epic file (supports both epic-39-*.md and epic-039-*.md formats)
|
||||||
EPIC_FILE=""
|
EPIC_FILE=""
|
||||||
# Pad epic ID with leading zero for 3-digit format (e.g., 40 -> 040)
|
# Pad epic ID with leading zero for 3-digit format (e.g., 40 -> 040)
|
||||||
|
|
@ -554,6 +674,7 @@ for story_file in "${STORIES[@]}"; do
|
||||||
else
|
else
|
||||||
log_warn "Skipping $story_id (waiting for $START_FROM)"
|
log_warn "Skipping $story_id (waiting for $START_FROM)"
|
||||||
((SKIPPED++))
|
((SKIPPED++))
|
||||||
|
update_story_metrics "skipped"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
@ -563,6 +684,7 @@ for story_file in "${STORIES[@]}"; do
|
||||||
if grep -q "^Status:.*Done" "$story_file" 2>/dev/null; then
|
if grep -q "^Status:.*Done" "$story_file" 2>/dev/null; then
|
||||||
log_warn "Skipping $story_id (Status: Done)"
|
log_warn "Skipping $story_id (Status: Done)"
|
||||||
((SKIPPED++))
|
((SKIPPED++))
|
||||||
|
update_story_metrics "skipped"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
@ -576,22 +698,27 @@ for story_file in "${STORIES[@]}"; do
|
||||||
if ! execute_dev_phase "$story_file"; then
|
if ! execute_dev_phase "$story_file"; then
|
||||||
log_error "Dev phase failed for $story_id"
|
log_error "Dev phase failed for $story_id"
|
||||||
((FAILED++))
|
((FAILED++))
|
||||||
|
update_story_metrics "failed"
|
||||||
|
add_metrics_issue "$story_id" "dev_phase_failed" "Development phase did not complete"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# REVIEW PHASE (Context 2 - Fresh)
|
# REVIEW PHASE (Context 2 - Fresh)
|
||||||
if [ "$SKIP_REVIEW" = false ]; then
|
if [ "$SKIP_REVIEW" = false ]; then
|
||||||
if ! execute_review_phase "$story_file"; then
|
if ! execute_review_phase "$story_file"; then
|
||||||
log_error "Review phase failed for $story_id"
|
log_error "Review phase failed for $story_id"
|
||||||
((FAILED++))
|
((FAILED++))
|
||||||
|
update_story_metrics "failed"
|
||||||
|
add_metrics_issue "$story_id" "review_failed" "Code review phase failed"
|
||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# COMMIT
|
# COMMIT
|
||||||
commit_story "$story_id"
|
commit_story "$story_id"
|
||||||
|
|
||||||
((COMPLETED++))
|
((COMPLETED++))
|
||||||
|
update_story_metrics "completed"
|
||||||
log_success "Story complete: $story_id ($COMPLETED/${#STORIES[@]})"
|
log_success "Story complete: $story_id ($COMPLETED/${#STORIES[@]})"
|
||||||
done
|
done
|
||||||
|
|
||||||
|
|
@ -613,6 +740,9 @@ generate_uat
|
||||||
END_TIME=$(date +%s)
|
END_TIME=$(date +%s)
|
||||||
DURATION=$((END_TIME - START_TIME))
|
DURATION=$((END_TIME - START_TIME))
|
||||||
|
|
||||||
|
# Finalize metrics with final counts
|
||||||
|
finalize_metrics "${#STORIES[@]}" "$COMPLETED" "$FAILED" "$SKIPPED" "$DURATION"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
log "=========================================="
|
log "=========================================="
|
||||||
log "EPIC EXECUTION COMPLETE"
|
log "EPIC EXECUTION COMPLETE"
|
||||||
|
|
@ -628,6 +758,7 @@ echo ""
|
||||||
echo " Deliverables:"
|
echo " Deliverables:"
|
||||||
echo " - Stories: $STORIES_DIR/"
|
echo " - Stories: $STORIES_DIR/"
|
||||||
echo " - UAT: $UAT_DIR/epic-${EPIC_ID}-uat.md"
|
echo " - UAT: $UAT_DIR/epic-${EPIC_ID}-uat.md"
|
||||||
|
echo " - Metrics: $METRICS_FILE"
|
||||||
echo " - Log: $LOG_FILE"
|
echo " - Log: $LOG_FILE"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,827 @@
|
||||||
|
#!/bin/bash
|
||||||
|
#
|
||||||
|
# BMAD UAT Validate - Automated UAT Scenario Execution with Self-Healing Fix Loop
|
||||||
|
#
|
||||||
|
# Usage: ./uat-validate.sh <epic-id> [options]
|
||||||
|
#
|
||||||
|
# Options:
|
||||||
|
# --gate-mode=MODE Validation mode: quick|full|skip (default: quick)
|
||||||
|
# --max-retries=N Max fix attempts before halt (default: 2)
|
||||||
|
# --skip-manual Skip manual-only scenarios (default: skip)
|
||||||
|
# --verbose Show detailed output
|
||||||
|
# --dry-run Show what would be executed without running
|
||||||
|
# --timeout=SECONDS Timeout per scenario (default: 30)
|
||||||
|
#
|
||||||
|
# Exit Codes:
|
||||||
|
# 0 - UAT PASS (all automatable scenarios passed)
|
||||||
|
# 1 - UAT FAIL (fixable, retries remain or self-heal succeeded)
|
||||||
|
# 2 - UAT FAIL (max retries exceeded)
|
||||||
|
#
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 1: Configuration
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||||
|
BMAD_DIR="$PROJECT_ROOT/.bmad"
|
||||||
|
|
||||||
|
UAT_DIR="$PROJECT_ROOT/docs/uat"
|
||||||
|
SPRINT_ARTIFACTS_DIR="$PROJECT_ROOT/docs/sprint-artifacts"
|
||||||
|
METRICS_DIR="$SPRINT_ARTIFACTS_DIR/metrics"
|
||||||
|
FIX_DIR="$SPRINT_ARTIFACTS_DIR/uat-fixes"
|
||||||
|
STORIES_DIR="$PROJECT_ROOT/docs/stories"
|
||||||
|
|
||||||
|
LOG_FILE="/tmp/bmad-uat-validate-$$.log"
|
||||||
|
|
||||||
|
# Default configuration
|
||||||
|
UAT_GATE_MODE="quick"
|
||||||
|
MAX_RETRIES=2
|
||||||
|
SKIP_MANUAL=true
|
||||||
|
VERBOSE=false
|
||||||
|
DRY_RUN=false
|
||||||
|
TIMEOUT_SECONDS=30
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
CYAN='\033[0;36m'
|
||||||
|
BOLD='\033[1m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 2: Helper Functions
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo -e "${BLUE}[UAT]${NC} $1"
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_success() {
|
||||||
|
echo -e "${GREEN}[PASS]${NC} $1"
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [PASS] $1" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_error() {
|
||||||
|
echo -e "${RED}[FAIL]${NC} $1"
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [FAIL] $1" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_warn() {
|
||||||
|
echo -e "${YELLOW}[!]${NC} $1"
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] [WARN] $1" >> "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_section() {
|
||||||
|
echo ""
|
||||||
|
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||||
|
echo -e "${BOLD} $1${NC}"
|
||||||
|
echo -e "${BOLD}───────────────────────────────────────────────────────────${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
log_header() {
|
||||||
|
echo ""
|
||||||
|
echo -e "${CYAN}${BOLD}═══════════════════════════════════════════════════════════${NC}"
|
||||||
|
echo -e "${CYAN}${BOLD} $1${NC}"
|
||||||
|
echo -e "${CYAN}${BOLD}═══════════════════════════════════════════════════════════${NC}"
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 3: Argument Parsing
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
EPIC_ID=""
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case $1 in
|
||||||
|
--gate-mode=*)
|
||||||
|
UAT_GATE_MODE="${1#*=}"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--max-retries=*)
|
||||||
|
MAX_RETRIES="${1#*=}"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--skip-manual)
|
||||||
|
SKIP_MANUAL=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--include-manual)
|
||||||
|
SKIP_MANUAL=false
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--verbose)
|
||||||
|
VERBOSE=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--dry-run)
|
||||||
|
DRY_RUN=true
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
--timeout=*)
|
||||||
|
TIMEOUT_SECONDS="${1#*=}"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
-*)
|
||||||
|
echo "Unknown option: $1"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
EPIC_ID="$1"
|
||||||
|
shift
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$EPIC_ID" ]; then
|
||||||
|
echo "Usage: $0 <epic-id> [options]"
|
||||||
|
echo ""
|
||||||
|
echo "Options:"
|
||||||
|
echo " --gate-mode=MODE Validation mode: quick|full|skip (default: quick)"
|
||||||
|
echo " --max-retries=N Max fix attempts before halt (default: 2)"
|
||||||
|
echo " --skip-manual Skip manual-only scenarios (default)"
|
||||||
|
echo " --include-manual Include manual scenarios in checklist"
|
||||||
|
echo " --verbose Detailed output"
|
||||||
|
echo " --dry-run Show what would be executed"
|
||||||
|
echo " --timeout=SECONDS Timeout per scenario (default: 30)"
|
||||||
|
echo ""
|
||||||
|
echo "Exit Codes:"
|
||||||
|
echo " 0 - UAT PASS"
|
||||||
|
echo " 1 - UAT FAIL (fixable)"
|
||||||
|
echo " 2 - UAT FAIL (max retries exceeded)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Validate gate mode
|
||||||
|
if [[ ! "$UAT_GATE_MODE" =~ ^(quick|full|skip)$ ]]; then
|
||||||
|
echo "Invalid gate mode: $UAT_GATE_MODE"
|
||||||
|
echo "Valid modes: quick, full, skip"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 4: UAT Document Loading
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
load_uat_document() {
|
||||||
|
local epic_id="$1"
|
||||||
|
|
||||||
|
# Find UAT document (try multiple patterns)
|
||||||
|
UAT_FILE=""
|
||||||
|
for pattern in "epic-${epic_id}-uat.md" "epic-0${epic_id}-uat.md" "${epic_id}-uat.md"; do
|
||||||
|
found=$(find "$UAT_DIR" -name "$pattern" 2>/dev/null | head -1)
|
||||||
|
if [ -n "$found" ]; then
|
||||||
|
UAT_FILE="$found"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [ -z "$UAT_FILE" ] || [ ! -f "$UAT_FILE" ]; then
|
||||||
|
log_error "UAT document not found for Epic $epic_id"
|
||||||
|
log_error "Searched in: $UAT_DIR"
|
||||||
|
log_error "Expected: epic-${epic_id}-uat.md"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Found UAT document: $UAT_FILE"
|
||||||
|
|
||||||
|
# Validate structure - check for scenarios section
|
||||||
|
if ! grep -qE "^##.*[Ss]cenario|^##.*[Tt]est|^##.*[Cc]riteria" "$UAT_FILE"; then
|
||||||
|
log_warn "UAT document may not have standard scenario sections"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Count scenario blocks (lines starting with ### or numbered items under Test Scenarios)
|
||||||
|
SCENARIO_COUNT=$(grep -cE "^###|^[0-9]+\." "$UAT_FILE" 2>/dev/null || echo "0")
|
||||||
|
log "Found approximately $SCENARIO_COUNT scenario entries"
|
||||||
|
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 5: Scenario Classification
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
# Arrays to store classified scenarios
|
||||||
|
declare -a AUTOMATABLE_SCENARIOS
|
||||||
|
declare -a SEMI_AUTO_SCENARIOS
|
||||||
|
declare -a MANUAL_SCENARIOS
|
||||||
|
|
||||||
|
classify_scenarios() {
|
||||||
|
local uat_file="$1"
|
||||||
|
|
||||||
|
# Reset arrays
|
||||||
|
AUTOMATABLE_SCENARIOS=()
|
||||||
|
SEMI_AUTO_SCENARIOS=()
|
||||||
|
MANUAL_SCENARIOS=()
|
||||||
|
|
||||||
|
# Read the UAT file and extract scenario blocks
|
||||||
|
local current_scenario=""
|
||||||
|
local current_name=""
|
||||||
|
local in_scenario=false
|
||||||
|
local scenario_num=0
|
||||||
|
|
||||||
|
while IFS= read -r line; do
|
||||||
|
# Detect scenario headers (### or numbered items)
|
||||||
|
if [[ "$line" =~ ^###[[:space:]]*(.*) ]] || [[ "$line" =~ ^([0-9]+)\.[[:space:]]+(.*) ]]; then
|
||||||
|
# Save previous scenario if exists
|
||||||
|
if [ -n "$current_scenario" ]; then
|
||||||
|
classify_single_scenario "$scenario_num" "$current_name" "$current_scenario"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Start new scenario
|
||||||
|
((scenario_num++))
|
||||||
|
if [[ "$line" =~ ^###[[:space:]]*(.*) ]]; then
|
||||||
|
current_name="${BASH_REMATCH[1]}"
|
||||||
|
else
|
||||||
|
current_name="${BASH_REMATCH[2]}"
|
||||||
|
fi
|
||||||
|
current_scenario="$line"
|
||||||
|
in_scenario=true
|
||||||
|
elif [ "$in_scenario" = true ]; then
|
||||||
|
# Continue accumulating scenario content
|
||||||
|
current_scenario+=$'\n'"$line"
|
||||||
|
fi
|
||||||
|
done < "$uat_file"
|
||||||
|
|
||||||
|
# Handle last scenario
|
||||||
|
if [ -n "$current_scenario" ]; then
|
||||||
|
classify_single_scenario "$scenario_num" "$current_name" "$current_scenario"
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Classification complete:"
|
||||||
|
log " Automatable: ${#AUTOMATABLE_SCENARIOS[@]}"
|
||||||
|
log " Semi-auto: ${#SEMI_AUTO_SCENARIOS[@]}"
|
||||||
|
log " Manual: ${#MANUAL_SCENARIOS[@]}"
|
||||||
|
}
|
||||||
|
|
||||||
|
classify_single_scenario() {
|
||||||
|
local id="$1"
|
||||||
|
local name="$2"
|
||||||
|
local content="$3"
|
||||||
|
|
||||||
|
# Check for automatable indicators
|
||||||
|
if echo "$content" | grep -qiE 'npx|npm run|yarn|node |curl |wget |pytest|jest|vitest|--version|/health|/api/|exit code|returns [0-9]|\.sh |bash '; then
|
||||||
|
# Extract command from code block if present
|
||||||
|
local cmd=""
|
||||||
|
cmd=$(echo "$content" | grep -oE '`[^`]+`' | head -1 | tr -d '`')
|
||||||
|
if [ -z "$cmd" ]; then
|
||||||
|
cmd=$(echo "$content" | grep -oE 'npx [a-zA-Z0-9_-]+.*|npm run [a-zA-Z0-9_:-]+.*|curl [^[:space:]]+.*' | head -1)
|
||||||
|
fi
|
||||||
|
AUTOMATABLE_SCENARIOS+=("$id|$name|$cmd")
|
||||||
|
[ "$VERBOSE" = true ] && log " [AUTO] Scenario $id: $name"
|
||||||
|
|
||||||
|
# Check for semi-automated indicators
|
||||||
|
elif echo "$content" | grep -qiE 'test-send|email|inbox|check your|verify.*manually|setup.*first|start.*server'; then
|
||||||
|
SEMI_AUTO_SCENARIOS+=("$id|$name|")
|
||||||
|
[ "$VERBOSE" = true ] && log " [SEMI] Scenario $id: $name"
|
||||||
|
|
||||||
|
# Everything else is manual
|
||||||
|
else
|
||||||
|
MANUAL_SCENARIOS+=("$id|$name|")
|
||||||
|
[ "$VERBOSE" = true ] && log " [MANUAL] Scenario $id: $name"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 6: Scenario Execution
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
# Arrays to store results
|
||||||
|
declare -a PASSED_SCENARIOS
|
||||||
|
declare -a FAILED_SCENARIOS
|
||||||
|
declare -a FAILED_DETAILS
|
||||||
|
|
||||||
|
execute_scenarios() {
|
||||||
|
local gate_mode="$1"
|
||||||
|
|
||||||
|
# Reset results
|
||||||
|
PASSED_SCENARIOS=()
|
||||||
|
FAILED_SCENARIOS=()
|
||||||
|
FAILED_DETAILS=()
|
||||||
|
|
||||||
|
# Skip mode - pass automatically
|
||||||
|
if [ "$gate_mode" = "skip" ]; then
|
||||||
|
log "Gate mode: skip - bypassing scenario execution"
|
||||||
|
echo "UAT_GATE_RESULT: PASS"
|
||||||
|
echo "UAT_SCENARIOS_PASSED: 0/0 (skipped)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Select scenarios based on gate mode
|
||||||
|
local scenarios_to_run=()
|
||||||
|
if [ "$gate_mode" = "quick" ]; then
|
||||||
|
scenarios_to_run=("${AUTOMATABLE_SCENARIOS[@]}")
|
||||||
|
elif [ "$gate_mode" = "full" ]; then
|
||||||
|
scenarios_to_run=("${AUTOMATABLE_SCENARIOS[@]}" "${SEMI_AUTO_SCENARIOS[@]}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ${#scenarios_to_run[@]} -eq 0 ]; then
|
||||||
|
log_warn "No automatable scenarios found - gate passes by default"
|
||||||
|
echo "UAT_GATE_RESULT: PASS"
|
||||||
|
echo "UAT_SCENARIOS_PASSED: 0/0 (none automatable)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_section "Executing ${#scenarios_to_run[@]} Scenarios"
|
||||||
|
|
||||||
|
for scenario_entry in "${scenarios_to_run[@]}"; do
|
||||||
|
IFS='|' read -r scenario_id scenario_name scenario_cmd <<< "$scenario_entry"
|
||||||
|
|
||||||
|
execute_single_scenario "$scenario_id" "$scenario_name" "$scenario_cmd"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Report results
|
||||||
|
local total=${#scenarios_to_run[@]}
|
||||||
|
local passed=${#PASSED_SCENARIOS[@]}
|
||||||
|
local failed=${#FAILED_SCENARIOS[@]}
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
log "Results: $passed/$total passed"
|
||||||
|
|
||||||
|
if [ $failed -eq 0 ]; then
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
execute_single_scenario() {
|
||||||
|
local scenario_id="$1"
|
||||||
|
local scenario_name="$2"
|
||||||
|
local scenario_cmd="$3"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
log "Scenario $scenario_id: $scenario_name"
|
||||||
|
|
||||||
|
# If no command extracted, try to infer from name
|
||||||
|
if [ -z "$scenario_cmd" ]; then
|
||||||
|
log_warn " No command detected - marking as manual verification needed"
|
||||||
|
FAILED_SCENARIOS+=("$scenario_id")
|
||||||
|
FAILED_DETAILS+=("$scenario_id|$scenario_name|No automatable command found|manual|1")
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$VERBOSE" = true ]; then
|
||||||
|
log " Command: $scenario_cmd"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$DRY_RUN" = true ]; then
|
||||||
|
echo " [DRY RUN] Would execute: $scenario_cmd"
|
||||||
|
PASSED_SCENARIOS+=("$scenario_id")
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Execute with timeout
|
||||||
|
local start_time=$(date +%s%N)
|
||||||
|
local output=""
|
||||||
|
local exit_code=0
|
||||||
|
local stderr_file="/tmp/uat-stderr-$$.txt"
|
||||||
|
|
||||||
|
# Run command with timeout
|
||||||
|
set +e
|
||||||
|
if command -v timeout >/dev/null 2>&1; then
|
||||||
|
output=$(timeout "$TIMEOUT_SECONDS" bash -c "$scenario_cmd" 2>"$stderr_file")
|
||||||
|
exit_code=$?
|
||||||
|
# timeout returns 124 on timeout
|
||||||
|
if [ $exit_code -eq 124 ]; then
|
||||||
|
exit_code=124
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
# macOS fallback using perl
|
||||||
|
output=$(perl -e 'alarm shift @ARGV; exec @ARGV' "$TIMEOUT_SECONDS" bash -c "$scenario_cmd" 2>"$stderr_file")
|
||||||
|
exit_code=$?
|
||||||
|
fi
|
||||||
|
set -e
|
||||||
|
|
||||||
|
local end_time=$(date +%s%N)
|
||||||
|
local duration_ms=$(( (end_time - start_time) / 1000000 ))
|
||||||
|
|
||||||
|
local stderr=""
|
||||||
|
[ -f "$stderr_file" ] && stderr=$(cat "$stderr_file")
|
||||||
|
rm -f "$stderr_file"
|
||||||
|
|
||||||
|
# Evaluate result
|
||||||
|
if [ $exit_code -eq 0 ]; then
|
||||||
|
log_success " Scenario $scenario_id: PASS (${duration_ms}ms)"
|
||||||
|
PASSED_SCENARIOS+=("$scenario_id")
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Scenario $scenario_id PASS: $scenario_cmd" >> "$LOG_FILE"
|
||||||
|
elif [ $exit_code -eq 124 ]; then
|
||||||
|
log_error " Scenario $scenario_id: FAIL (timeout after ${TIMEOUT_SECONDS}s)"
|
||||||
|
FAILED_SCENARIOS+=("$scenario_id")
|
||||||
|
FAILED_DETAILS+=("$scenario_id|$scenario_name|$scenario_cmd|timeout|$exit_code|$output|$stderr")
|
||||||
|
else
|
||||||
|
log_error " Scenario $scenario_id: FAIL (exit code $exit_code)"
|
||||||
|
if [ -n "$stderr" ] && [ "$VERBOSE" = true ]; then
|
||||||
|
echo " Error: $stderr"
|
||||||
|
fi
|
||||||
|
FAILED_SCENARIOS+=("$scenario_id")
|
||||||
|
FAILED_DETAILS+=("$scenario_id|$scenario_name|$scenario_cmd|error|$exit_code|$output|$stderr")
|
||||||
|
fi
|
||||||
|
|
||||||
|
return $exit_code
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 7: Gate Evaluation
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
evaluate_gate() {
|
||||||
|
local total=${#AUTOMATABLE_SCENARIOS[@]}
|
||||||
|
local passed=${#PASSED_SCENARIOS[@]}
|
||||||
|
local failed=${#FAILED_SCENARIOS[@]}
|
||||||
|
|
||||||
|
log_section "Gate Evaluation"
|
||||||
|
|
||||||
|
if [ $failed -eq 0 ]; then
|
||||||
|
log_success "All automatable scenarios passed"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
log_error "$failed scenario(s) failed"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 8: Self-Healing Loop
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
generate_fix_context() {
|
||||||
|
local epic_id="$1"
|
||||||
|
local attempt="$2"
|
||||||
|
|
||||||
|
mkdir -p "$FIX_DIR"
|
||||||
|
|
||||||
|
local fix_file="$FIX_DIR/epic-${epic_id}-fix-context-${attempt}.md"
|
||||||
|
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
# Find template
|
||||||
|
local template="$PROJECT_ROOT/src/modules/bmm/workflows/5-validation/uat-validate/uat-fix-context-template.md"
|
||||||
|
|
||||||
|
if [ -f "$template" ]; then
|
||||||
|
# Render template with basic variable substitution
|
||||||
|
sed -e "s/{epic_id}/$epic_id/g" \
|
||||||
|
-e "s/{attempt}/$attempt/g" \
|
||||||
|
-e "s/{timestamp}/$timestamp/g" \
|
||||||
|
-e "s/{max_retries}/$MAX_RETRIES/g" \
|
||||||
|
-e "s/{next_attempt}/$((attempt + 1))/g" \
|
||||||
|
-e "s/{failure_count}/${#FAILED_SCENARIOS[@]}/g" \
|
||||||
|
-e "s|{uat_doc_path}|$UAT_FILE|g" \
|
||||||
|
"$template" > "$fix_file"
|
||||||
|
else
|
||||||
|
# Create minimal fix context without template
|
||||||
|
cat > "$fix_file" << EOF
|
||||||
|
# UAT Fix Context - Epic $epic_id (Attempt $attempt)
|
||||||
|
|
||||||
|
**Generated:** $timestamp
|
||||||
|
**Epic:** $epic_id
|
||||||
|
**Gate Result:** FAIL (${#PASSED_SCENARIOS[@]}/${#AUTOMATABLE_SCENARIOS[@]} scenarios passed)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
This document contains the context needed to fix UAT failures for Epic $epic_id.
|
||||||
|
|
||||||
|
**Failures to fix:** ${#FAILED_SCENARIOS[@]}
|
||||||
|
**Fix attempt:** $attempt of $MAX_RETRIES
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Append failed scenarios details
|
||||||
|
echo "" >> "$fix_file"
|
||||||
|
echo "## Failed Scenarios" >> "$fix_file"
|
||||||
|
echo "" >> "$fix_file"
|
||||||
|
|
||||||
|
for detail in "${FAILED_DETAILS[@]}"; do
|
||||||
|
IFS='|' read -r scenario_id scenario_name cmd error_type exit_code output stderr <<< "$detail"
|
||||||
|
|
||||||
|
cat >> "$fix_file" << EOF
|
||||||
|
### Scenario $scenario_id: $scenario_name
|
||||||
|
|
||||||
|
**Command Executed:**
|
||||||
|
\`\`\`bash
|
||||||
|
$cmd
|
||||||
|
\`\`\`
|
||||||
|
|
||||||
|
**Error Type:** $error_type
|
||||||
|
**Exit Code:** $exit_code
|
||||||
|
|
||||||
|
**Output:**
|
||||||
|
\`\`\`
|
||||||
|
$output
|
||||||
|
\`\`\`
|
||||||
|
|
||||||
|
**Error Output:**
|
||||||
|
\`\`\`
|
||||||
|
$stderr
|
||||||
|
\`\`\`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
# Add context references section
|
||||||
|
cat >> "$fix_file" << EOF
|
||||||
|
|
||||||
|
## Context References
|
||||||
|
|
||||||
|
The following files provide additional context for fixing these failures:
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| \`$UAT_FILE\` | Full UAT document with all scenarios |
|
||||||
|
| \`$STORIES_DIR/${epic_id}-*\` | Story files with acceptance criteria |
|
||||||
|
| \`$METRICS_DIR/epic-${epic_id}-metrics.yaml\` | Execution metrics |
|
||||||
|
|
||||||
|
## Fix Instructions
|
||||||
|
|
||||||
|
Address the failures above in priority order. For each fix:
|
||||||
|
|
||||||
|
1. **Analyze** - Understand why the scenario failed
|
||||||
|
2. **Locate** - Find the relevant code files
|
||||||
|
3. **Fix** - Implement the minimum change to resolve the failure
|
||||||
|
4. **Verify** - Run the scenario command locally to confirm fix
|
||||||
|
5. **Commit** - Use message format: \`fix(epic-$epic_id): {description}\`
|
||||||
|
|
||||||
|
### Constraints
|
||||||
|
|
||||||
|
- Only fix the identified failures - do not refactor unrelated code
|
||||||
|
- Run the specific failing commands to verify each fix
|
||||||
|
- Run project tests after all fixes: \`npm test\`
|
||||||
|
- If a fix requires changes that would break other scenarios, document the tradeoff
|
||||||
|
|
||||||
|
## After Fixing
|
||||||
|
|
||||||
|
Once all fixes are committed, the UAT validation will automatically re-run.
|
||||||
|
|
||||||
|
- **If all pass:** Epic continues to next phase
|
||||||
|
- **If failures remain:** Another fix context will be generated (attempt $((attempt + 1)))
|
||||||
|
- **If max retries exceeded:** Chain halts for human intervention
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Generated by UAT Validate Workflow*
|
||||||
|
*BMAD Method - Epic Chain Self-Healing*
|
||||||
|
*Fix Context: epic-${epic_id}-fix-context-${attempt}.md*
|
||||||
|
EOF
|
||||||
|
|
||||||
|
log "Fix context generated: $fix_file"
|
||||||
|
echo "$fix_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_quick_dev_fix() {
|
||||||
|
local fix_context_file="$1"
|
||||||
|
local epic_id="$2"
|
||||||
|
local attempt="$3"
|
||||||
|
|
||||||
|
log "Spawning quick-dev fix session (attempt $attempt/$MAX_RETRIES)"
|
||||||
|
|
||||||
|
local fix_prompt="You are Barry, the Quick Flow Solo Dev.
|
||||||
|
|
||||||
|
Load and process this fix context document:
|
||||||
|
$fix_context_file
|
||||||
|
|
||||||
|
Your task:
|
||||||
|
1. Read the failed scenarios and error details from the fix context
|
||||||
|
2. Analyze root cause for each failure
|
||||||
|
3. Implement targeted fixes
|
||||||
|
4. Run the failing commands to verify fixes
|
||||||
|
5. Stage changes: git add -A
|
||||||
|
6. Commit with message: fix(epic-${epic_id}): UAT fix #${attempt}
|
||||||
|
|
||||||
|
Constraints:
|
||||||
|
- Only fix the identified failures
|
||||||
|
- Do not refactor unrelated code
|
||||||
|
- Run tests after fixes
|
||||||
|
|
||||||
|
When done, output exactly:
|
||||||
|
FIX_COMPLETE: {number_fixed}/${#FAILED_SCENARIOS[@]}"
|
||||||
|
|
||||||
|
if [ "$DRY_RUN" = true ]; then
|
||||||
|
echo "[DRY RUN] Would spawn Claude for fixes with prompt:"
|
||||||
|
echo " Fix context: $fix_context_file"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Execute in isolated context
|
||||||
|
local result
|
||||||
|
result=$(claude --dangerously-skip-permissions -p "$fix_prompt" 2>&1) || true
|
||||||
|
|
||||||
|
echo "$result" >> "$LOG_FILE"
|
||||||
|
|
||||||
|
if echo "$result" | grep -q "FIX_COMPLETE"; then
|
||||||
|
log_success "Quick-dev fix session completed"
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
log_warn "Quick-dev fix session may not have completed cleanly"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
self_healing_loop() {
|
||||||
|
local epic_id="$1"
|
||||||
|
local attempt=0
|
||||||
|
|
||||||
|
while [ $attempt -lt $MAX_RETRIES ]; do
|
||||||
|
((attempt++))
|
||||||
|
|
||||||
|
log_section "Self-Healing Fix Loop (Attempt $attempt/$MAX_RETRIES)"
|
||||||
|
|
||||||
|
# Generate fix context
|
||||||
|
local fix_file
|
||||||
|
fix_file=$(generate_fix_context "$epic_id" "$attempt")
|
||||||
|
|
||||||
|
# Run quick-dev fix
|
||||||
|
if ! run_quick_dev_fix "$fix_file" "$epic_id" "$attempt"; then
|
||||||
|
log_warn "Fix attempt $attempt may have issues"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Re-run validation
|
||||||
|
log "Re-validating after fix attempt $attempt..."
|
||||||
|
|
||||||
|
# Reset and re-execute
|
||||||
|
PASSED_SCENARIOS=()
|
||||||
|
FAILED_SCENARIOS=()
|
||||||
|
FAILED_DETAILS=()
|
||||||
|
|
||||||
|
if execute_scenarios "$UAT_GATE_MODE"; then
|
||||||
|
log_success "UAT passed after fix attempt $attempt"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
log_warn "UAT still failing after attempt $attempt"
|
||||||
|
done
|
||||||
|
|
||||||
|
log_error "Max retries ($MAX_RETRIES) exceeded"
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Section 9: Output Signals and Metrics
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
update_metrics() {
|
||||||
|
local epic_id="$1"
|
||||||
|
local gate_status="$2"
|
||||||
|
local fix_attempts="$3"
|
||||||
|
|
||||||
|
mkdir -p "$METRICS_DIR"
|
||||||
|
|
||||||
|
local metrics_file="$METRICS_DIR/epic-${epic_id}-metrics.yaml"
|
||||||
|
local timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
||||||
|
|
||||||
|
# Check if yq is available for YAML manipulation
|
||||||
|
if command -v yq >/dev/null 2>&1; then
|
||||||
|
if [ -f "$metrics_file" ]; then
|
||||||
|
yq -i ".validation.gate_executed = true" "$metrics_file"
|
||||||
|
yq -i ".validation.gate_status = \"$gate_status\"" "$metrics_file"
|
||||||
|
yq -i ".validation.fix_attempts = $fix_attempts" "$metrics_file"
|
||||||
|
yq -i ".validation.scenarios_passed = ${#PASSED_SCENARIOS[@]}" "$metrics_file"
|
||||||
|
yq -i ".validation.scenarios_failed = ${#FAILED_SCENARIOS[@]}" "$metrics_file"
|
||||||
|
yq -i ".validation.timestamp = \"$timestamp\"" "$metrics_file"
|
||||||
|
else
|
||||||
|
# Create new metrics file
|
||||||
|
cat > "$metrics_file" << EOF
|
||||||
|
epic_id: "$epic_id"
|
||||||
|
validation:
|
||||||
|
gate_executed: true
|
||||||
|
gate_status: "$gate_status"
|
||||||
|
fix_attempts: $fix_attempts
|
||||||
|
scenarios_passed: ${#PASSED_SCENARIOS[@]}
|
||||||
|
scenarios_failed: ${#FAILED_SCENARIOS[@]}
|
||||||
|
timestamp: "$timestamp"
|
||||||
|
EOF
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
# Fallback: append to file or create new
|
||||||
|
if [ ! -f "$metrics_file" ]; then
|
||||||
|
cat > "$metrics_file" << EOF
|
||||||
|
epic_id: "$epic_id"
|
||||||
|
validation:
|
||||||
|
gate_executed: true
|
||||||
|
gate_status: "$gate_status"
|
||||||
|
fix_attempts: $fix_attempts
|
||||||
|
scenarios_passed: ${#PASSED_SCENARIOS[@]}
|
||||||
|
scenarios_failed: ${#FAILED_SCENARIOS[@]}
|
||||||
|
timestamp: "$timestamp"
|
||||||
|
EOF
|
||||||
|
else
|
||||||
|
# Simple append for validation section
|
||||||
|
log_warn "yq not found - metrics update may be incomplete"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Metrics updated: $metrics_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
output_signals() {
|
||||||
|
local gate_status="$1"
|
||||||
|
local fix_attempts="$2"
|
||||||
|
|
||||||
|
local total=${#AUTOMATABLE_SCENARIOS[@]}
|
||||||
|
local passed=${#PASSED_SCENARIOS[@]}
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "UAT_GATE_RESULT: $gate_status"
|
||||||
|
echo "UAT_FIX_ATTEMPTS: $fix_attempts"
|
||||||
|
echo "UAT_SCENARIOS_PASSED: $passed/$total"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_summary() {
|
||||||
|
local gate_status="$1"
|
||||||
|
local fix_attempts="$2"
|
||||||
|
|
||||||
|
log_header "UAT VALIDATION COMPLETE"
|
||||||
|
|
||||||
|
echo " Epic: $EPIC_ID"
|
||||||
|
echo " Gate Mode: $UAT_GATE_MODE"
|
||||||
|
echo " Gate Result: $gate_status"
|
||||||
|
echo ""
|
||||||
|
echo " Scenarios:"
|
||||||
|
echo " Automatable: ${#AUTOMATABLE_SCENARIOS[@]}"
|
||||||
|
echo " Semi-automated: ${#SEMI_AUTO_SCENARIOS[@]}"
|
||||||
|
echo " Manual: ${#MANUAL_SCENARIOS[@]}"
|
||||||
|
echo ""
|
||||||
|
echo " Results:"
|
||||||
|
echo " Passed: ${#PASSED_SCENARIOS[@]}"
|
||||||
|
echo " Failed: ${#FAILED_SCENARIOS[@]}"
|
||||||
|
echo " Fix Attempts: $fix_attempts"
|
||||||
|
echo ""
|
||||||
|
echo " Artifacts:"
|
||||||
|
echo " Log: $LOG_FILE"
|
||||||
|
echo " UAT Document: $UAT_FILE"
|
||||||
|
if [ ${#FAILED_SCENARIOS[@]} -gt 0 ] && [ -d "$FIX_DIR" ]; then
|
||||||
|
echo " Fix Contexts: $FIX_DIR/"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Main Execution
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
log_header "UAT VALIDATION: Epic $EPIC_ID"
|
||||||
|
log "Gate mode: $UAT_GATE_MODE"
|
||||||
|
log "Max retries: $MAX_RETRIES"
|
||||||
|
log "Timeout: ${TIMEOUT_SECONDS}s"
|
||||||
|
|
||||||
|
# Ensure directories exist
|
||||||
|
mkdir -p "$METRICS_DIR"
|
||||||
|
mkdir -p "$FIX_DIR"
|
||||||
|
|
||||||
|
# Step 1: Load UAT document
|
||||||
|
log_section "Loading UAT Document"
|
||||||
|
if ! load_uat_document "$EPIC_ID"; then
|
||||||
|
echo "UAT_GATE_RESULT: FAIL"
|
||||||
|
echo "UAT_FIX_ATTEMPTS: 0"
|
||||||
|
echo "UAT_SCENARIOS_PASSED: 0/0"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 2: Classify scenarios
|
||||||
|
log_section "Classifying Scenarios"
|
||||||
|
classify_scenarios "$UAT_FILE"
|
||||||
|
|
||||||
|
# Step 3: Execute scenarios
|
||||||
|
if ! execute_scenarios "$UAT_GATE_MODE"; then
|
||||||
|
# Gate failed - check if we should try self-healing
|
||||||
|
if [ "$DRY_RUN" = false ] && [ $MAX_RETRIES -gt 0 ]; then
|
||||||
|
if ! self_healing_loop "$EPIC_ID"; then
|
||||||
|
# Max retries exceeded
|
||||||
|
update_metrics "$EPIC_ID" "FAIL" "$MAX_RETRIES"
|
||||||
|
output_signals "FAIL" "$MAX_RETRIES"
|
||||||
|
print_summary "FAIL" "$MAX_RETRIES"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
# No self-healing or dry-run
|
||||||
|
update_metrics "$EPIC_ID" "FAIL" "0"
|
||||||
|
output_signals "FAIL" "0"
|
||||||
|
print_summary "FAIL" "0"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 4: Gate passed
|
||||||
|
FINAL_ATTEMPTS=0
|
||||||
|
if [ ${#FAILED_SCENARIOS[@]} -gt 0 ]; then
|
||||||
|
# Passed after retries
|
||||||
|
FINAL_ATTEMPTS=$((MAX_RETRIES - $(ls -1 "$FIX_DIR"/epic-${EPIC_ID}-fix-context-*.md 2>/dev/null | wc -l) + 1))
|
||||||
|
fi
|
||||||
|
|
||||||
|
update_metrics "$EPIC_ID" "PASS" "$FINAL_ATTEMPTS"
|
||||||
|
output_signals "PASS" "$FINAL_ATTEMPTS"
|
||||||
|
print_summary "PASS" "$FINAL_ATTEMPTS"
|
||||||
|
|
||||||
|
log_success "UAT validation passed for Epic $EPIC_ID"
|
||||||
|
exit 0
|
||||||
|
|
@ -9,10 +9,10 @@ epic_file: "{epic_file_path}"
|
||||||
|
|
||||||
# Execution timing
|
# Execution timing
|
||||||
execution:
|
execution:
|
||||||
start_time: "{start_timestamp}" # ISO 8601 format
|
start_time: "{start_timestamp}" # ISO 8601 format
|
||||||
end_time: "{end_timestamp}" # ISO 8601 format
|
end_time: "{end_timestamp}" # ISO 8601 format
|
||||||
duration_seconds: 0 # Calculated from start/end
|
duration_seconds: 0 # Calculated from start/end
|
||||||
duration_formatted: "" # Human readable, e.g., "1.5 hours"
|
duration_formatted: "" # Human readable, e.g., "1.5 hours"
|
||||||
|
|
||||||
# Story counts
|
# Story counts
|
||||||
stories:
|
stories:
|
||||||
|
|
@ -44,13 +44,13 @@ uat:
|
||||||
# Validation gate results
|
# Validation gate results
|
||||||
validation:
|
validation:
|
||||||
gate_executed: false
|
gate_executed: false
|
||||||
gate_mode: "quick" # quick | full | skip
|
gate_mode: "quick" # quick | full | skip
|
||||||
timestamp: ""
|
timestamp: ""
|
||||||
results:
|
results:
|
||||||
passed: 0
|
passed: 0
|
||||||
failed: 0
|
failed: 0
|
||||||
skipped: 0
|
skipped: 0
|
||||||
gate_status: "PENDING" # PASS | FAIL | PENDING | SKIPPED
|
gate_status: "PENDING" # PASS | FAIL | PENDING | SKIPPED
|
||||||
blocking_issues: []
|
blocking_issues: []
|
||||||
|
|
||||||
# Self-healing fix loop tracking
|
# Self-healing fix loop tracking
|
||||||
|
|
@ -76,19 +76,19 @@ issues: []
|
||||||
|
|
||||||
# Dependencies (from chain plan)
|
# Dependencies (from chain plan)
|
||||||
dependencies:
|
dependencies:
|
||||||
requires: [] # Epic IDs this epic depends on
|
requires: [] # Epic IDs this epic depends on
|
||||||
enables: [] # Epic IDs that depend on this epic
|
enables: [] # Epic IDs that depend on this epic
|
||||||
|
|
||||||
# Git information
|
# Git information
|
||||||
git:
|
git:
|
||||||
commits: 0 # Number of commits for this epic
|
commits: 0 # Number of commits for this epic
|
||||||
branch: "" # Branch used (if feature branch)
|
branch: "" # Branch used (if feature branch)
|
||||||
first_commit: "" # SHA of first commit
|
first_commit: "" # SHA of first commit
|
||||||
last_commit: "" # SHA of last commit
|
last_commit: "" # SHA of last commit
|
||||||
|
|
||||||
# Estimated token usage
|
# Estimated token usage
|
||||||
tokens:
|
tokens:
|
||||||
estimated_calls: 0 # stories * 2 (dev + review)
|
estimated_calls: 0 # stories * 2 (dev + review)
|
||||||
estimated_input: 0 # estimated_calls * 8000
|
estimated_input: 0 # estimated_calls * 8000
|
||||||
estimated_output: 0 # estimated_calls * 4000
|
estimated_output: 0 # estimated_calls * 4000
|
||||||
estimated_total: 0 # input + output
|
estimated_total: 0 # input + output
|
||||||
|
|
|
||||||
|
|
@ -7,13 +7,13 @@
|
||||||
execution:
|
execution:
|
||||||
# Automatically commit after each story completes
|
# Automatically commit after each story completes
|
||||||
auto_commit: true
|
auto_commit: true
|
||||||
|
|
||||||
# Run tests before transitioning to review phase
|
# Run tests before transitioning to review phase
|
||||||
run_tests_before_review: true
|
run_tests_before_review: true
|
||||||
|
|
||||||
# Maximum retries for failed phases
|
# Maximum retries for failed phases
|
||||||
max_retries: 2
|
max_retries: 2
|
||||||
|
|
||||||
# Timeout per phase in seconds (0 = no timeout)
|
# Timeout per phase in seconds (0 = no timeout)
|
||||||
phase_timeout: 0
|
phase_timeout: 0
|
||||||
|
|
||||||
|
|
@ -24,21 +24,21 @@ review:
|
||||||
# standard - Block on acceptance criteria, security, major quality
|
# standard - Block on acceptance criteria, security, major quality
|
||||||
# strict - Block on any quality issue, no auto-fixing
|
# strict - Block on any quality issue, no auto-fixing
|
||||||
mode: standard
|
mode: standard
|
||||||
|
|
||||||
# Allow reviewer to auto-fix issues
|
# Allow reviewer to auto-fix issues
|
||||||
auto_fix_enabled: true
|
auto_fix_enabled: true
|
||||||
|
|
||||||
# Require all tests pass before approval
|
# Require all tests pass before approval
|
||||||
require_passing_tests: true
|
require_passing_tests: true
|
||||||
|
|
||||||
# Issue fix policy thresholds
|
# Issue fix policy thresholds
|
||||||
fix_policy:
|
fix_policy:
|
||||||
# Always fix HIGH severity issues
|
# Always fix HIGH severity issues
|
||||||
always_fix_high: true
|
always_fix_high: true
|
||||||
|
|
||||||
# Fix MEDIUM severity if total issues exceed this threshold
|
# Fix MEDIUM severity if total issues exceed this threshold
|
||||||
medium_fix_threshold: 5
|
medium_fix_threshold: 5
|
||||||
|
|
||||||
# Never auto-fix LOW severity (just document)
|
# Never auto-fix LOW severity (just document)
|
||||||
fix_low: false
|
fix_low: false
|
||||||
|
|
||||||
|
|
@ -46,13 +46,13 @@ review:
|
||||||
context:
|
context:
|
||||||
# Clear context between phases (recommended)
|
# Clear context between phases (recommended)
|
||||||
isolate_phases: true
|
isolate_phases: true
|
||||||
|
|
||||||
# Include architecture doc in dev context
|
# Include architecture doc in dev context
|
||||||
include_architecture: true
|
include_architecture: true
|
||||||
|
|
||||||
# Include PRD in dev context
|
# Include PRD in dev context
|
||||||
include_prd: false
|
include_prd: false
|
||||||
|
|
||||||
# Max lines of code context to include
|
# Max lines of code context to include
|
||||||
max_code_context: 500
|
max_code_context: 500
|
||||||
|
|
||||||
|
|
@ -60,16 +60,16 @@ context:
|
||||||
uat:
|
uat:
|
||||||
# Generate UAT after all stories complete
|
# Generate UAT after all stories complete
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|
||||||
# UAT output location (relative to project root)
|
# UAT output location (relative to project root)
|
||||||
output_dir: docs/uat
|
output_dir: docs/uat
|
||||||
|
|
||||||
# Include edge case scenarios
|
# Include edge case scenarios
|
||||||
include_edge_cases: true
|
include_edge_cases: true
|
||||||
|
|
||||||
# Minimum scenarios to generate
|
# Minimum scenarios to generate
|
||||||
min_scenarios: 3
|
min_scenarios: 3
|
||||||
|
|
||||||
# Maximum scenarios to generate
|
# Maximum scenarios to generate
|
||||||
max_scenarios: 10
|
max_scenarios: 10
|
||||||
|
|
||||||
|
|
@ -77,13 +77,13 @@ uat:
|
||||||
git:
|
git:
|
||||||
# Commit message prefix
|
# Commit message prefix
|
||||||
commit_prefix: "feat"
|
commit_prefix: "feat"
|
||||||
|
|
||||||
# Include epic ID in commit message
|
# Include epic ID in commit message
|
||||||
include_epic_in_commit: true
|
include_epic_in_commit: true
|
||||||
|
|
||||||
# Push after each commit
|
# Push after each commit
|
||||||
auto_push: false
|
auto_push: false
|
||||||
|
|
||||||
# Branch naming pattern (use {epic_id} and {story_id} as placeholders)
|
# Branch naming pattern (use {epic_id} and {story_id} as placeholders)
|
||||||
branch_pattern: "feature/epic-{epic_id}"
|
branch_pattern: "feature/epic-{epic_id}"
|
||||||
|
|
||||||
|
|
@ -91,10 +91,10 @@ git:
|
||||||
parallel:
|
parallel:
|
||||||
# Enable parallel story execution
|
# Enable parallel story execution
|
||||||
enabled: false
|
enabled: false
|
||||||
|
|
||||||
# Maximum concurrent stories
|
# Maximum concurrent stories
|
||||||
max_concurrent: 3
|
max_concurrent: 3
|
||||||
|
|
||||||
# Only parallelize independent stories (no dependencies)
|
# Only parallelize independent stories (no dependencies)
|
||||||
respect_dependencies: true
|
respect_dependencies: true
|
||||||
|
|
||||||
|
|
@ -102,13 +102,13 @@ parallel:
|
||||||
logging:
|
logging:
|
||||||
# Log level: debug | info | warn | error
|
# Log level: debug | info | warn | error
|
||||||
level: info
|
level: info
|
||||||
|
|
||||||
# Save execution log
|
# Save execution log
|
||||||
save_log: true
|
save_log: true
|
||||||
|
|
||||||
# Log output location
|
# Log output location
|
||||||
log_dir: docs/sprints
|
log_dir: docs/sprints
|
||||||
|
|
||||||
# Include Claude responses in log (verbose)
|
# Include Claude responses in log (verbose)
|
||||||
log_responses: false
|
log_responses: false
|
||||||
|
|
||||||
|
|
@ -116,9 +116,9 @@ logging:
|
||||||
notifications:
|
notifications:
|
||||||
# Notify on completion
|
# Notify on completion
|
||||||
on_complete: false
|
on_complete: false
|
||||||
|
|
||||||
# Notify on failure
|
# Notify on failure
|
||||||
on_failure: false
|
on_failure: false
|
||||||
|
|
||||||
# Notification method: slack | email | none
|
# Notification method: slack | email | none
|
||||||
method: none
|
method: none
|
||||||
|
|
|
||||||
|
|
@ -124,8 +124,8 @@ This epic is **successful** when a user can:
|
||||||
|
|
||||||
| Metric | Value |
|
| Metric | Value |
|
||||||
|--------|-------|
|
|--------|-------|
|
||||||
| Scenarios Tested | __ / __ |
|
| Scenarios Tested | \_\_ / \_\_ |
|
||||||
| Scenarios Passed | __ / __ |
|
| Scenarios Passed | \_\_ / \_\_ |
|
||||||
| Critical Issues | |
|
| Critical Issues | |
|
||||||
| Major Issues | |
|
| Major Issues | |
|
||||||
| Minor Issues | |
|
| Minor Issues | |
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,102 @@
|
||||||
|
# Step 1: Load UAT Document
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Load and validate the UAT document for the specified epic, extracting all test scenarios for classification and execution.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| Input | Source | Required |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| epic_id | CLI argument | Yes |
|
||||||
|
| uat_dir | Configuration | Yes (default: `docs/uat`) |
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 1.1 Locate UAT Document
|
||||||
|
|
||||||
|
Search for UAT document using these patterns in order:
|
||||||
|
1. `{uat_dir}/epic-{epic_id}-uat.md`
|
||||||
|
2. `{uat_dir}/epic-0{epic_id}-uat.md` (zero-padded)
|
||||||
|
3. `{uat_dir}/{epic_id}-uat.md`
|
||||||
|
|
||||||
|
**If not found:** Exit with error code 1 and message:
|
||||||
|
```
|
||||||
|
UAT document not found for Epic {epic_id}
|
||||||
|
Searched in: {uat_dir}
|
||||||
|
Expected: epic-{epic_id}-uat.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.2 Validate Document Structure
|
||||||
|
|
||||||
|
Confirm document contains at least one of these sections:
|
||||||
|
- `## Test Scenarios`
|
||||||
|
- `## Acceptance Criteria`
|
||||||
|
- `## Scenarios`
|
||||||
|
- `## Success Criteria`
|
||||||
|
|
||||||
|
**Warning if missing:** Log warning but continue (scenarios may be inline)
|
||||||
|
|
||||||
|
### 1.3 Parse Scenarios
|
||||||
|
|
||||||
|
Extract scenario blocks by detecting:
|
||||||
|
- Headers starting with `###` (individual scenario titles)
|
||||||
|
- Numbered items `1.`, `2.`, etc. under scenario sections
|
||||||
|
- Checkbox items `- [ ]` in criteria sections
|
||||||
|
|
||||||
|
For each scenario, extract:
|
||||||
|
- **Scenario ID**: Numeric index or explicit ID
|
||||||
|
- **Scenario Name**: Title text
|
||||||
|
- **Steps**: Given/When/Then or numbered steps
|
||||||
|
- **Verification Command**: Code block or CLI reference (if present)
|
||||||
|
- **Expected Result**: Success criteria text
|
||||||
|
|
||||||
|
### 1.4 Build Scenario List
|
||||||
|
|
||||||
|
Create structured list of scenarios:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
scenarios:
|
||||||
|
- id: 1
|
||||||
|
name: "Project Initialization"
|
||||||
|
steps:
|
||||||
|
- "Run npx heimdall init"
|
||||||
|
- "Verify config file created"
|
||||||
|
verification_command: "npx heimdall --version"
|
||||||
|
expected_result: "displays a version number"
|
||||||
|
raw_content: |
|
||||||
|
### 1. Project Initialization
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| Output | Location | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| scenario_list | Memory/State | Array of parsed scenario objects |
|
||||||
|
| scenario_count | Console | Total number of scenarios found |
|
||||||
|
| uat_file_path | State | Path to loaded UAT document |
|
||||||
|
|
||||||
|
## Completion Signal
|
||||||
|
|
||||||
|
```
|
||||||
|
UAT_LOADED: {scenario_count} scenarios from {uat_file_path}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Error | Action |
|
||||||
|
|-------|--------|
|
||||||
|
| File not found | Exit 1 with clear error message |
|
||||||
|
| Empty file | Exit 1 with "UAT document is empty" |
|
||||||
|
| No scenarios detected | Log warning, return empty list (gate passes by default) |
|
||||||
|
| Parse error | Log warning for specific section, continue with partial results |
|
||||||
|
|
||||||
|
## Example Output
|
||||||
|
|
||||||
|
```
|
||||||
|
[UAT] Loading UAT document for Epic 1
|
||||||
|
[UAT] Found: docs/uat/epic-1-uat.md
|
||||||
|
[UAT] Parsed 9 scenarios
|
||||||
|
UAT_LOADED: 9 scenarios from docs/uat/epic-1-uat.md
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,156 @@
|
||||||
|
# Step 2: Classify Scenarios
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Categorize each scenario by its executability level to determine which can be automated, which need partial automation, and which require manual verification.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| Input | Source | Required |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| scenario_list | Step 1 | Yes |
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 2.1 Classification Categories
|
||||||
|
|
||||||
|
| Classification | Description | Gate Behavior |
|
||||||
|
|----------------|-------------|---------------|
|
||||||
|
| **Automatable** | Can be fully executed via shell command | Execute and verify |
|
||||||
|
| **Semi-automated** | Requires setup, then automated verification | Execute with warning |
|
||||||
|
| **Manual** | Requires human interaction or visual verification | Skip, add to checklist |
|
||||||
|
|
||||||
|
### 2.2 Detect Automatable Scenarios
|
||||||
|
|
||||||
|
Check scenario content for these indicators (case-insensitive):
|
||||||
|
|
||||||
|
**CLI/Command indicators:**
|
||||||
|
- `npx`, `npm run`, `yarn`, `pnpm`
|
||||||
|
- `node `, `python `, `ruby `
|
||||||
|
- `curl `, `wget `, `http`
|
||||||
|
- `bash `, `sh `, `./`
|
||||||
|
|
||||||
|
**Test framework indicators:**
|
||||||
|
- `pytest`, `jest`, `vitest`, `mocha`
|
||||||
|
- `npm test`, `yarn test`
|
||||||
|
|
||||||
|
**Verification indicators:**
|
||||||
|
- `--version`, `--help`
|
||||||
|
- `/health`, `/api/`, `/status`
|
||||||
|
- `exit code`, `returns 0`, `returns 1`
|
||||||
|
- `outputs`, `prints`, `displays`
|
||||||
|
|
||||||
|
**Database/Config indicators:**
|
||||||
|
- `db migrate`, `db status`
|
||||||
|
- `config validate`, `config check`
|
||||||
|
|
||||||
|
### 2.3 Detect Semi-Automated Scenarios
|
||||||
|
|
||||||
|
Scenarios with commands that require prior setup:
|
||||||
|
|
||||||
|
**Setup-required indicators:**
|
||||||
|
- "Start the server first"
|
||||||
|
- "Ensure database is running"
|
||||||
|
- "In a separate terminal"
|
||||||
|
- "After deploying"
|
||||||
|
|
||||||
|
**Partial automation indicators:**
|
||||||
|
- `test-send`, `send test`
|
||||||
|
- "check your email/inbox"
|
||||||
|
- "verify in browser"
|
||||||
|
- Manual setup + automated verification
|
||||||
|
|
||||||
|
### 2.4 Classify as Manual
|
||||||
|
|
||||||
|
Scenarios without detectable automation path:
|
||||||
|
|
||||||
|
**Manual-only indicators:**
|
||||||
|
- "Railway dashboard", "Vercel dashboard"
|
||||||
|
- "Open browser", "Navigate to"
|
||||||
|
- "Visual inspection", "Visually verify"
|
||||||
|
- "Two terminals", "Side by side"
|
||||||
|
- "User should see", "Observe that"
|
||||||
|
- No code blocks or CLI references
|
||||||
|
|
||||||
|
### 2.5 Extract Commands
|
||||||
|
|
||||||
|
For automatable/semi-automated scenarios, extract the verification command:
|
||||||
|
|
||||||
|
1. Look for inline code: `` `command here` ``
|
||||||
|
2. Look for code blocks: ```bash ... ```
|
||||||
|
3. Look for CLI patterns: `npx ...`, `npm run ...`, `curl ...`
|
||||||
|
4. Look for expected patterns after "Run:" or "Execute:"
|
||||||
|
|
||||||
|
### 2.6 Build Classification Result
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
classification:
|
||||||
|
total: 9
|
||||||
|
automatable: 6
|
||||||
|
semi_automated: 2
|
||||||
|
manual: 1
|
||||||
|
|
||||||
|
automatable_scenarios:
|
||||||
|
- id: 1
|
||||||
|
name: "Project Initialization"
|
||||||
|
command: "npx heimdall --version"
|
||||||
|
expected: "displays a version number"
|
||||||
|
|
||||||
|
- id: 3
|
||||||
|
name: "Database Migration"
|
||||||
|
command: "npx heimdall db migrate"
|
||||||
|
expected: "success message"
|
||||||
|
|
||||||
|
semi_automated_scenarios:
|
||||||
|
- id: 7
|
||||||
|
name: "Email Notification"
|
||||||
|
command: "curl -X POST localhost:3000/test-send"
|
||||||
|
expected: "email received"
|
||||||
|
note: "Requires manual inbox verification"
|
||||||
|
|
||||||
|
manual_scenarios:
|
||||||
|
- id: 9
|
||||||
|
name: "Dashboard Visual Check"
|
||||||
|
note: "Requires browser inspection"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| Output | Location | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| automatable | Array | Scenarios to execute automatically |
|
||||||
|
| semi_automated | Array | Scenarios needing setup + execution |
|
||||||
|
| manual | Array | Scenarios requiring human verification |
|
||||||
|
| classification_summary | Console | Counts per category |
|
||||||
|
|
||||||
|
## Completion Signal
|
||||||
|
|
||||||
|
```
|
||||||
|
SCENARIOS_CLASSIFIED: {automatable}/{semi_automated}/{manual}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Output
|
||||||
|
|
||||||
|
```
|
||||||
|
[UAT] Classifying 9 scenarios...
|
||||||
|
[AUTO] Scenario 1: Project Initialization
|
||||||
|
[AUTO] Scenario 2: Configuration Setup
|
||||||
|
[AUTO] Scenario 3: Database Migration
|
||||||
|
[AUTO] Scenario 4: Connection Validation
|
||||||
|
[AUTO] Scenario 5: Worker Process Startup
|
||||||
|
[AUTO] Scenario 6: Job Queue Testing
|
||||||
|
[SEMI] Scenario 7: Email Notification
|
||||||
|
[SEMI] Scenario 8: Webhook Delivery
|
||||||
|
[MANUAL] Scenario 9: Dashboard Visual Check
|
||||||
|
|
||||||
|
SCENARIOS_CLASSIFIED: 6/2/1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Classification Confidence
|
||||||
|
|
||||||
|
For edge cases, use this priority:
|
||||||
|
1. If command is clearly present → Automatable
|
||||||
|
2. If command present but requires setup → Semi-automated
|
||||||
|
3. If no command detected → Manual
|
||||||
|
|
||||||
|
When in doubt, classify as semi-automated (attempts execution, flags for review).
|
||||||
|
|
@ -0,0 +1,202 @@
|
||||||
|
# Step 3: Execute Scenarios
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Run automatable scenarios via shell commands, capture results, and determine pass/fail status for each.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| Input | Source | Required |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| automatable | Step 2 | Yes |
|
||||||
|
| semi_automated | Step 2 | For full gate mode |
|
||||||
|
| gate_mode | CLI | Yes (quick/full/skip) |
|
||||||
|
| timeout | Config | No (default: 30s) |
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 3.1 Filter by Gate Mode
|
||||||
|
|
||||||
|
| Mode | Scenarios Executed |
|
||||||
|
|------|-------------------|
|
||||||
|
| `quick` | Automatable only |
|
||||||
|
| `full` | Automatable + Semi-automated |
|
||||||
|
| `skip` | None (gate passes automatically) |
|
||||||
|
|
||||||
|
**Skip mode behavior:**
|
||||||
|
```
|
||||||
|
[UAT] Gate mode: skip - bypassing scenario execution
|
||||||
|
UAT_GATE_RESULT: PASS
|
||||||
|
UAT_SCENARIOS_PASSED: 0/0 (skipped)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Handle Empty Scenario List
|
||||||
|
|
||||||
|
If no scenarios to execute:
|
||||||
|
```
|
||||||
|
[UAT] No automatable scenarios found - gate passes by default
|
||||||
|
UAT_GATE_RESULT: PASS
|
||||||
|
UAT_SCENARIOS_PASSED: 0/0 (none automatable)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.3 Execute Each Scenario
|
||||||
|
|
||||||
|
For each scenario in the execution list:
|
||||||
|
|
||||||
|
#### 3.3.1 Extract Command
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From scenario object
|
||||||
|
command="${scenario.command}"
|
||||||
|
|
||||||
|
# If no command extracted, mark as failed
|
||||||
|
if [ -z "$command" ]; then
|
||||||
|
status="FAIL"
|
||||||
|
error="No automatable command found"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3.3.2 Execute with Timeout
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Using GNU timeout
|
||||||
|
output=$(timeout $TIMEOUT_SECONDS bash -c "$command" 2>&1)
|
||||||
|
exit_code=$?
|
||||||
|
|
||||||
|
# macOS fallback (perl-based timeout)
|
||||||
|
output=$(perl -e 'alarm shift @ARGV; exec @ARGV' $TIMEOUT_SECONDS bash -c "$command" 2>&1)
|
||||||
|
exit_code=$?
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3.3.3 Capture Results
|
||||||
|
|
||||||
|
Record for each execution:
|
||||||
|
- **stdout**: Command output
|
||||||
|
- **stderr**: Error output (captured via 2>&1)
|
||||||
|
- **exit_code**: Process exit code
|
||||||
|
- **duration_ms**: Execution time in milliseconds
|
||||||
|
- **status**: PASS or FAIL
|
||||||
|
|
||||||
|
### 3.4 Result Evaluation Rules
|
||||||
|
|
||||||
|
| Condition | Result | Notes |
|
||||||
|
|-----------|--------|-------|
|
||||||
|
| Exit code 0 | PASS | Command succeeded |
|
||||||
|
| Exit code 124 | FAIL (timeout) | Exceeded timeout limit |
|
||||||
|
| Exit code 127 | FAIL (not found) | Command not found |
|
||||||
|
| Exit code non-zero | FAIL (error) | Command failed |
|
||||||
|
|
||||||
|
### 3.5 Expected Output Matching (Optional)
|
||||||
|
|
||||||
|
If scenario has `expected_result`, apply flexible matching:
|
||||||
|
|
||||||
|
**Contains match** (default):
|
||||||
|
```bash
|
||||||
|
if echo "$output" | grep -qi "$expected"; then
|
||||||
|
match="true"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
**Exit code only match:**
|
||||||
|
```bash
|
||||||
|
# If expected contains "exit 0" or "returns 0"
|
||||||
|
if [ $exit_code -eq 0 ]; then
|
||||||
|
match="true"
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** Output matching is advisory. Primary pass/fail is based on exit code.
|
||||||
|
|
||||||
|
### 3.6 Record Execution Results
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
execution_results:
|
||||||
|
- scenario_id: 1
|
||||||
|
name: "Project Initialization"
|
||||||
|
command: "npx heimdall --version"
|
||||||
|
status: "PASS"
|
||||||
|
exit_code: 0
|
||||||
|
output: "1.0.0"
|
||||||
|
duration_ms: 1250
|
||||||
|
|
||||||
|
- scenario_id: 3
|
||||||
|
name: "Database Migration"
|
||||||
|
command: "npx heimdall db migrate"
|
||||||
|
status: "FAIL"
|
||||||
|
exit_code: 1
|
||||||
|
output: ""
|
||||||
|
stderr: "Error: relation 'events' does not exist"
|
||||||
|
duration_ms: 3400
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.7 Progress Output
|
||||||
|
|
||||||
|
During execution, output progress:
|
||||||
|
|
||||||
|
```
|
||||||
|
[UAT] Executing 6 scenarios...
|
||||||
|
|
||||||
|
Scenario 1: Project Initialization
|
||||||
|
Command: npx heimdall --version
|
||||||
|
[PASS] (1250ms)
|
||||||
|
|
||||||
|
Scenario 2: Configuration Setup
|
||||||
|
Command: npx heimdall config validate
|
||||||
|
[PASS] (890ms)
|
||||||
|
|
||||||
|
Scenario 3: Database Migration
|
||||||
|
Command: npx heimdall db migrate
|
||||||
|
[FAIL] (3400ms)
|
||||||
|
Error: relation 'events' does not exist
|
||||||
|
```
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| Output | Location | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| results | Array | Per-scenario execution results |
|
||||||
|
| passed_count | State | Number of passed scenarios |
|
||||||
|
| failed_count | State | Number of failed scenarios |
|
||||||
|
| failed_details | Array | Details for failed scenarios |
|
||||||
|
|
||||||
|
## Completion Signal
|
||||||
|
|
||||||
|
```
|
||||||
|
SCENARIOS_EXECUTED: {passed}/{total}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Error | Action |
|
||||||
|
|-------|--------|
|
||||||
|
| Command not found | Record as FAIL with "command not found" message |
|
||||||
|
| Timeout exceeded | Record as FAIL with "timeout after Xs" message |
|
||||||
|
| Permission denied | Record as FAIL with permission error |
|
||||||
|
| Shell syntax error | Record as FAIL with syntax error details |
|
||||||
|
|
||||||
|
## Example Complete Output
|
||||||
|
|
||||||
|
```
|
||||||
|
[UAT] Executing 6 scenarios (gate mode: quick)
|
||||||
|
|
||||||
|
Scenario 1: Project Initialization
|
||||||
|
[PASS] npx heimdall --version (1250ms)
|
||||||
|
|
||||||
|
Scenario 2: Configuration Setup
|
||||||
|
[PASS] npx heimdall config validate (890ms)
|
||||||
|
|
||||||
|
Scenario 3: Database Migration
|
||||||
|
[FAIL] npx heimdall db migrate (3400ms)
|
||||||
|
Error: relation 'events' does not exist
|
||||||
|
|
||||||
|
Scenario 4: Connection Validation
|
||||||
|
[PASS] npx heimdall db status (450ms)
|
||||||
|
|
||||||
|
Scenario 5: Worker Process Startup
|
||||||
|
[PASS] npx heimdall worker --check (2100ms)
|
||||||
|
|
||||||
|
Scenario 6: Job Queue Testing
|
||||||
|
[PASS] npx heimdall queue test (1800ms)
|
||||||
|
|
||||||
|
SCENARIOS_EXECUTED: 5/6
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,237 @@
|
||||||
|
# Step 4: Evaluate Gate
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Determine overall pass/fail status based on execution results. If failed and self-healing is enabled, generate fix context and trigger the quick-dev fix loop.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| Input | Source | Required |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| results | Step 3 | Yes |
|
||||||
|
| passed_count | Step 3 | Yes |
|
||||||
|
| failed_count | Step 3 | Yes |
|
||||||
|
| failed_details | Step 3 | Yes |
|
||||||
|
| max_retries | CLI | Yes (default: 2) |
|
||||||
|
| current_attempt | State | Yes (starts at 0) |
|
||||||
|
| self_heal_enabled | Config | Yes (default: true) |
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 4.1 Check All Results
|
||||||
|
|
||||||
|
```
|
||||||
|
if failed_count == 0:
|
||||||
|
gate_status = PASS
|
||||||
|
→ Skip to Step 5 (Report Results)
|
||||||
|
else:
|
||||||
|
gate_status = FAIL
|
||||||
|
→ Continue to 4.2
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Evaluate Retry Eligibility
|
||||||
|
|
||||||
|
```
|
||||||
|
if current_attempt >= max_retries:
|
||||||
|
→ Max retries exceeded, halt
|
||||||
|
exit_code = 2
|
||||||
|
|
||||||
|
if self_heal_enabled == false:
|
||||||
|
→ Self-healing disabled, report failure
|
||||||
|
exit_code = 1
|
||||||
|
|
||||||
|
→ Continue to 4.3 (Generate Fix Context)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 Generate Fix Context Document
|
||||||
|
|
||||||
|
Create fix context document at:
|
||||||
|
```
|
||||||
|
{sprint_artifacts}/uat-fixes/epic-{epic_id}-fix-context-{attempt}.md
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4.3.1 Load Template
|
||||||
|
|
||||||
|
Load from: `{workflow_path}/uat-fix-context-template.md`
|
||||||
|
|
||||||
|
If template not found, use inline default structure.
|
||||||
|
|
||||||
|
#### 4.3.2 Populate Template Variables
|
||||||
|
|
||||||
|
| Variable | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| `{epic_id}` | Epic ID being validated |
|
||||||
|
| `{attempt}` | Current fix attempt number |
|
||||||
|
| `{timestamp}` | ISO 8601 timestamp |
|
||||||
|
| `{max_retries}` | Maximum retry limit |
|
||||||
|
| `{next_attempt}` | attempt + 1 |
|
||||||
|
| `{failure_count}` | Number of failed scenarios |
|
||||||
|
| `{passed}` | Number of passed scenarios |
|
||||||
|
| `{total}` | Total scenarios executed |
|
||||||
|
|
||||||
|
#### 4.3.3 Append Failed Scenario Details
|
||||||
|
|
||||||
|
For each failed scenario:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
### Scenario {id}: {name}
|
||||||
|
|
||||||
|
**Command Executed:**
|
||||||
|
```bash
|
||||||
|
{command}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Result:**
|
||||||
|
{expected_result}
|
||||||
|
|
||||||
|
**Actual Result:**
|
||||||
|
```
|
||||||
|
{actual_output}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error Output:**
|
||||||
|
```
|
||||||
|
{stderr}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Exit Code:** {exit_code}
|
||||||
|
|
||||||
|
**Related Story:** {story_id} (if determinable)
|
||||||
|
|
||||||
|
**Root Cause Hint:**
|
||||||
|
{analyze_error_for_hints}
|
||||||
|
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4.3.4 Add Fix Instructions
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## Fix Instructions
|
||||||
|
|
||||||
|
Address the failures above in priority order. For each fix:
|
||||||
|
|
||||||
|
1. **Analyze** - Understand why the scenario failed
|
||||||
|
2. **Locate** - Find the relevant code files
|
||||||
|
3. **Fix** - Implement the minimum change to resolve
|
||||||
|
4. **Verify** - Run the scenario command locally
|
||||||
|
5. **Commit** - Use: `fix(epic-{epic_id}): {description}`
|
||||||
|
|
||||||
|
### Constraints
|
||||||
|
|
||||||
|
- Only fix identified failures
|
||||||
|
- Do not refactor unrelated code
|
||||||
|
- Run tests after fixes: `npm test`
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Trigger Quick-Dev Fix
|
||||||
|
|
||||||
|
Spawn a fresh Claude session for fixes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
fix_prompt="You are Barry, the Quick Flow Solo Dev.
|
||||||
|
|
||||||
|
Load and process this fix context document:
|
||||||
|
{fix_context_file}
|
||||||
|
|
||||||
|
Your task:
|
||||||
|
1. Read the failed scenarios and error details
|
||||||
|
2. Analyze root cause for each failure
|
||||||
|
3. Implement targeted fixes
|
||||||
|
4. Run the failing commands to verify fixes
|
||||||
|
5. Stage changes: git add -A
|
||||||
|
6. Commit: fix(epic-{epic_id}): UAT fix #{attempt}
|
||||||
|
|
||||||
|
Constraints:
|
||||||
|
- Only fix identified failures
|
||||||
|
- Do not refactor unrelated code
|
||||||
|
- Run tests after fixes
|
||||||
|
|
||||||
|
When done, output:
|
||||||
|
FIX_COMPLETE: {fixed_count}/{failure_count}"
|
||||||
|
|
||||||
|
claude --dangerously-skip-permissions -p "$fix_prompt"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 Increment and Re-validate
|
||||||
|
|
||||||
|
After fix session completes:
|
||||||
|
|
||||||
|
1. Increment `current_attempt`
|
||||||
|
2. Return to Step 3 (Execute Scenarios)
|
||||||
|
3. Re-run validation with fresh results
|
||||||
|
|
||||||
|
```
|
||||||
|
current_attempt += 1
|
||||||
|
→ Go to Step 3: Execute Scenarios
|
||||||
|
→ Then return to Step 4: Evaluate Gate
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.6 Gate Decision Flowchart
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐
|
||||||
|
│ Execute Results │
|
||||||
|
└────────┬────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────┐
|
||||||
|
│All Pass?│──Yes──▶ GATE_PASS (exit 0)
|
||||||
|
└────┬────┘
|
||||||
|
│No
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│ Retries Left?│──No──▶ GATE_FAIL (exit 2)
|
||||||
|
└──────┬───────┘ Max retries exceeded
|
||||||
|
│Yes
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│Self-Heal On? │──No──▶ GATE_FAIL (exit 1)
|
||||||
|
└──────┬───────┘ Fixable but disabled
|
||||||
|
│Yes
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│Generate Fix │
|
||||||
|
│Context Doc │
|
||||||
|
└──────┬───────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│Spawn Quick- │
|
||||||
|
│Dev Fix │
|
||||||
|
└──────┬───────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│Re-validate │──────▶ (Back to Execute)
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| Output | Location | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| gate_status | State | PASS or FAIL |
|
||||||
|
| fix_context_file | Path | Generated fix context (if failed) |
|
||||||
|
| fix_attempt | State | Current attempt number |
|
||||||
|
|
||||||
|
## Completion Signal
|
||||||
|
|
||||||
|
```
|
||||||
|
GATE_EVALUATED: PASS
|
||||||
|
```
|
||||||
|
or
|
||||||
|
```
|
||||||
|
GATE_EVALUATED: FAIL
|
||||||
|
FIX_CONTEXT: {fix_context_file}
|
||||||
|
FIX_ATTEMPT: {attempt}/{max_retries}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Error | Action |
|
||||||
|
|-------|--------|
|
||||||
|
| Template not found | Use inline default template |
|
||||||
|
| Fix context write fails | Log error, continue without self-healing |
|
||||||
|
| Claude session fails | Log output, count as failed attempt |
|
||||||
|
| Fix didn't resolve issue | Increment attempt, retry if allowed |
|
||||||
|
|
@ -0,0 +1,235 @@
|
||||||
|
# Step 5: Report Results
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Update metrics files with validation results and output parseable signals for orchestration scripts (epic-chain.sh) to consume.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
| Input | Source | Required |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| gate_status | Step 4 | Yes |
|
||||||
|
| results | Step 3 | Yes |
|
||||||
|
| passed_count | Step 3 | Yes |
|
||||||
|
| failed_count | Step 3 | Yes |
|
||||||
|
| fix_attempts | State | Yes |
|
||||||
|
| epic_id | CLI | Yes |
|
||||||
|
| metrics_dir | Config | Yes |
|
||||||
|
|
||||||
|
## Process
|
||||||
|
|
||||||
|
### 5.1 Update Metrics File
|
||||||
|
|
||||||
|
Update or create metrics file at:
|
||||||
|
```
|
||||||
|
{metrics_dir}/epic-{epic_id}-metrics.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.1.1 Using yq (if available)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
yq -i '.validation.gate_executed = true' "$metrics_file"
|
||||||
|
yq -i '.validation.gate_status = "PASS"' "$metrics_file"
|
||||||
|
yq -i '.validation.gate_mode = "quick"' "$metrics_file"
|
||||||
|
yq -i '.validation.fix_attempts = 0' "$metrics_file"
|
||||||
|
yq -i '.validation.scenarios_passed = 5' "$metrics_file"
|
||||||
|
yq -i '.validation.scenarios_failed = 1' "$metrics_file"
|
||||||
|
yq -i '.validation.timestamp = "2026-01-05T17:30:00Z"' "$metrics_file"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.1.2 Fallback (no yq)
|
||||||
|
|
||||||
|
Create or append validation section:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
validation:
|
||||||
|
gate_executed: true
|
||||||
|
gate_mode: "quick"
|
||||||
|
timestamp: "2026-01-05T17:30:00Z"
|
||||||
|
results:
|
||||||
|
passed: 5
|
||||||
|
failed: 1
|
||||||
|
skipped: 3
|
||||||
|
gate_status: "PASS"
|
||||||
|
fix_attempts: 0
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.1.3 Record Blocking Issues (if failed)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
validation:
|
||||||
|
# ... other fields ...
|
||||||
|
blocking_issues:
|
||||||
|
- scenario_id: 3
|
||||||
|
name: "Database Migration"
|
||||||
|
error: "relation 'events' does not exist"
|
||||||
|
command: "npx heimdall db migrate"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 5.1.4 Record Fix History (if self-healing occurred)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
validation:
|
||||||
|
# ... other fields ...
|
||||||
|
fix_history:
|
||||||
|
- attempt: 1
|
||||||
|
timestamp: "2026-01-05T17:35:00Z"
|
||||||
|
failed_scenarios: ["scenario-3"]
|
||||||
|
fix_context: "docs/sprint-artifacts/uat-fixes/epic-1-fix-context-1.md"
|
||||||
|
result: "partial"
|
||||||
|
- attempt: 2
|
||||||
|
timestamp: "2026-01-05T17:42:00Z"
|
||||||
|
failed_scenarios: []
|
||||||
|
result: "success"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Output Parseable Signals
|
||||||
|
|
||||||
|
Print to stdout in format that epic-chain.sh can parse:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Always output these signals
|
||||||
|
echo "UAT_GATE_RESULT: $gate_status"
|
||||||
|
echo "UAT_FIX_ATTEMPTS: $fix_attempts"
|
||||||
|
echo "UAT_SCENARIOS_PASSED: $passed_count/$total_count"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Additional signals on failure:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo "UAT_FIX_REQUIRED: true"
|
||||||
|
echo "UAT_FIX_CONTEXT: $fix_context_file"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Additional signals on max retries:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo "UAT_MAX_RETRIES: true"
|
||||||
|
echo "UAT_HALT_CHAIN: true"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.3 Set Exit Code
|
||||||
|
|
||||||
|
| Status | Exit Code | Meaning |
|
||||||
|
|--------|-----------|---------|
|
||||||
|
| PASS | 0 | All scenarios passed |
|
||||||
|
| FAIL (fixable) | 1 | Failed but retries remain or self-heal disabled |
|
||||||
|
| FAIL (max retries) | 2 | Max retries exceeded, chain should halt |
|
||||||
|
|
||||||
|
### 5.4 Print Human-Readable Summary
|
||||||
|
|
||||||
|
```
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
UAT VALIDATION COMPLETE
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
Epic: 1
|
||||||
|
Gate Mode: quick
|
||||||
|
Gate Result: PASS
|
||||||
|
|
||||||
|
Scenarios:
|
||||||
|
Automatable: 6
|
||||||
|
Semi-automated: 2
|
||||||
|
Manual: 1
|
||||||
|
|
||||||
|
Results:
|
||||||
|
Passed: 6
|
||||||
|
Failed: 0
|
||||||
|
Fix Attempts: 0
|
||||||
|
|
||||||
|
Artifacts:
|
||||||
|
Log: /tmp/bmad-uat-validate-12345.log
|
||||||
|
UAT Document: docs/uat/epic-1-uat.md
|
||||||
|
Metrics: docs/sprint-artifacts/metrics/epic-1-metrics.yaml
|
||||||
|
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.5 Failure Summary (if applicable)
|
||||||
|
|
||||||
|
```
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
UAT VALIDATION FAILED
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
Epic: 1
|
||||||
|
Gate Mode: quick
|
||||||
|
Gate Result: FAIL
|
||||||
|
|
||||||
|
Scenarios:
|
||||||
|
Automatable: 6
|
||||||
|
Semi-automated: 2
|
||||||
|
Manual: 1
|
||||||
|
|
||||||
|
Results:
|
||||||
|
Passed: 5
|
||||||
|
Failed: 1
|
||||||
|
Fix Attempts: 2
|
||||||
|
|
||||||
|
Blocking Issues:
|
||||||
|
- Scenario 3: Database Migration
|
||||||
|
Error: relation 'events' does not exist
|
||||||
|
|
||||||
|
Fix Context:
|
||||||
|
docs/sprint-artifacts/uat-fixes/epic-1-fix-context-2.md
|
||||||
|
|
||||||
|
Action Required:
|
||||||
|
Manual intervention needed - max retries exceeded
|
||||||
|
|
||||||
|
═══════════════════════════════════════════════════════════
|
||||||
|
```
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
| Output | Location | Description |
|
||||||
|
|--------|----------|-------------|
|
||||||
|
| Updated metrics | YAML file | Validation results persisted |
|
||||||
|
| Signals | stdout | Parseable output for orchestration |
|
||||||
|
| Summary | stdout | Human-readable summary |
|
||||||
|
| Exit code | Process | 0=pass, 1=fail-fixable, 2=fail-max-retries |
|
||||||
|
|
||||||
|
## Completion Signal
|
||||||
|
|
||||||
|
```
|
||||||
|
RESULTS_REPORTED: {metrics_file_path}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Signal Reference
|
||||||
|
|
||||||
|
### For Orchestration Scripts
|
||||||
|
|
||||||
|
| Signal | Values | Description |
|
||||||
|
|--------|--------|-------------|
|
||||||
|
| `UAT_GATE_RESULT` | PASS, FAIL | Overall gate status |
|
||||||
|
| `UAT_FIX_ATTEMPTS` | 0-N | Number of fix attempts made |
|
||||||
|
| `UAT_SCENARIOS_PASSED` | X/Y | Passed/Total ratio |
|
||||||
|
| `UAT_FIX_REQUIRED` | true | Indicates fixes were needed |
|
||||||
|
| `UAT_FIX_CONTEXT` | path | Location of fix context doc |
|
||||||
|
| `UAT_MAX_RETRIES` | true | Max retries exceeded |
|
||||||
|
| `UAT_HALT_CHAIN` | true | Chain should stop |
|
||||||
|
|
||||||
|
### Parsing in Shell
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In epic-chain.sh
|
||||||
|
uat_output=$("$SCRIPT_DIR/uat-validate.sh" "$epic_id" 2>&1)
|
||||||
|
|
||||||
|
if echo "$uat_output" | grep -q "UAT_GATE_RESULT: PASS"; then
|
||||||
|
log_success "UAT passed"
|
||||||
|
else
|
||||||
|
log_error "UAT failed"
|
||||||
|
|
||||||
|
if echo "$uat_output" | grep -q "UAT_HALT_CHAIN: true"; then
|
||||||
|
log_error "Halting chain - max retries exceeded"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
| Error | Action |
|
||||||
|
|-------|--------|
|
||||||
|
| Metrics file write fails | Log warning, continue with signal output |
|
||||||
|
| yq not available | Use fallback append method |
|
||||||
|
| Invalid metrics file | Create new file with validation section |
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
# UAT Fix Context - Epic {epic_id} (Attempt {attempt})
|
# UAT Fix Context - Epic {epic_id} (Attempt {attempt})
|
||||||
|
|
||||||
**Generated:** {timestamp}
|
**Generated:** {timestamp}
|
||||||
**Epic:** {epic_name}
|
**Epic:** {epic_id}
|
||||||
**Gate Result:** FAIL ({passed}/{total} scenarios passed)
|
**Gate Result:** FAIL ({passed}/{total} scenarios passed)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -12,101 +12,16 @@ This document contains the context needed to fix UAT failures for Epic {epic_id}
|
||||||
|
|
||||||
**Failures to fix:** {failure_count}
|
**Failures to fix:** {failure_count}
|
||||||
**Fix attempt:** {attempt} of {max_retries}
|
**Fix attempt:** {attempt} of {max_retries}
|
||||||
|
**UAT Document:** `{uat_doc_path}`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Failed Scenarios
|
## Quick Start
|
||||||
|
|
||||||
{#each failed_scenario}
|
1. Read each failed scenario below
|
||||||
### Scenario {scenario_id}: {scenario_name}
|
2. Identify root cause from error output
|
||||||
|
3. Locate and fix the relevant code
|
||||||
**Command Executed:**
|
4. Verify fix: run the failing command
|
||||||
```bash
|
5. Commit: `fix(epic-{epic_id}): {description}`
|
||||||
{command}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Expected Result:**
|
|
||||||
{expected_result}
|
|
||||||
|
|
||||||
**Actual Result:**
|
|
||||||
```
|
|
||||||
{actual_output}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Error Output:**
|
|
||||||
```
|
|
||||||
{stderr}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Exit Code:** {exit_code}
|
|
||||||
|
|
||||||
**Related Story:** {story_id}
|
|
||||||
|
|
||||||
**Acceptance Criteria:**
|
|
||||||
{#each acceptance_criteria}
|
|
||||||
- {criterion}
|
|
||||||
{/each}
|
|
||||||
|
|
||||||
**Root Cause Hint:**
|
|
||||||
{root_cause_hint}
|
|
||||||
|
|
||||||
**Files Likely Involved:**
|
|
||||||
{#each likely_files}
|
|
||||||
- `{file_path}`
|
|
||||||
{/each}
|
|
||||||
|
|
||||||
---
|
---
|
||||||
{/each}
|
|
||||||
|
|
||||||
## Fix Instructions
|
|
||||||
|
|
||||||
Address the failures above in priority order. For each fix:
|
|
||||||
|
|
||||||
1. **Analyze** - Understand why the scenario failed
|
|
||||||
2. **Locate** - Find the relevant code files
|
|
||||||
3. **Fix** - Implement the minimum change to resolve the failure
|
|
||||||
4. **Verify** - Run the scenario command locally to confirm fix
|
|
||||||
5. **Commit** - Use message format: `fix(epic-{epic_id}): {description}`
|
|
||||||
|
|
||||||
### Priority Order
|
|
||||||
|
|
||||||
{#each failed_scenario}
|
|
||||||
{priority}. **Scenario {scenario_id}**: {one_line_description}
|
|
||||||
{/each}
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Constraints
|
|
||||||
|
|
||||||
- Only fix the identified failures - do not refactor unrelated code
|
|
||||||
- Run the specific failing commands to verify each fix
|
|
||||||
- Run project tests after all fixes: `npm test`
|
|
||||||
- Commit with conventional format: `fix(epic-{epic_id}): {description}`
|
|
||||||
- If a fix requires changes that would break other scenarios, document the tradeoff
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Context Files
|
|
||||||
|
|
||||||
The following files may provide additional context:
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `{uat_doc_path}` | Full UAT document with all scenarios |
|
|
||||||
| `{story_files}` | Story files with complete acceptance criteria |
|
|
||||||
| `{architecture_doc}` | System architecture reference |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## After Fixing
|
|
||||||
|
|
||||||
Once all fixes are committed, the UAT validation will automatically re-run.
|
|
||||||
|
|
||||||
- **If all pass:** Epic continues to next phase
|
|
||||||
- **If failures remain:** Another fix context will be generated (attempt {next_attempt})
|
|
||||||
- **If max retries exceeded:** Chain halts for human intervention
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
*Generated by UAT Validate Workflow*
|
|
||||||
*BMAD Method - Epic Chain Self-Healing*
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue