461 lines
11 KiB
Markdown
461 lines
11 KiB
Markdown
---
|
|
description: "Finds and runs next test gate"
|
|
prerequisites: "test gates in project"
|
|
argument-hint: "no arguments needed - auto-detects next gate"
|
|
allowed-tools: ["Bash", "Read"]
|
|
---
|
|
|
|
# ⚠️ PROJECT-SPECIFIC COMMAND - Requires test gates infrastructure
|
|
|
|
This command requires:
|
|
|
|
- ~/.claude/lib/testgates_discovery.py (test gate discovery script)
|
|
- docs/epics.md (or similar) with test gate definitions
|
|
- user-testing/scripts/ directory with validation scripts
|
|
- user-testing/reports/ directory for results
|
|
|
|
The file path checks in Step 3.5 are project-specific examples that should be customized for your project's implementation structure
|
|
|
|
## Test Gate Finder & Executor
|
|
|
|
**Your task**: Find the next test gate to run, show the user what's needed, and execute it if they confirm.
|
|
|
|
## Step 1: Discover Test Gates and Prerequisites
|
|
|
|
First, check if the required infrastructure exists:
|
|
|
|
```bash
|
|
|
|
# ============================================
|
|
|
|
# PRE-FLIGHT CHECKS (Infrastructure Validation)
|
|
|
|
# ============================================
|
|
|
|
TESTGATES_SCRIPT="$HOME/.claude/lib/testgates_discovery.py"
|
|
|
|
# Check if discovery script exists
|
|
|
|
if [[ ! -f "$TESTGATES_SCRIPT" ]]; then
|
|
echo "❌ Test gates discovery script not found"
|
|
echo " Expected: $TESTGATES_SCRIPT"
|
|
echo ""
|
|
echo " This command requires the testgates_discovery.py library."
|
|
echo " It is designed for projects with test gate infrastructure."
|
|
exit 1
|
|
fi
|
|
|
|
# Check for epic definition files
|
|
|
|
EPICS_FILE=""
|
|
for file in "docs/epics.md" "docs/EPICS.md" "docs/test-gates.md" "EPICS.md"; do
|
|
if [[ -f "$file" ]]; then
|
|
EPICS_FILE="$file"
|
|
echo "📁 Found epics file: $EPICS_FILE"
|
|
break
|
|
fi
|
|
done
|
|
|
|
if [[ -z "$EPICS_FILE" ]]; then
|
|
echo "⚠️ No epics definition file found"
|
|
echo " Searched: docs/epics.md, docs/EPICS.md, docs/test-gates.md, EPICS.md"
|
|
echo " Test gate discovery may fail without this file."
|
|
fi
|
|
|
|
# Check for user-testing directory structure
|
|
|
|
if [[ ! -d "user-testing" ]]; then
|
|
echo "⚠️ No user-testing/ directory found"
|
|
echo " This command expects user-testing/scripts/ and user-testing/reports/"
|
|
echo " Creating minimal structure..."
|
|
mkdir -p user-testing/scripts user-testing/reports
|
|
fi
|
|
|
|
```text
|
|
|
|
Run the discovery script to get test gate configuration:
|
|
|
|
```bash
|
|
python3 "$TESTGATES_SCRIPT" . --format json > /tmp/testgates_config.json 2>/dev/null
|
|
|
|
```text
|
|
|
|
If this fails or produces empty output, tell the user:
|
|
|
|
```text
|
|
|
|
❌ Failed to discover test gates from epic definition file
|
|
Make sure docs/epics.md (or similar) exists with story and test gate definitions.
|
|
|
|
```text
|
|
|
|
## Step 2: Check Which Gates Have Already Passed
|
|
|
|
Parse the config to get list of all test gates in order:
|
|
|
|
```bash
|
|
cat /tmp/testgates_config.json | python3 -c "
|
|
import json, sys
|
|
config = json.load(sys.stdin)
|
|
gates = config.get('test_gates', {})
|
|
for gate_id in sorted(gates.keys()):
|
|
print(gate_id)
|
|
"
|
|
|
|
```text
|
|
|
|
For each gate, check if it has passed by looking for a report with "PROCEED":
|
|
|
|
```bash
|
|
gate_id="TG-X.Y" # Replace with actual gate ID
|
|
|
|
# Check subdirectory first: user-testing/reports/TG-X.Y/
|
|
|
|
if [ -d "user-testing/reports/$gate_id" ]; then
|
|
report=$(find "user-testing/reports/$gate_id" -name "*report.md" 2>/dev/null | head -1)
|
|
if [ -n "$report" ] && grep -q "PROCEED" "$report" 2>/dev/null; then
|
|
echo "$gate_id: PASSED"
|
|
fi
|
|
fi
|
|
|
|
# Check main directory: user-testing/reports/TG-X.Y_*_report.md
|
|
|
|
if [ ! -d "user-testing/reports/$gate_id" ]; then
|
|
report=$(find "user-testing/reports" -maxdepth 1 -name "${gate_id}_*report.md" 2>/dev/null | head -1)
|
|
if [ -n "$report" ] && grep -q "PROCEED" "$report" 2>/dev/null; then
|
|
echo "$gate_id: PASSED"
|
|
fi
|
|
fi
|
|
|
|
```text
|
|
|
|
Build a list of passed gates.
|
|
|
|
## Step 3: Find Next Test Gate
|
|
|
|
Walk through all gates in sorted order. For each gate:
|
|
|
|
1. **Skip if already passed** (from Step 2)
|
|
2. **Check if prerequisites are met:**
|
|
- Get the gate's `requires` array from the config
|
|
- Check if all required test gates have passed
|
|
3. **First non-passed gate with prerequisites met = next gate**
|
|
|
|
Get gate info from config:
|
|
|
|
```bash
|
|
gate_id="TG-X.Y"
|
|
cat /tmp/testgates_config.json | python3 -c "
|
|
import json, sys
|
|
config = json.load(sys.stdin)
|
|
gate = config['test_gates'].get('$gate_id', {})
|
|
print('Name:', gate.get('name', 'Unknown'))
|
|
print('Requires:', ','.join(gate.get('requires', [])))
|
|
print('Script:', gate.get('script', 'N/A'))
|
|
"
|
|
|
|
```text
|
|
|
|
## Step 3.5: Check Story Implementation Status
|
|
|
|
Before suggesting a test gate, check if the required story is actually implemented.
|
|
|
|
**Check common implementation indicators based on gate type:**
|
|
|
|
```bash
|
|
gate_id="TG-X.Y" # e.g., "TG-2.3"
|
|
|
|
# Define expected files for each gate (examples)
|
|
|
|
case "$gate_id" in
|
|
"TG-1.1")
|
|
# Agent Framework - check for strands setup
|
|
files=("requirements.txt")
|
|
;;
|
|
"TG-1.2")
|
|
# Word Parser - check for parser implementation
|
|
files=("src/agents/input_parser/word_parser.py" "src/parsers/word_parser.py")
|
|
;;
|
|
"TG-1.3")
|
|
# Excel Parser - check for parser implementation
|
|
files=("src/agents/input_parser/excel_parser.py" "src/parsers/excel_parser.py")
|
|
;;
|
|
"TG-2.3")
|
|
# Core Templates - check for 5 key template files
|
|
files=(
|
|
"src/templates/secil/title_slide.html.j2"
|
|
"src/templates/secil/big_number.html.j2"
|
|
"src/templates/secil/three_metrics.html.j2"
|
|
"src/templates/secil/bullet_list.html.j2"
|
|
"src/templates/secil/chart_template.html.j2"
|
|
)
|
|
;;
|
|
"TG-3.3")
|
|
# PptxGenJS POC - check for Node.js conversion script
|
|
files=("src/converters/conversion_scripts/convert_to_pptx.js")
|
|
;;
|
|
"TG-3.4")
|
|
# Full Pipeline - check for complete conversion implementation
|
|
files=("src/converters/nodejs_bridge.py" "src/converters/conversion_scripts/convert_to_pptx.js")
|
|
;;
|
|
"TG-4.2")
|
|
# Checkpoint Flow - check for orchestration with checkpoints
|
|
files=("src/orchestration/checkpoints.py")
|
|
;;
|
|
"TG-4.6")
|
|
# E2E MVP - check for main orchestrator
|
|
files=("src/main.py" "src/orchestration/orchestrator.py")
|
|
;;
|
|
*)
|
|
# Unknown gate - skip file checks
|
|
files=()
|
|
;;
|
|
esac
|
|
|
|
# Check if files exist
|
|
|
|
missing_files=()
|
|
for file in "${files[@]}"; do
|
|
if [ ! -f "$file" ]; then
|
|
missing_files+=("$file")
|
|
fi
|
|
done
|
|
|
|
# Output result
|
|
|
|
if [ ${#missing_files[@]} -gt 0 ]; then
|
|
echo "STORY_NOT_READY"
|
|
printf '%s\n' "${missing_files[@]}"
|
|
else
|
|
echo "STORY_READY"
|
|
fi
|
|
|
|
```text
|
|
|
|
**Store the story readiness status** to use in Step 4.
|
|
|
|
## Step 4: Show Gate Status to User
|
|
|
|
**Format output like this:**
|
|
|
|
If some gates already passed:
|
|
|
|
```text
|
|
|
|
================================================================================
|
|
Passed Gates:
|
|
✅ TG-1.1 - Agent Framework Validation (PASSED)
|
|
✅ TG-1.2 - Word Parser Validation (PASSED)
|
|
|
|
🎯 Next Test Gate: TG-1.3 - Excel Parser Validation
|
|
================================================================================
|
|
|
|
```text
|
|
|
|
If story is NOT READY (implementation files missing from Step 3.5):
|
|
|
|
```text
|
|
|
|
⏳ Story [X.Y] NOT IMPLEMENTED
|
|
|
|
Required story: Story [X.Y] - [Story Name]
|
|
|
|
Missing implementation files:
|
|
❌ src/templates/secil/title_slide.html.j2
|
|
❌ src/templates/secil/big_number.html.j2
|
|
❌ src/templates/secil/three_metrics.html.j2
|
|
❌ src/templates/secil/bullet_list.html.j2
|
|
❌ src/templates/secil/chart_template.html.j2
|
|
|
|
Please complete Story [X.Y] implementation first.
|
|
|
|
Once complete, run: /usertestgates
|
|
|
|
```text
|
|
|
|
If gate is READY (story implemented AND all prerequisite gates passed):
|
|
|
|
```text
|
|
|
|
✅ This gate is READY to run
|
|
|
|
Prerequisites: All prerequisite test gates have passed
|
|
Story Status: ✅ Story [X.Y] implemented
|
|
|
|
Script: user-testing/scripts/TG-1.3_excel_parser_validation.py
|
|
|
|
Run TG-1.3 now? (Y/N)
|
|
|
|
```text
|
|
|
|
If gate is NOT READY (prerequisite gates not passed):
|
|
|
|
```text
|
|
|
|
⏳ Complete these test gates first:
|
|
|
|
❌ TG-1.1 - Agent Framework Validation (not passed)
|
|
|
|
Once complete, run: /usertestgates
|
|
|
|
```text
|
|
|
|
## Step 5: Execute Gate if User Confirms
|
|
|
|
If gate is ready and user types Y or Yes:
|
|
|
|
### Detect if Test Gate is Interactive
|
|
|
|
Check if the test gate script contains `input()` calls (interactive):
|
|
|
|
```bash
|
|
gate_script="user-testing/scripts/TG-X.Y_*_validation.py"
|
|
if grep -q "input(" "$gate_script" 2>/dev/null; then
|
|
echo "INTERACTIVE"
|
|
else
|
|
echo "NON_INTERACTIVE"
|
|
fi
|
|
|
|
```text
|
|
|
|
### For NON-INTERACTIVE Gates
|
|
|
|
Run directly:
|
|
|
|
```bash
|
|
python3 user-testing/scripts/TG-X.Y_*_validation.py
|
|
|
|
```text
|
|
|
|
Show the exit code and interpret:
|
|
|
|
- Exit 0 → ✅ PROCEED
|
|
- Exit 1 → ⚠️ REFINE
|
|
- Exit 2 → 🚨 ESCALATE
|
|
- Exit 130 → ⚠️ Interrupted
|
|
|
|
Check for report in `user-testing/reports/TG-X.Y/` and mention it
|
|
|
|
### For INTERACTIVE Gates (Agent-Guided Mode)
|
|
|
|
**Step 5a: Run Parse Phase**
|
|
|
|
```bash
|
|
python3 user-testing/scripts/TG-X.Y_*_validation.py --phase=parse
|
|
|
|
```text
|
|
|
|
This outputs parsed data to `/tmp/tg-X.Y-parse-results.json`
|
|
|
|
**Step 5b: Load Parse Results and Collect User Answers**
|
|
|
|
Load the parse results:
|
|
|
|
```bash
|
|
cat /tmp/tg-X.Y-parse-results.json
|
|
|
|
```text
|
|
|
|
For TG-1.3 (Excel Parser), the parse results contain:
|
|
|
|
- `workbooks`: Array of parsed workbook data
|
|
- `total_checks`: Number of validation checks needed (e.g., 30)
|
|
|
|
For each workbook, you need to ask the user to validate 6 checks. The validation questions are:
|
|
|
|
1. Sheet Extraction: "All sheets identified and named correctly?"
|
|
2. Table Accuracy: "Headers and rows extracted completely?"
|
|
3. Metrics Calculation: "Min/max/mean/trend computed accurately?"
|
|
4. Chart Suggestions: "Appropriate chart types suggested?"
|
|
5. Edge Cases: "Formulas, empty cells, special chars handled?"
|
|
6. Data Contract: "Output matches expected JSON schema?"
|
|
|
|
**For each check:**
|
|
7. Show the user the parsed data (from `/tmp/` or parse results)
|
|
8. Ask: "Check N/30: [description] - How do you assess this? (PASS/FAIL/PARTIAL/N/A)"
|
|
9. Collect: status (PASS/FAIL/PARTIAL/N/A) and optional notes
|
|
10. Store in answers array
|
|
|
|
**Step 5c: Create Answers JSON**
|
|
|
|
Create `/tmp/tg-X.Y-answers.json`:
|
|
|
|
```json
|
|
{
|
|
"test_gate": "TG-X.Y",
|
|
"test_date": "2025-10-10T12:00:00",
|
|
"checks": [
|
|
{
|
|
"check_num": 1,
|
|
"status": "PASS",
|
|
"notes": "All sheets extracted correctly"
|
|
},
|
|
{
|
|
"check_num": 2,
|
|
"status": "PASS",
|
|
"notes": "Headers and data accurate"
|
|
}
|
|
]
|
|
}
|
|
|
|
```text
|
|
|
|
**Step 5d: Run Report Phase**
|
|
|
|
```bash
|
|
python3 user-testing/scripts/TG-X.Y_*_validation.py --phase=report --answers=/tmp/tg-X.Y-answers.json
|
|
|
|
```text
|
|
|
|
This generates the final report in `user-testing/reports/TG-X.Y/` with:
|
|
|
|
- User's validation answers
|
|
- Recommendation (PROCEED/REFINE/ESCALATE)
|
|
- Exit code (0/1/2)
|
|
|
|
Show the exit code and interpret:
|
|
|
|
- Exit 0 → ✅ PROCEED
|
|
- Exit 1 → ⚠️ REFINE
|
|
- Exit 2 → 🚨 ESCALATE
|
|
|
|
## Special Cases
|
|
|
|
**All gates passed:**
|
|
|
|
```text
|
|
|
|
================================================================================
|
|
🎉 ALL TEST GATES PASSED!
|
|
================================================================================
|
|
|
|
✅ TG-1.1 - Agent Framework Validation
|
|
✅ TG-1.2 - Word Parser Validation
|
|
...
|
|
✅ TG-4.6 - End-to-End MVP Validation
|
|
|
|
MVP is complete! 🎉
|
|
|
|
```text
|
|
|
|
**No gates found:**
|
|
|
|
```text
|
|
|
|
❌ No test gates configured. Check /tmp/testgates_config.json
|
|
|
|
```text
|
|
|
|
---
|
|
|
|
## Execution Notes
|
|
|
|
- Use bash commands with proper error handling
|
|
- Check gate completion ONLY via report files (not implementation files)
|
|
- Get all gate info dynamically from `/tmp/testgates_config.json`
|
|
- Keep output clean and focused
|
|
- **Always show progress** (passed gates list)
|
|
- **Always show next step** (what gate is next)
|
|
- **Make it actionable** (clear instructions)
|
|
- **Let test gate scripts validate story completion** - don't check files here!
|