11 KiB
11 KiB
| description | prerequisites | argument-hint | allowed-tools | ||
|---|---|---|---|---|---|
| Finds and runs next test gate | test gates in project | no arguments needed - auto-detects next gate |
|
⚠️ PROJECT-SPECIFIC COMMAND - Requires test gates infrastructure
This command requires:
- ~/.claude/lib/testgates_discovery.py (test gate discovery script)
- docs/epics.md (or similar) with test gate definitions
- user-testing/scripts/ directory with validation scripts
- user-testing/reports/ directory for results
The file path checks in Step 3.5 are project-specific examples that should be customized for your project's implementation structure
Test Gate Finder & Executor
Your task: Find the next test gate to run, show the user what's needed, and execute it if they confirm.
Step 1: Discover Test Gates and Prerequisites
First, check if the required infrastructure exists:
# ============================================
# PRE-FLIGHT CHECKS (Infrastructure Validation)
# ============================================
TESTGATES_SCRIPT="$HOME/.claude/lib/testgates_discovery.py"
# Check if discovery script exists
if [[ ! -f "$TESTGATES_SCRIPT" ]]; then
echo "❌ Test gates discovery script not found"
echo " Expected: $TESTGATES_SCRIPT"
echo ""
echo " This command requires the testgates_discovery.py library."
echo " It is designed for projects with test gate infrastructure."
exit 1
fi
# Check for epic definition files
EPICS_FILE=""
for file in "docs/epics.md" "docs/EPICS.md" "docs/test-gates.md" "EPICS.md"; do
if [[ -f "$file" ]]; then
EPICS_FILE="$file"
echo "📁 Found epics file: $EPICS_FILE"
break
fi
done
if [[ -z "$EPICS_FILE" ]]; then
echo "⚠️ No epics definition file found"
echo " Searched: docs/epics.md, docs/EPICS.md, docs/test-gates.md, EPICS.md"
echo " Test gate discovery may fail without this file."
fi
# Check for user-testing directory structure
if [[ ! -d "user-testing" ]]; then
echo "⚠️ No user-testing/ directory found"
echo " This command expects user-testing/scripts/ and user-testing/reports/"
echo " Creating minimal structure..."
mkdir -p user-testing/scripts user-testing/reports
fi
```text
Run the discovery script to get test gate configuration:
```bash
python3 "$TESTGATES_SCRIPT" . --format json > /tmp/testgates_config.json 2>/dev/null
```text
If this fails or produces empty output, tell the user:
```text
❌ Failed to discover test gates from epic definition file
Make sure docs/epics.md (or similar) exists with story and test gate definitions.
```text
## Step 2: Check Which Gates Have Already Passed
Parse the config to get list of all test gates in order:
```bash
cat /tmp/testgates_config.json | python3 -c "
import json, sys
config = json.load(sys.stdin)
gates = config.get('test_gates', {})
for gate_id in sorted(gates.keys()):
print(gate_id)
"
```text
For each gate, check if it has passed by looking for a report with "PROCEED":
```bash
gate_id="TG-X.Y" # Replace with actual gate ID
# Check subdirectory first: user-testing/reports/TG-X.Y/
if [ -d "user-testing/reports/$gate_id" ]; then
report=$(find "user-testing/reports/$gate_id" -name "*report.md" 2>/dev/null | head -1)
if [ -n "$report" ] && grep -q "PROCEED" "$report" 2>/dev/null; then
echo "$gate_id: PASSED"
fi
fi
# Check main directory: user-testing/reports/TG-X.Y_*_report.md
if [ ! -d "user-testing/reports/$gate_id" ]; then
report=$(find "user-testing/reports" -maxdepth 1 -name "${gate_id}_*report.md" 2>/dev/null | head -1)
if [ -n "$report" ] && grep -q "PROCEED" "$report" 2>/dev/null; then
echo "$gate_id: PASSED"
fi
fi
```text
Build a list of passed gates.
## Step 3: Find Next Test Gate
Walk through all gates in sorted order. For each gate:
1. **Skip if already passed** (from Step 2)
2. **Check if prerequisites are met:**
- Get the gate's `requires` array from the config
- Check if all required test gates have passed
3. **First non-passed gate with prerequisites met = next gate**
Get gate info from config:
```bash
gate_id="TG-X.Y"
cat /tmp/testgates_config.json | python3 -c "
import json, sys
config = json.load(sys.stdin)
gate = config['test_gates'].get('$gate_id', {})
print('Name:', gate.get('name', 'Unknown'))
print('Requires:', ','.join(gate.get('requires', [])))
print('Script:', gate.get('script', 'N/A'))
"
```text
## Step 3.5: Check Story Implementation Status
Before suggesting a test gate, check if the required story is actually implemented.
**Check common implementation indicators based on gate type:**
```bash
gate_id="TG-X.Y" # e.g., "TG-2.3"
# Define expected files for each gate (examples)
case "$gate_id" in
"TG-1.1")
# Agent Framework - check for strands setup
files=("requirements.txt")
;;
"TG-1.2")
# Word Parser - check for parser implementation
files=("src/agents/input_parser/word_parser.py" "src/parsers/word_parser.py")
;;
"TG-1.3")
# Excel Parser - check for parser implementation
files=("src/agents/input_parser/excel_parser.py" "src/parsers/excel_parser.py")
;;
"TG-2.3")
# Core Templates - check for 5 key template files
files=(
"src/templates/secil/title_slide.html.j2"
"src/templates/secil/big_number.html.j2"
"src/templates/secil/three_metrics.html.j2"
"src/templates/secil/bullet_list.html.j2"
"src/templates/secil/chart_template.html.j2"
)
;;
"TG-3.3")
# PptxGenJS POC - check for Node.js conversion script
files=("src/converters/conversion_scripts/convert_to_pptx.js")
;;
"TG-3.4")
# Full Pipeline - check for complete conversion implementation
files=("src/converters/nodejs_bridge.py" "src/converters/conversion_scripts/convert_to_pptx.js")
;;
"TG-4.2")
# Checkpoint Flow - check for orchestration with checkpoints
files=("src/orchestration/checkpoints.py")
;;
"TG-4.6")
# E2E MVP - check for main orchestrator
files=("src/main.py" "src/orchestration/orchestrator.py")
;;
*)
# Unknown gate - skip file checks
files=()
;;
esac
# Check if files exist
missing_files=()
for file in "${files[@]}"; do
if [ ! -f "$file" ]; then
missing_files+=("$file")
fi
done
# Output result
if [ ${#missing_files[@]} -gt 0 ]; then
echo "STORY_NOT_READY"
printf '%s\n' "${missing_files[@]}"
else
echo "STORY_READY"
fi
```text
**Store the story readiness status** to use in Step 4.
## Step 4: Show Gate Status to User
**Format output like this:**
If some gates already passed:
```text
================================================================================
Passed Gates:
✅ TG-1.1 - Agent Framework Validation (PASSED)
✅ TG-1.2 - Word Parser Validation (PASSED)
🎯 Next Test Gate: TG-1.3 - Excel Parser Validation
================================================================================
```text
If story is NOT READY (implementation files missing from Step 3.5):
```text
⏳ Story [X.Y] NOT IMPLEMENTED
Required story: Story [X.Y] - [Story Name]
Missing implementation files:
❌ src/templates/secil/title_slide.html.j2
❌ src/templates/secil/big_number.html.j2
❌ src/templates/secil/three_metrics.html.j2
❌ src/templates/secil/bullet_list.html.j2
❌ src/templates/secil/chart_template.html.j2
Please complete Story [X.Y] implementation first.
Once complete, run: /usertestgates
```text
If gate is READY (story implemented AND all prerequisite gates passed):
```text
✅ This gate is READY to run
Prerequisites: All prerequisite test gates have passed
Story Status: ✅ Story [X.Y] implemented
Script: user-testing/scripts/TG-1.3_excel_parser_validation.py
Run TG-1.3 now? (Y/N)
```text
If gate is NOT READY (prerequisite gates not passed):
```text
⏳ Complete these test gates first:
❌ TG-1.1 - Agent Framework Validation (not passed)
Once complete, run: /usertestgates
```text
## Step 5: Execute Gate if User Confirms
If gate is ready and user types Y or Yes:
### Detect if Test Gate is Interactive
Check if the test gate script contains `input()` calls (interactive):
```bash
gate_script="user-testing/scripts/TG-X.Y_*_validation.py"
if grep -q "input(" "$gate_script" 2>/dev/null; then
echo "INTERACTIVE"
else
echo "NON_INTERACTIVE"
fi
```text
### For NON-INTERACTIVE Gates
Run directly:
```bash
python3 user-testing/scripts/TG-X.Y_*_validation.py
```text
Show the exit code and interpret:
- Exit 0 → ✅ PROCEED
- Exit 1 → ⚠️ REFINE
- Exit 2 → 🚨 ESCALATE
- Exit 130 → ⚠️ Interrupted
Check for report in `user-testing/reports/TG-X.Y/` and mention it
### For INTERACTIVE Gates (Agent-Guided Mode)
**Step 5a: Run Parse Phase**
```bash
python3 user-testing/scripts/TG-X.Y_*_validation.py --phase=parse
```text
This outputs parsed data to `/tmp/tg-X.Y-parse-results.json`
**Step 5b: Load Parse Results and Collect User Answers**
Load the parse results:
```bash
cat /tmp/tg-X.Y-parse-results.json
```text
For TG-1.3 (Excel Parser), the parse results contain:
- `workbooks`: Array of parsed workbook data
- `total_checks`: Number of validation checks needed (e.g., 30)
For each workbook, you need to ask the user to validate 6 checks. The validation questions are:
1. Sheet Extraction: "All sheets identified and named correctly?"
2. Table Accuracy: "Headers and rows extracted completely?"
3. Metrics Calculation: "Min/max/mean/trend computed accurately?"
4. Chart Suggestions: "Appropriate chart types suggested?"
5. Edge Cases: "Formulas, empty cells, special chars handled?"
6. Data Contract: "Output matches expected JSON schema?"
**For each check:**
7. Show the user the parsed data (from `/tmp/` or parse results)
8. Ask: "Check N/30: [description] - How do you assess this? (PASS/FAIL/PARTIAL/N/A)"
9. Collect: status (PASS/FAIL/PARTIAL/N/A) and optional notes
10. Store in answers array
**Step 5c: Create Answers JSON**
Create `/tmp/tg-X.Y-answers.json`:
```json
{
"test_gate": "TG-X.Y",
"test_date": "2025-10-10T12:00:00",
"checks": [
{
"check_num": 1,
"status": "PASS",
"notes": "All sheets extracted correctly"
},
{
"check_num": 2,
"status": "PASS",
"notes": "Headers and data accurate"
}
]
}
```text
**Step 5d: Run Report Phase**
```bash
python3 user-testing/scripts/TG-X.Y_*_validation.py --phase=report --answers=/tmp/tg-X.Y-answers.json
```text
This generates the final report in `user-testing/reports/TG-X.Y/` with:
- User's validation answers
- Recommendation (PROCEED/REFINE/ESCALATE)
- Exit code (0/1/2)
Show the exit code and interpret:
- Exit 0 → ✅ PROCEED
- Exit 1 → ⚠️ REFINE
- Exit 2 → 🚨 ESCALATE
## Special Cases
**All gates passed:**
```text
================================================================================
🎉 ALL TEST GATES PASSED!
================================================================================
✅ TG-1.1 - Agent Framework Validation
✅ TG-1.2 - Word Parser Validation
...
✅ TG-4.6 - End-to-End MVP Validation
MVP is complete! 🎉
```text
**No gates found:**
```text
❌ No test gates configured. Check /tmp/testgates_config.json
```text
---
## Execution Notes
- Use bash commands with proper error handling
- Check gate completion ONLY via report files (not implementation files)
- Get all gate info dynamically from `/tmp/testgates_config.json`
- Keep output clean and focused
- **Always show progress** (passed gates list)
- **Always show next step** (what gate is next)
- **Make it actionable** (clear instructions)
- **Let test gate scripts validate story completion** - don't check files here!