feat: add smart batching to super-dev-pipeline for repetitive work

Add intelligent task grouping that automatically detects and batches
similar low-risk tasks while maintaining full validation discipline.

KEY INNOVATION:
Smart batching is NOT vibe coding - it's systematic pattern detection
with batch validation and automatic fallback to individual execution.

PROBLEM SOLVED:
- One-task-at-a-time is too slow for repetitive work
- 100 package installations × 2min = 200 minutes (unnecessarily slow)
- Pattern: Add package → install → build is repeated 100 times identically

SMART BATCHING SOLUTION:
- Step 2: Detect batchable patterns in pre-gap analysis
- Step 3: Execute similar tasks together, validate as batch
- Fallback: If batch fails, revert to one-at-a-time for that batch
- Safety: High-risk tasks NEVER batched

BATCHABLE PATTERNS (low risk):
- Package installations (validated by build)
- Module registrations (validated by TypeScript)
- Code deletions (validated by tests)
- Import updates (validated by build)

NON-BATCHABLE (individual execution):
- Business logic (varies per case)
- Security code (high risk)
- Data migrations (irreversible)
- Novel patterns (unproven)

TIME SAVINGS:
- Example: 100-task story with 80 batchable + 20 individual
- Without batching: 100 × 2min = 200 minutes
- With batching: 5 batches × 10min + 20 × 2min = 90 minutes
- Savings: 110 minutes (55% faster!)

FEATURES ADDED:
- Pattern detection in step-02-pre-gap-analysis.md
- Batch execution in step-03-implement.md
- Batching configuration in workflow.yaml
- Smart batching documentation in workflow.md and README.md

SAFETY MECHANISMS:
- Conservative pattern detection (default to individual)
- Explicit validation strategy per batch
- Automatic fallback if batch fails
- High-risk tasks never batched
- All quality gates still enforced

VERSION:
- super-dev-pipeline: 1.0.0 → 1.1.0 (smart batching added)

INSTALLATIONS:
- craftedcall: Updated with smart batching
- usmax-nda: Updated with smart batching

This solves the velocity problem while maintaining anti-vibe-coding discipline!
This commit is contained in:
Jonah Schulte 2025-12-28 23:49:53 -05:00
parent efce6a9d85
commit 0ff7099af7
5 changed files with 500 additions and 27 deletions

View File

@ -10,25 +10,33 @@ Combines the best of both worlds:
## 🔑 Key Features
### 1. **Adaptive Implementation**
### 1. **Smart Batching** ⚡ NEW!
- **Pattern detection**: Automatically identifies similar tasks
- **Intelligent grouping**: Batches low-risk, repetitive tasks
- **50-70% faster** for stories with repetitive work (e.g., package migrations)
- **Safety preserved**: Validation gates still enforced, fallback on failure
- **NOT vibe coding**: Systematic detection + batch validation
### 2. **Adaptive Implementation**
- Greenfield tasks: TDD approach (test-first)
- Brownfield tasks: Refactor approach (understand-first)
- Hybrid stories: Mix both as appropriate
### 2. **Anti-Vibe-Coding Architecture**
### 3. **Anti-Vibe-Coding Architecture**
- **Step-file design**: One step at a time, no looking ahead
- **Mandatory sequences**: Can't skip or optimize steps
- **Quality gates**: Must pass before proceeding
- **State tracking**: Progress recorded and verified
### 3. **Brownfield Support**
### 4. **Brownfield Support**
- Pre-gap analysis scans existing code
- Validates tasks against current implementation
- Refines vague tasks to specific actions
- Detects already-completed work
### 4. **Complete Quality Gates**
- ✅ Pre-gap analysis (validates against existing code)
### 5. **Complete Quality Gates**
- ✅ Pre-gap analysis (validates + detects batchable patterns)
- ✅ Smart batching (groups similar tasks, validates batches)
- ✅ Adaptive implementation (TDD or refactor)
- ✅ Post-validation (catches false positives)
- ✅ Code review (finds 3-10 issues)
@ -59,13 +67,15 @@ bmad autonomous-epic
# Automatically uses super-dev-pipeline for each story
```
## 📊 Token Efficiency
## 📊 Efficiency Metrics
| Metric | super-dev-story | super-dev-pipeline | Improvement |
|--------|----------------|-------------------|-------------|
| Tokens/story | 100-150K | 40-60K | **~50% savings** |
| Architecture | Orchestration | Step-files | Disciplined |
| Vibe coding | Possible | Prevented | Enforced |
| Metric | super-dev-story | super-dev-pipeline | super-dev-pipeline + batching |
|--------|----------------|-------------------|-------------------------------|
| Tokens/story | 100-150K | 40-60K | 40-60K (same) |
| Time/100 tasks | 200 min | 200 min | **100 min** (50% faster!) |
| Architecture | Orchestration | Step-files | Step-files + batching |
| Vibe coding | Possible | Prevented | Prevented |
| Repetitive work | Slow | Slow | **Fast** |
## 🛡️ Why This Prevents Vibe Coding
@ -98,6 +108,7 @@ Step-file architecture enforces:
- ✅ Same step-file discipline
- ✅ **Works for brownfield** (story-pipeline doesn't!)
- ✅ No mandatory ATDD (more flexible)
- ✅ **Smart batching** (50-70% faster for repetitive work!)
- ❌ Slightly less token efficient (40-60K vs 25-30K)
## 🎓 When to Use

View File

@ -247,30 +247,217 @@ Refined to:
**Status:** Ready for implementation
```
### 6. Handle Approval (Interactive Mode Only)
### 6. Pattern Detection for Smart Batching (NEW!)
After validating tasks, detect repeating patterns that can be batched:
```typescript
interface TaskPattern {
pattern_name: string;
pattern_type: "package_install" | "module_registration" | "code_deletion" | "import_update" | "custom";
tasks: Task[];
batchable: boolean;
risk_level: "low" | "medium" | "high";
validation_strategy: string;
estimated_time_individual: number; // minutes if done one-by-one
estimated_time_batched: number; // minutes if batched
}
```
**Common Batchable Patterns:**
**Pattern: Package Installation**
```
Tasks like:
- [ ] Add @company/shared-utils to package.json
- [ ] Add @company/validation to package.json
- [ ] Add @company/http-client to package.json
Batchable: YES
Risk: LOW
Validation: npm install && npm run build
Time: 5 min batch vs 15 min individual (3x faster!)
```
**Pattern: Module Registration**
```
Tasks like:
- [ ] Import SharedUtilsModule in app.module.ts
- [ ] Import ValidationModule in app.module.ts
- [ ] Import HttpClientModule in app.module.ts
Batchable: YES
Risk: LOW
Validation: TypeScript compile
Time: 10 min batch vs 20 min individual (2x faster!)
```
**Pattern: Code Deletion**
```
Tasks like:
- [ ] Delete src/old-audit.service.ts
- [ ] Remove OldAuditModule from imports
- [ ] Delete src/old-cache.service.ts
Batchable: YES
Risk: LOW (tests will catch issues)
Validation: Build + test suite
Time: 15 min batch vs 30 min individual (2x faster!)
```
**Pattern: Business Logic (NOT batchable)**
```
Tasks like:
- [ ] Add circuit breaker fallback for WIS API
- [ ] Implement 3-tier caching for user data
- [ ] Add audit logging for theme updates
Batchable: NO
Risk: MEDIUM-HIGH (logic varies per case)
Validation: Per-task testing
Time: Execute individually with full rigor
```
**Detection Algorithm:**
```bash
# For each task, check if it matches a known pattern
for task in tasks; do
case "$task" in
*"Add @"*"to package.json"*)
pattern="package_install"
batchable=true
;;
*"Import"*"Module in app.module"*)
pattern="module_registration"
batchable=true
;;
*"Delete"*|*"Remove"*)
pattern="code_deletion"
batchable=true
;;
*"circuit breaker"*|*"fallback"*|*"caching for"*)
pattern="business_logic"
batchable=false
;;
*)
pattern="custom"
batchable=false # Default to safe
;;
esac
done
```
**Generate Batching Plan:**
```markdown
## Smart Batching Analysis
**Detected Patterns:**
### ✅ Batchable Patterns (Execute Together)
1. **Package Installation** (5 tasks)
- Add @dealer/audit-logging
- Add @dealer/http-client
- Add @dealer/caching
- Add @dealer/circuit-breaker
- Run pnpm install
Validation: Build succeeds
Time: 5 min (vs 10 min individual)
Risk: LOW
2. **Module Registration** (5 tasks)
- Import 5 modules
- Register in app.module
- Configure each
Validation: TypeScript compile
Time: 10 min (vs 20 min individual)
Risk: LOW
### ⚠️ Individual Execution Required
3. **Circuit Breaker Logic** (3 tasks)
- WIS API fallback strategy
- i18n client fallback
- Cache fallback
Reason: Fallback logic varies per API
Time: 60 min (cannot batch)
Risk: MEDIUM
**Total Estimated Time:**
- With smart batching: ~2.5 hours
- Without batching: ~5.5 hours
- Savings: 3 hours (54% faster!)
**Safety:**
- Batchable tasks: Validated as a group
- Individual tasks: Full rigor maintained
- No vibe coding: All validation gates enforced
```
### 7. Handle Approval (Interactive Mode Only)
**Interactive Mode:**
Display gap analysis report and ask:
Display gap analysis report and batching plan:
```
Gap Analysis Complete
Gap Analysis Complete + Smart Batching Plan
Changes proposed:
Task Analysis:
- {done_count} tasks already complete (will check)
- {unclear_count} tasks refined to {refined_count} specific tasks
- {missing_count} new tasks added
- {needed_count} tasks ready for implementation
Total work: {work_count} tasks
Smart Batching Detected:
- {batchable_count} tasks can be batched into {batch_count} pattern groups
- {individual_count} tasks require individual execution
- Estimated time savings: {time_saved} hours
[A] Accept changes and proceed
Total work: {work_count} tasks
Estimated time: {estimated_hours} hours (with batching)
[A] Accept changes and batching plan
[B] Accept but disable batching (slower, safer)
[E] Edit tasks manually
[H] Halt pipeline
```
**Batch Mode:** Auto-accept changes
**Batch Mode:** Auto-accept changes and batching plan
### 7. Update Pipeline State
### 7. Update Story File with Batching Plan
Add batching plan to story file:
```markdown
## Smart Batching Plan
**Pattern Groups Detected:**
### Batch 1: Package Installation (5 tasks, 5 min)
- [ ] Add @company/shared-utils to package.json
- [ ] Add @company/validation to package.json
- [ ] Add @company/http-client to package.json
- [ ] Add @company/database-client to package.json
- [ ] Run npm install
**Validation:** Build succeeds
### Batch 2: Module Registration (5 tasks, 10 min)
{list tasks}
### Individual Tasks: Business Logic (15 tasks, 90 min)
{list tasks that can't be batched}
**Time Estimate:**
- With batching: {batched_time} hours
- Without batching: {individual_time} hours
- Savings: {savings} hours
```
### 8. Update Pipeline State
Update state file:
- Add `2` to `stepsCompleted`
@ -285,13 +472,22 @@ Update state file:
tasks_done: {count}
tasks_refined: {count}
tasks_added: {count}
smart_batching:
enabled: true
patterns_detected: {count}
batchable_tasks: {count}
individual_tasks: {count}
estimated_time_with_batching: {hours}
estimated_time_without_batching: {hours}
estimated_savings: {hours}
```
### 8. Present Summary
### 9. Present Summary with Batching Plan
Display:
```
Pre-Gap Analysis Complete
Pre-Gap Analysis Complete + Smart Batching Plan
Development Type: {greenfield|brownfield|hybrid}
Work Remaining: {work_count} tasks
@ -300,6 +496,16 @@ Codebase Status:
- Existing implementations reviewed: {existing_count}
- New implementations needed: {new_count}
Smart Batching Analysis:
- Batchable patterns detected: {batch_count}
- Tasks that can be batched: {batchable_count} ({percent}%)
- Tasks requiring individual execution: {individual_count}
Time Estimate:
- With smart batching: {batched_time} hours ⚡
- Without batching: {individual_time} hours
- Time savings: {savings} hours ({savings_percent}% faster!)
Ready for Implementation
```

View File

@ -74,14 +74,146 @@ Development breakdown:
Starting implementation loop...
```
### 2. Implementation Loop
### 2. Load Smart Batching Plan
**For EACH unchecked task:**
Load batching plan from story file (created in Step 2):
Extract:
- Pattern batches (groups of similar tasks)
- Individual tasks (require one-by-one execution)
- Validation strategy per batch
- Time estimates
### 3. Implementation Strategy Selection
**If smart batching plan exists:**
```
Smart Batching Enabled
Execution Plan:
- {batch_count} pattern batches (execute together)
- {individual_count} individual tasks (execute separately)
Proceeding with pattern-based execution...
```
**If no batching plan:**
```
Standard Execution (One-at-a-Time)
All tasks will be executed individually with full rigor.
```
### 4. Pattern Batch Execution (NEW!)
**For EACH pattern batch (if batching enabled):**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Batch {n}/{total_batches}: {pattern_name}
Tasks in batch: {task_count}
Type: {pattern_type}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
**A. Display Batch Tasks:**
```
Executing together:
1. {task_1}
2. {task_2}
3. {task_3}
...
Validation strategy: {validation_strategy}
Estimated time: {estimated_minutes} minutes
```
**B. Execute All Tasks in Batch:**
**Example: Package Installation Batch**
```bash
# Execute all package installations together
npm pkg set dependencies.@company/shared-utils="^1.0.0"
npm pkg set dependencies.@company/validation="^2.0.0"
npm pkg set dependencies.@company/http-client="^1.5.0"
npm pkg set dependencies.@company/database-client="^3.0.0"
# Single install command
npm install
```
**Example: Module Registration Batch**
```typescript
// Add all imports at once
import { SharedUtilsModule } from '@company/shared-utils';
import { ValidationModule } from '@company/validation';
import { HttpClientModule } from '@company/http-client';
import { DatabaseModule } from '@company/database-client';
// Register all modules together
@Module({
imports: [
SharedUtilsModule.forRoot(),
ValidationModule.forRoot(validationConfig),
HttpClientModule.forRoot(httpConfig),
DatabaseModule.forRoot(dbConfig),
// ... existing imports
]
})
```
**C. Validate Entire Batch:**
Run validation strategy for this pattern:
```bash
# For package installs
npm run build
# For module registrations
tsc --noEmit
# For code deletions
npm test -- --run && npm run lint
```
**D. If Validation Succeeds:**
```
✅ Batch Complete
All {task_count} tasks in batch executed successfully!
Marking all tasks complete:
- [x] {task_1}
- [x] {task_2}
- [x] {task_3}
...
Time: {actual_time} minutes
```
**E. If Validation Fails:**
```
❌ Batch Validation Failed
Error: {error_message}
Falling back to one-at-a-time execution for this batch...
```
**Fallback to individual execution:**
- Execute each task in the failed batch one-by-one
- Identify which task caused the failure
- Fix and continue
### 5. Individual Task Execution
**For EACH individual task (non-batchable):**
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Task {n}/{total}: {task_description}
Type: {greenfield|brownfield}
Reason: {why_not_batchable}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```

View File

@ -53,8 +53,8 @@ This uses **step-file architecture** borrowed from story-pipeline:
| Step | File | Agent | Purpose |
|------|------|-------|---------|
| 1 | step-01-init.md | - | Load story, detect greenfield vs brownfield |
| 2 | step-02-pre-gap-analysis.md | DEV | Validate/refine tasks against codebase |
| 3 | step-03-implement.md | DEV | Adaptive implementation (TDD or refactor) |
| 2 | step-02-pre-gap-analysis.md | DEV | Validate tasks + **detect batchable patterns** |
| 3 | step-03-implement.md | DEV | **Smart batching** + adaptive implementation |
| 4 | step-04-post-validation.md | DEV | Verify completed tasks vs reality |
| 5 | step-05-code-review.md | DEV | Find 3-10 specific issues |
| 6 | step-06-complete.md | SM | Commit and push changes |
@ -70,9 +70,87 @@ This uses **step-file architecture** borrowed from story-pipeline:
### What's ENHANCED:
- ✅ Pre-gap analysis is MORE thorough (validates against existing code)
- ✅ **Smart Batching** - detects and groups similar tasks automatically
- ✅ Implementation is ADAPTIVE (TDD for new, refactor for existing)
- ✅ Works for both greenfield and brownfield
### What's NEW:
- ⚡ **Pattern Detection** - automatically identifies batchable tasks
- ⚡ **Intelligent Grouping** - groups similar tasks for batch execution
- ⚡ **Time Optimization** - 50-70% faster for repetitive work
- ⚡ **Safety Preserved** - validation gates enforce discipline
---
## SMART BATCHING FEATURE
### What is Smart Batching?
**Smart batching** is an intelligent optimization that groups similar, low-risk tasks for batch execution while maintaining full validation discipline.
**NOT Vibe Coding:**
- ✅ Pattern detection is systematic (not guesswork)
- ✅ Batches are validated as a group (not skipped)
- ✅ Failure triggers fallback to one-at-a-time
- ✅ High-risk tasks always executed individually
**When It Helps:**
- Large stories with repetitive tasks (100+ tasks)
- Package migration work (installing multiple packages)
- Module refactoring (same pattern across files)
- Code cleanup (delete old implementations)
**Time Savings:**
```
Example: 100-task story
- Without batching: 100 tasks × 2 min = 200 minutes (3.3 hours)
- With batching: 6 batches × 10 min + 20 individual × 2 min = 100 minutes (1.7 hours)
- Savings: 100 minutes (50% faster!)
```
### Batchable Pattern Types
| Pattern | Example Tasks | Risk | Validation |
|---------|--------------|------|------------|
| **Package Install** | Add dependencies | LOW | Build succeeds |
| **Module Registration** | Import modules | LOW | TypeScript compiles |
| **Code Deletion** | Remove old code | LOW | Tests pass |
| **Import Updates** | Update import paths | LOW | Build succeeds |
| **Config Changes** | Update settings | LOW | App starts |
### NON-Batchable (Individual Execution)
| Pattern | Example Tasks | Risk | Why Individual |
|---------|--------------|------|----------------|
| **Business Logic** | Circuit breaker fallbacks | MEDIUM-HIGH | Logic varies per case |
| **Security Code** | Auth/authorization | HIGH | Mistakes are critical |
| **Data Migrations** | Schema changes | HIGH | Irreversible |
| **API Integration** | External service calls | MEDIUM | Error handling varies |
| **Novel Patterns** | First-time implementation | MEDIUM | Unproven approach |
### How It Works
**Step 2 (Pre-Gap Analysis):**
1. Analyzes all tasks
2. Detects repeating patterns
3. Categorizes as batchable or individual
4. Generates batching plan with time estimates
5. Adds plan to story file
**Step 3 (Implementation):**
1. Loads batching plan
2. Executes pattern batches first
3. Validates each batch
4. Fallback to individual if batch fails
5. Executes individual tasks with full rigor
**Safety Mechanisms:**
- Pattern detection uses conservative rules (default to individual)
- Each batch has explicit validation strategy
- Failed batch triggers automatic fallback
- High-risk tasks never batched
- All validation gates still enforced
---
## EXECUTION MODES

View File

@ -1,7 +1,7 @@
name: super-dev-pipeline
description: "Step-file architecture for super-dev workflow - works for both greenfield and brownfield development with disciplined execution"
description: "Step-file architecture with smart batching - works for both greenfield and brownfield development with disciplined execution and intelligent task grouping"
author: "BMad"
version: "1.0.0"
version: "1.1.0" # Added smart batching feature
# Critical variables from config
config_source: "{project-root}/_bmad/bmm/config.yaml"
@ -26,11 +26,57 @@ modes:
description: "Human-in-the-loop with menu navigation between steps"
checkpoint_on_failure: true
requires_approval: true
smart_batching: true # User can approve batching plan
batch:
description: "Unattended execution for autonomous-epic"
checkpoint_on_failure: true
requires_approval: false
fail_fast: true
smart_batching: true # Auto-enabled for efficiency
# Smart batching configuration
smart_batching:
enabled: true
detect_patterns: true
default_to_safe: true # When uncertain, execute individually
min_batch_size: 3 # Minimum tasks to form a batch
fallback_on_failure: true # Revert to individual if batch fails
# Batchable pattern definitions
batchable_patterns:
- pattern: "package_installation"
keywords: ["Add", "package.json", "npm install", "dependency"]
risk_level: "low"
validation: "npm install && npm run build"
- pattern: "module_registration"
keywords: ["Import", "Module", "app.module", "register"]
risk_level: "low"
validation: "tsc --noEmit"
- pattern: "code_deletion"
keywords: ["Delete", "Remove", "rm ", "unlink"]
risk_level: "low"
validation: "npm test && npm run build"
- pattern: "import_update"
keywords: ["Update import", "Change import", "import from"]
risk_level: "low"
validation: "npm run build"
# Non-batchable pattern definitions (always execute individually)
individual_patterns:
- pattern: "business_logic"
keywords: ["circuit breaker", "fallback", "caching for", "strategy"]
risk_level: "medium"
- pattern: "security"
keywords: ["auth", "permission", "security", "encrypt"]
risk_level: "high"
- pattern: "data_migration"
keywords: ["migration", "schema", "ALTER TABLE", "database"]
risk_level: "high"
# Agent role definitions (loaded once, switched as needed)
agents: